Try this. Quote from -
A compendium of tweaks and fixes for the PC version 0e. Deleting your "user.settings" and "user.settings.bak" files after a new patch To avoid any possible performance deterioration as a result of patching the game to a newer version after applying the .ini tweaks below, it is best to navigate to your ".../Documents/The Witcher 3" folder and delete the "user.settings" and "user.settings.bak" files, then start up the game as normal, reset your settings to what they were before and only after that reapply all of the .ini tweaks that you had before!
8. Fix stuttering caused by too frequent garbage collection The game uses a garbage collection algorithm to clear up assets that are no longer required as you play the game, but its low default upper memory limit can cause stuttering in crowded places, most notably Novigrad. To force garbage collection to trigger less frequently and thus reduce/eliminate its associated stuttering, navigate to "...\The Witcher 3 Wild Hunt\bin\config\base" and open "gc.ini" with a text editor (e.g. Notepad), then find and tweak the following variable:
ObjectMemoryTrigger=512