Bargoss: I'm getting the exact same thing and it's driving me fucking crazy. I was using an Phenom II Black edition X4 955 and a 7870, but the 7870 died so I replaced it with an r9 380x. With the 7870, I had most post processing on and most of the graphics set to low/med, but I had background characters set to high. When in towns, my fps was the exact same as outside of towns, which was a steady 45. Now, with my r9 380x, I can't even get above fucking 32 fps in towns, even if I set every single video option the lowest it can go and turn all post processing off. Meanwhile, I can get 50-60 fps outside of towns with everything set to high and a few to ultra.
Also, by towns I mean Crow's Perch, Oxenfurt, and Novigrad. I have yet to get to skellige or the DLC, but I don't imagine those towns will be much different.
Oh, and I'm also quite positive there's a memory leak, as I start the game and it runs at 2.3gb of memory used, none of it precached. In a few hours, task manager will tell me it's 7+gb of not-precached memory used, which will eventually get to the point where it crashes explorer because windows uses 1gb to run and I just have 8gb of ram.
I'm running the game on max settings with a FX 8350 processor and R9 380 card with 16GB of 1866 at 2560X1080 with no problems. Maybe look into upgrading your processor.