Ever since coming back to the game I've noticed Novigrad was chugging worse than usual. At first I thought it has to do with the Blood & Wine update ruining some optimization but then I took a look at the CPU usage during gameplay.
CPU usage in Novigrad was reaching 100% and bottlenecking the GPU.
What more GoG Galaxy overlay was taking up 25% CPU power during gameplay, even tho I had Galaxy off and launched the game without Galaxy.

For some unbeknown reason GoG has decided to force GoG overlay on all their games even when you don't launch them via Galaxy! And the overlay takes ridiculous processing power while you're playing for some reason.

THE FIX:
1) Launch Galaxy.
2) click the GoG.Com icon on the top left -> Settings-> Uncheck "In-Game overlay".
3) Shut down Galaxy.
4) Play/