In many past playthroughs with the unofficial patches, I have experienced annoying slowdowns and fps at certain locations during the game. It mostly happens in the Hub when I return back from Irwin's farm and Adytum after I spend time level grinding at the deathclaw warehouse until level 21 and then enter the city of Adytum. These slowdowns happen for about five seconds before normal fps is restored.

What the heck is causing this and how do I fix it?