I'm discovering new weapons but don't have enough space to pick them up. Is there a place where I can dump my belongings so I can switch them round later?

It's been awhile since I played through this game, but I recall only revisiting a few areas. For the most part, the game doesn't give your much opportunity to come back to previously visited areas. With that said, I never tried stockpiling any of my inventory anywhere to see if is there when I go to retrieve it.

I suggest your carefully consider what you carry so you can deal with any situation. Personally, I prefer to carry a hand to hand combat weapon, a shotgun, a pistol, a crossbow (I like how every hit is a "kill" if you wait long enough), a gep gun, some medical packs/batteries, and a few grenades/mines.

Not on vanilla. There are mods that allow it for some areas (such as JC's office) but the game will consider even revisiting a previous area as a whole new map. I mean it's a new map even when you go back to Hell's kitchen for a different mission, but they will remain there even if you switch maps so long as you didn't get a clear end of mission cutscene (helicopter, usually)