Should I keep it on or off?
I heard you miss some quests if you keep it off.
Is there perhaps an option where it can be only turned on in combat mode?

I’m curious, too. The mini map was on during my first no-DLC game but I’m considering turning the map off for the next play through –after DLC 16 is released.

I played a few hours with it off. The game is so much more immersive with it off, but I did feel like I missed too much. I'm about to start my second play through where I'll turn it off for the entire game, but for the first play through I left it on, but turned almost all the map icons off.

The only UI I have displayed is health, useable item status and effect durations + stuff like breath meter, horse stamina and enemy health. So for vast majority of time, there's no UI displayed. I also don't use fast travel.

Effect of this change? I have memorized most major roads and can properly navigate trough Novigrad using landmarks. Going around the place feels a lot more satisfying, and when exploring, no obtrusive UI is in the way. Furthermore, when trying to get somewhere, instead of staring at minimap like an idiot, I have to plan my path beforehead and it just feels a lot better to think "Okay, so I'll go straight forward until I hit the river, then left, after a small bridge I go right and I'll check with the map once I reach this house over there." than to look at your constant overhead GPS. I'm also noticing a lot more details and little touches about the environment. I really wish they did something similar to Morrowind and NPCs actually tried to actually verbally navigate you towards your destination, it would be a lot more immersive - and sadly, require a lot more cooperation between quest writers and level designers, so it'd be a lot of work. Oh well. Quests actually get added to your map pernamently (exclamation marks stay there even after you leave an area) and you can detect most of the question marks by spotting monsters guarding them - as long as they actually render in time, ehm. So the only thing you have to do is to:
a) Keep an eye on your surroundings - most of the time, when there's a quest giver, he's trying to talk to you. Be wary of any lone NPC or small groups of NPCs along the way
b) Every now and again check the world map and take a look at the upper right with numbers of discovered stuff displayed. The moment you see 1 or more question marks there, it means you have found a quest and didn't notice it at the time.

Yeah, I kept playing last night with my end game character and turned *everything* off. I have to echo one of Fenixp's comments that I tried to touch on my original post: The sense of knowing the streets and roads... That was what I meant by "immersive". I can look at a mountain now and know what mountain it is from any side, navigate all of Velen by sight of "the tree". It grounds you in the world and makes it that much more engrossing.

And I realized shortly into my session last night that was actually finding plenty of the 'local area' quests wandering around, which felt nice and organic.