I can't seem to start this quest. I've been talking to Maxson a bunch of times, sometimes even using Mentats to increase my charisma but to no avail. Is there a level requirement to this quest or do I need to finish that rescue the initiate mission first?

Have you finished the "Become an Initiate" quest from Cabbot, by bringing back the Brotherhood Tape from The Glow? (or did you lock-pick the entrance)

Have you rescued the initiate first (Quest from Talus on level 1)? You get a prompt reward, if you've already freed the prisoner in the Hub. (Usually this quest is optional, so I'd be surprised if this were the cause)

Have you been talking to Vree on level 3, yet? Maybe the info about super mutants she gives triggers the interest of Maxson in a scout mission.

Are you able to offer to scout in the dialog with Maxson? I'm not sure if this quest is properly listed as a quest at all.

*** minor spoiler ***
You can try going to the northwestern part of the map, until you find the Military Base. Once you entered the map, you can report back to Maxson.

Have you tried going to the location, first? I don't think talking with Maxson is required to start this quest.
Find the place, enter, leave, then go to the Brotherhood and talk with Maxson.

If you have Int 1, don't bother.

Yeah I don't think you need to get "given" this quest to move the plot along, or even get the reward for it. Come to think of it I'm not even sure you do get a reward for this quest in XP, although once you've found what he's looking for (you'll know it when you've found it) he'll be (possibly) willing to help you. At the very least you should get some different dialogue then, and (possibly) a more "direct" "mission" (as such).

Yes, I've finished it and my lockpick skill is not high enough to pick the entrance. I still haven't done this quest. Yes, I've talked to Vree and she gave me these holo-tapes which I already uploaded to my pip-boy. No, he just give me some questions to answer which doesn't really point to any missions. Thanks! This really started it!
EDIT: Maxson still doesn't give me options to scout the northern wastes nor any topic to talk about it even if I talk to the other Elders.

Give it a try and just scout the northern wastes, find a certain location, enter the map and go right back to Maxson to report, to see if he offers different dialog options. You can always go back to your current savegame, if this doesn't work.