Hello!

On the Dark Knight System tab, you'll see four different panels: Shadow Vigilante, Gotham Protector, Worst Nightmare, and World's Greatest Detective. Each of these have different types of challenges. You need to do them in order. The game doesn't bring this up for some reason.

Edit: something I forgot to mention. The Dark Knight System challenges only progress through the campaign. You cannot use the challenge maps to progress through the DKS. Another point the game fails to mention.

What's more, some of the predator challenges (Worst Nightmare), not all of them, ONLY trigger in "offical" predator rooms (where you're locked in until you take out all the thugs) and not in outdoor locations where you can grapple away and leave the scene. (such as the deck of the Final Offer, Penguin's ship) Some of the challenges can be done on any armed enemy group, but if you keep trying to do a challenge and it won't register, that's because it's limited to official predator encounters. That means, yes, it's possible to run out of predator rooms and be unable to complete this list during a playthrough.

Dark Knight challenge progress do carry over into New Game+ if I can recall, so if you missed your chances on completing any remaining Worst Nightmare challenges, you can still get those done on a second playthrough. But as you noted, they all must be done in story-based predator rooms so if you still did not finished all the Worst Nightmare challenges, then you're pretty much in a tough spot for 100% completion. Also doesn't help New Game+ makes the game harder so screw-ups can be more frequent.

You still have I Am The Night mode but seeing how dying forces you to play through the entire game again, then good luck on doing some of the more annoying challenges like taking down three thugs with a single propane tank, that one is frustrating.