Is each one of the start menu options really a unique stand alone game? (base game, warlords, beyond sword, colonization)

Or is there one option I can use all the time and have access to the base game and all of the additional content?

Warlords and Beyond the Sword are expansions to the Civilization IV base game. Beyond the Sword was the last expansion and includes everything except a few scenarios that are unique to vanilla and Warlords. So Beyond the Sword is the full experience.

Civilization IV: Colonization is a standalone game, a remake of the classic Colonization game using the Civ IV engine.

Most of them come with their own mod and include unique mechanics. For example, one of them lets you play as the Barbarians and your goal is to destroy any traces of civilization. You can just launch them on vanills or Warlords (the only time I don't play Beyond the Sword). It's worth it, they're really fun. But if you really want to play them on BTS, there are some fan made "ports" by user isenchine over there as CivFanatics that adapts them to BTS 3.19. For example: https://forums.civfanatics.com/threads/barbarians-for-bts-3-19.563282/