I just bought HoMM 2, but the music in the game is not at all what I remember. Are there different soundtracks for different versions or something?

There are two soundtracks, one from vanilla game and the other from the expansion. GOG uses the first one, though (my retail gold version uses the other one).

IIRC someone found a way to do it, but you'll have to search for a thread about HoMM2 soundtrack. If I wasn't on my phone, I'd find it for you quickly.

Cheers, I'll give it a try. :)

There are 3 (three) different soundtracks from different editions of HoMM II:
1. HoMM II: The Succession Wars DOS/Windows v1.0 CD with original soundtrack (released November 4, 1996).
2. HoMM II: The Price of Loyalty expansion pack DOS/Windows v2.0 CD with updated soundtrack (released May 5, 1997).
3. HoMM II: Gold (CD2, Windows only, ingame version 2.1) from Heroes of Might and Magic Millenium Edition 3CD set (collectors edition released August 26, 1999). Very strange soundtrack here (mixed with HoMM III tracks in CD-Audio quality and few strange tracks).

Each HoMM II CD contain 1 Data and 42 CD-Audio tracks.

I explained in my forum post here how to choose your in-game town musics (soundtracks) in HoMM 2.