I do not like to have music playing in any game, so I tried to use the options to reduce the music, without result (the position of the settings makes no diference to the volume of the music).

I have installed the gog version of the game, although I still have the original 2 CDs boxes French version of this game and finished it back then.
I then installed Zengar's patch to get high resolution, which worked perfectly for me.

I am replaying the game on a Windows 7 64bits laptop

I tried editing outpost .ini, but the game overwrites the relevant music settings with default ones.
I though about renaming the music folder, but the game would probably simply crash if it could not find the files.
I read though pages of old posts but did not find a solution (could have missed, there are so many posts...)

Does anyone have a solution to this problem, other than shutting down the sound altogether (I want the ambience sound, and only that) ?

Maybe this can help:
http://www.gog.com/forum/outcast/music_volume_not_changing

That was it, thank you :-)

I edited soundcard_other.ini, set all 5 music settings to 0, and no more annoying music.

Glad you figured it out. Let me humbly suggest that you check out the soundtrack when you're not playing though, because it's one of the more striking game soundtracks I've encountered. The tracks are included in the bonus content with the GOG version.