I remember my original cd copy of oblivion (just the game, no expantions) had an option to put an enchantment (chameleon) on items that if all worn at the same time adds up to more then 100% chameleon that makes the player character undetectable to all (even when attacking), which completly broke combat in the game.
Was this fixed in the GOG version?

I wasn't thinking it was a bug, just "something that the programmers missed".
I think so because I remember that it was not possible to enchant the "reflect" spell on items, so clearly the programmers cared about balance (at least a little).
Is there a mod compatible with the gog version that makes the game balanced and the combat challenging (not just cancel chameleon) ?

Well, it depends what you mean by "balance" I guess. You are probably the best TES expert when it comes to tweaking the game in all possible ways and directions, and it's amazing how much is possible in this respect. For me, everything you are doing is "shifting" or "modifying" balance. That it's possible to do this speaks for the flexibility of the game design, not for a lack of balance.
My "hobby horse" is role-playing, trying to find diverse ways to enjoy the game in a meta way, play different roles, deliberately not use certain game functions (for example avoid fighting, use no weapons, no spells, etc.), try focus on specific game aspects and view the game from different perspectives. By doing this I'm also influencing and changing the game balance, and again this speaks for the flexibility of the game design rather than against its balance.

Like in the real world, to achieve balance (like between personal safety and personal freedom, or between individual and social rights and duties in our days) is very complicated, and in most cases it goes along with limited flexibility and bureaucracy. Nothing wrong with this, but as a gamer I prefer flexibility.

Here's one way to think of the situation in many TES games:

For a typical game, the task of balancing a game is generally placed on the developers of a game.

For a TES game, however, much of the responsibility is shifted to the player, in terms of the decisions they make in terms of what skills to use, as well as whether and how to mod the game. In other words, some of what is usually the developer's responsibility is shifted to the player.

Incidentally, the games can still have different balance issues. For example, in Oblivion, the fastest way to kill enemies at high levels is to use spells or weapon enchantments containing weakness effects. From what I understand about Skyrim (which I have not player), on the other hand, the fastest way to kill enemies (assuming no Fortify Restoration abuse) is to use weapons that have been significantly upgraded with the help of the Smithing skill, and then use Enchantments and weapon perks to get further multipliers on top of that, then add elemental damage enchantments (that can be boosted by Destruction perks) to add a bit more damage on top of that.

There are other games with poor game balance that can be rather frustrating if bad decisions are made and can become way too easy if the system is exploited to its fullest; Final Fantasy 2 and 8 come to mind, for example. (It's worth noting that the way to make FF2 trivial isn't what you'd expect; getting more HP actually makes the late game harder, while avoiding heavy armor is how you make the game easy.)

YES - it's what makes these games unique, and attractive for me (and unattractive for others).