Stealth in the NWN2 OC is just incredibly unreliable. I'm given to understand that in many cases the problem was that the level-designers scripted how the enemies react when you get close enough to aggro, but forgot to account for stealth.

Ah, so i'd be better off stealthing neeshka, then aggroing the enemy with my fighter and having her sneak past and backstab the mages?

That should work, although a more reliable solution is to use the invisibility spell (which works differently than stealth and doesn't suffer this problem). I literally walked through one of the sections of the game where your character has to solo things, and the monsters didn't even blink as I passed through. Sadly, unless you're a spellcaster who can cast this spell, you can't access it until you reach Neverwinter and recruit either the Sorcerer or Bard NPC (and even then you have to level them up to teach them the spell)

Do keep in mind that stealth still has a chance to fail even when it is working correctly, and every 6 seconds every nearby enemy gets another chance to try and spot your characters. This means that even a 95% chance of success starts to look pretty bad if there are five guys nearby who can spot you and you stick around for 30 seconds and give them ten chances each (since spot and listen count as a separate chance). Once you've got your stealth score in the mid-20's, though, many enemies simply have a 0% chance to spot you and it becomes much more reliable.

Keep in mind that you do not need to be hidden to do sneak attacks.
It is enough when the rogue attacs an enemy that attacs somebody else.
Just send your tank in first (Khelgar is a good choice if you are not a tank yourself).
Usually the enemies attac the first thing they see.
Give Khelgar heavy armor and a shield and have Elanee cast stoneskin on him.
I´m not sure if you can cast barkskin on others or only on yourself.

so basically putting points into hide and move silently is completely pointless in this game then?

In the original campaign, that's sadly the case. Because Neeshka cannot use conversation skills, it's not like there's much else to spend the points on so you may as well. Later on, improved invisibility will be your go-to choice for her.

However, stealth works considerably better in Mask of the Betrayer and Storm of Zehir.

Mask of the Betrayer uses epic-level characters, which tend to sport stealth scores in the 30's as a baseline. This is high enough that almost no observer has a chance of succeeding a spot or listen check against them. Storm of Zehir has several areas designed for stealthing. Just make sure you're not underleveled when you reach them (SoZ is free-roam, so it's possible to arrive in some places before you're intended to)

In DnD2 your damage is multiplied by sneak attacs.
My fighter9/thief10+x could do more than 150 points of damage, a kensai could do much more.
This is enough to kill almost every human enemy (especially mages before they can cast
protection spells)
when ALL of the following conditions are met:
-you are invisible or use stealth (the enemy cannot see you)
-you attac the enemy from behind
-you use a weapon that can be used by pure thiefs
-The enemy is not immun to sneak attacks

In DnD3 you add 1-6 points of damage for every odd rogue level. (Some other classes
have sneak attacs too).
When at least ONE of following conditions are met and the enemy is not immun:
-you are invisible or use stealth
-you attac the enemy from behind
-you attac an enemy that attacs somebody else
-the enemy cannot defend himself (stunned, knockdown, maybe something else)

In all games the invisibility/stealth ends when you attac.

In DnD2 I used the thief as first strike to start the fight by taking out the hardest enemy (mage). If he had boots of speed he can kill any group of enemies (unless they are immun to sneak attacks or they can see through invisibility) by running away and hiding again.

In DnD3 I use stealth only for scouting. For me the rogue is a fighter that does lots of damage as long as they attac something else or they are disabled (the mage casts a stun spell or something like that)

In 3rd edition "improved invisibility" does not end when you attack an enemy, making it a very deadly spell for rogues. However, most CRPG's don't implement this properly. The only one I'm aware of that does it right is Temple of Elemental Evil.

Backstabs are a 2nd edition D&D thing. You don't need to use stealth anymore, and in fact stealth is pretty useless in the OC.

In 3rd edition backstabbing's been replaced by sneak attacks (NWN 2 is actually based on 3.5 ed but sneak attacks haven't changed), which trigger when a rogue, or any other character with sneak attack dices, attacks someone who's not attacking him.
It can trigger out of stealth, sure, but the most efficient way to use them is to actually wait for the enemies to beat on your tank, then attack the same one as your tank with your rogue. You'll do a lot of bonus damage that way. Just be careful not to let the enemy attack your rogue or not only will you lose your sneak attack bonus damage, your squishy rogue is also likely to die fast.
Sneak attacks can also work with a ranged weapon provided the rogue is very close to the target.

Keep in mind that all enemies that are immune to critical hits are also immune to sneak attacks. This includes all undead, elementals, constructs and a few other things.

Yea The last D&D game i played was baldurs gate two and Had an assassin/mage dual class who did crazy backstab damage so I was pretty surprised to see how different rogues work in a 3.5 ed game.