It seems to check some property associated with the monsters, but that property seems to be reset after clearing it. Not really certain how it is actually suppose to work.

The problem with stealth is that monsters seem innately aware of the player. If you use "hide in plain sight" a monster will still follow you despite your invisibility. Slowly though, indicating it is using something like sound to track you. Sneaking up while under cover often results in similar behavior (inflated skills to 20 for testing purposes), making it seem unlikely to ever get the right conditions for it to trigger.

Of course, just using a stealth check is a little overpowered as there's no delay in recasting stealth and one would expect damage from ambush to decrease with each attack until normal (on the idea that the monster should anticipate the attack the more it gets attacked).

Thanks for the information regardless.