How does it work? It seems horribly inconsistent. I'll be "out of sight" and pickpocket and they attack, and I'll do the same thing and nothing happens.

Is there some numerical formula available? I really don't know HOW it works and the manual and all of the guides that I've looked at are in the line of "bump and take!" without any kind of actual advice.

Edit: Does the spell invisibility help with stealing?

I've only pickpocketed with an already high stealth skill which presumably factors into it. There does seem to be some level of failure involved. Characters will often give a verbal warning ("I saw that!") when they know you're pickpocketing them; if you cancel the attempt at that point without taking anything they won't attack.

Invisibility is ideal; you can pickpocket from any angle, even in front of the person.