Can anyone explain how exactly the game determines whether or not you will take an alignment hit for attacking/killing someone? At this point my guy is at -80 alignment from constantly beating up and robbing people (usually I don't kill them though), and now for some reason killing random people no longer causes my alignment to decrease.

However recently I went into a dungeon in the Morbihan Plains and killed a bunch of undead stuff, and killing the boss enemies caused my alignment to dramatically decrease.

What's up with that? I'd have expected killing good people to decrease alignment and killing bad people to either have no effect or raise it, but apparently it doesn't work that way.

Some monsters have their alignment set to the wrong value, and certain items are cursed to permanently lower your alignment.
Are you using the unofficial patch?

I am.

It's possible that, in the case of "evil fighting evil", the game bases alignment hits on the difference between your own alignment and your victim's alignment. So, if you're more evil than the undead, killing them might count as killing "good" characters.

That said, there are some known oddities regarding alignment, and reaction, mostly related to attacking initially non-hostile critters. In other words, killing somebody who already dislikes you in self-defense might be fine, depending on their alignment, but initiating a fight by attacking them first isn't.

Case in point, I have observed a certain party member (who absolutely demands that you be a saintly paragon of justice) object to my good-aligned character's attempt to remotely blow up some mechanized arachnids, who were all alignment 0 - killing them wouldn't lower my own. The apparent reason he was objecting? I was wearing the +20 reaction suit, so the mechas were "friendly" towards me. :/

There's also a well-known issue with fighting groups of certain evil critters, where attacking them similarly can cause good-aligned followers to complain, and can lower your alignment even if you're good. Which, again, may or may not be related to attacking them before they turn hostile.

TL;DR: Nobody seems to know exactly how this stuff works, so for now you just have to observe and make your own conclusions.