At the start of U5 I had just under 60 karma. After freeing some prisoners in Yew I had 65 karma.

I never attacked any innocents. I did "steal" some items though, but I noticed no karma drop after taking an item. But just now I've noticed I have only 16 karma!

I can't explain such a drastic drop, unless either it is tracking items stolen and there's a delay in when it shows your real karma, or it was a bug.

Have you taken steps to acquire a Black Badge? Some of those cause a large karma hit.

There is karma loss from stealing. That's why it's best to rob the Castle Britannia treasury early in the game if you go that route for a boost at the start of the game. What caught me in my early games was that that karma loss from theft includes picking crops for food, even at Iolo's hut IIRC.
It's fairly easy to fix. Do the Shrine quests for an initial boost, then you can trade gold for more karma at previously visited Shrines.

A couple other ways to restore karma:

1. Give money to the poor. This only works once per town trip, so it's more tedious than donating money to a shrine, but it's also much cheaper.

2. If your karma is less than the starting value of 75, getting a game over will restore it to 75. (If it's higher than 75, you actually don't lose any, IIRC.) Unfortunately, this means that at least the main character will suffer XP loss from resurrection which is based on your karma; if you have 0 karma when this happens, you will be level 1 with 0 experience after being revived. It's recommended that you have your companions stay at the inn if you use this method, and be careful once you're revived (as you will be only level 1 with 30 HP).

I never robbed Castle Britannia. But as I said I did *ahem* take some items in Empath Abbey, Yew, and Britain. Thing is though is that, as I said, I saw no karma loss immediately after I took an item. But then later I checked my karma on a whim and noticed the drop.

I checked a few guides online and they seemed to indicate that U5 doesn't track stealing. This error is probably what I get for spoiling myself.

I guess my only real solution is to actually live virtuously.

(Sort of sucks I can't farm at Iolo's hut though.)

Or, alternatively, ignore karma and don't die (or more precisely, don't revive anyone who does die); the only effect of karma that I've noticed is that it affects the XP loss on resurrection.

I think some NPC responses also depend on your karma.

I know Annon in Britain "trust not yet" my Virtue, which I thought may mean I had to get my karma high enough for Annon to tell me his secret.

From what I've read, this is true.

By the way, I started a new game (I've been restarting a lot as I get the hang of U5) and noticed my karma is 75, while the last time I've checked I seemed to always have around 60 karma at the beginning. The difference though is that I didn't take Iolo's crops this time.

:/

Update: I just experimented and found that my karma drops immediately after taking the crops from Iolo's Hut, with no delay. I believe me farming at Iolo's was the source of my karma issues all this time.