Yes, I know. This is about opinions amarite?

Climbing into the wayback machine...
The keyboard puzzle in Myst. If you didn't have perfect pitch, might as well give up and go hunt down the cheat sheet...and even then it required timing and care.
Also the turret game others already mentioned in KotOR. I didn't like swoop racing either, but it was the turret that really gave me fits.

Although i found it okay, I hear Astro Chicken is some peoples nightmare. (SQ3)
I'd go for any mini-game that seems wholly out of place, e.g. Bioshocks pipes and Mass Effects circular frogger-esque game. Or the lockpicking in Oblivion.
edit: anything that involves playing music, as I am inept.

Lock picking in Fallout 3. Cannot get the hang of it.

it also has multiplayer.

I'm not sure if its only for the very easy locks (since I never maxed the lock picking) but I found that if you turn the pin to about 10/11 o'clock it always opens for me.

The difference in mouse speed was remarkable in the PC version, and I didn't have a mouse that would change on the fly at the time. I had to pick up the mouse and move it across the mouse pad four time to make about a quarter turn. I managed to get it after a few tries, but it was the most annoying minigame I've encountered. At least the racing game used the keys to move back and forth and the movement and slide was adequate.
My wife managed to get the hang of the racing game well enough to complete the one required race. But the shrieks of frustration coming from her study after *hours* of trying to escape Taris was truly amazing. While she still doesn't play twitch-based games, she has gotten better through playing games that require accurate and timely reactions, but she still saves before leaving any planet every time she replays KOTOR and reloads immediately if it hits a random attack. But of course, the Taris attack is unskippable.

Maybe we all live in the Matrix after all, so nothing is ever real

Mystical Ninja (for the n64) has this really brutal minigame where you have to swim up a waterfall. You achieve this by pressing A rapidly... really really rapidly
It's timed
and shit falls on your head, bumping you down a great deal.

I'd be distracted too if feces were falling on my head as I swam vertically up a waterfall.
As for bad minigames, I wouldn't say it's the worst of all time, but the tile swapping hacking game in BioShock wore thin pretty quickly. It wasn't so much bad as it was just unnecessarily distractive to the game flow.

Oh yeah. I was confused about how that whole speech game worked at first. Then once I figured it out, it was remained as arbitrary as before I understood how to play it.

That was just bad design overall on that one. It was easier and faster to just bribe the NPC in question since the game just showered you in gold via loot from enemies and any thieving you may have done. Did you ever notice how every merchant had this magical amount of gold that did not mean anything because no matter how expensive the item was you could still get all the gold for it unlike in Morrowind where something that cost 30,000 gold could only bring you 10,000 gold at the most if you found the mudcrab merchant?

Morrowind merchants would wear armour/weapons if you sold them, and had a limited gold supply. I kinda liked that, it stopped me from stealing then selling all the pillows and pots in Balmora.

Hahahahaaa! I think I was actually doing something along the same lines early on in my first playthrough of the game. Hey! Recession calls to desperate measures. XD
I remember finding a really badass sword somewhere on top of the wardrobe in one of the houses. The problem was that there was a guard right next to it. So I actually planned the entire robbery down to where I would run if he notices me and where to offload the phat lewt as fast as possible. It was almost like a mini game in its own...yeah definitely one of the worst mini games I ever played. Heh. :P