Janus3003: You're not alone, I had that problem last night, too. I don't know any fixes, but then again, I'm not sure there can be any fix. I got the impression that it's a quirk of the game engine itself.
Basically, my guess is that without a crosshair or reference of some sort (such as Doom's centered guns), you can't tell precisely where you're pointed. I think it's not so much that we're drifting diagonally, but that we're pointed just a degree or two off of where we think we're pointing.
I'm glad I'm not the only one experiencing this, but it doesn't make any sense. Daggerfall is an older game with similar controls, but doesn't have this problem. I've played hundreds of games without crosshairs, it's very easy to tell where you're pointing.
Trust me, I've spent over an hour testing this in Battlespire and it's absolutely drifting, and it only drifts when you're facing certain directions. This is not simply a matter of perception; movement is bugged in some way when facing to certain compass directions.
Like I said, you can literally stand dead-on in the middle of a hallway, hand off the mouse, and only press up and down, and it will slowly rock you to one side of the hall until you're bumping into the wall. I refuse to believe that the game is meant to be played this way. Something is wrong.