I don't know about PC, but my parents bought our very first computer when I was about 5 in the very early 90s - it would have been the Macintosh Classic II, so not exactly a PC. I remember having a lot of educational programs on that computer, we had Math Rabbit and Reader Rabbit, spent an unusually large amount of time playing around in Kid Pix and also played a game, if you can call it that, called McGee. I tried for ages to find evidence of its existence, and then just over a year ago noticed LGR had it in his collection. Earlier this year I also found a video of it on YouTube, although that was the Amiga version, not the Mac.
A few years after that we upgraded to a Colour Mac and it came with a copy of the game Myst. I must have been around 9 or 10, and considering what I was like as a child, I should have found its atmosphere quite creepy, but for some odd reason found it relaxing. I managed to solve 3 of the ages without a walkthrough! We also had The Simpsons Virtual Springfield and Cartoon Studio, but most of the games we played on this computer came in the form of demos that were on a CD that came with a magazine called Mac World.
We must have got our first PC in early 2000, and we didn't have many games on it, because it was the first computer we had that had Internet access, so we were kept busy with that, but around July, we got The Sims, and for Christmas that year, between me and my sister we got The Sims: Livin' Large, Rollercoaster Tycoon and Sim City 3000.
So first game on first computer, if you count edutainment and Macs would be McGee, If you don't count edutainment, it would be Myst, and if you only wanted PCs, then The Sims.