Doing it for all games will be nearly impossible. Not completely, but nearly. DOSBox'ed games and other emulated games can obviously already do this, but at the cost of needing an entire simulated machine. For native Windows games... Oy.
If I can devspeak for a moment: theoretically you can do this by running the app through a shim layer for OpenGL/DirectX calls (and a handful of remaining native calls that deal with resolution, like GetSystemMetrics()). This is much easier said than done, because windowed and full-screen mode are really quite different. If the app isn't going through DirectDraw and you need to divert GDI output it's probably even harder (shimming GetDC() isn't for the faint of heart). For OpenGL games it should be slightly easier because OpenGL is much more agnostic when it comes to its output (you'll notice that most OpenGL games will offer windowed output themselves, when DirectX games often don't). It occurs to me that the Wine project might have some useful material/knowledge, though, since they do this sort of translation on a much bigger scale.
In short, this would be a lot of work. You'd run into a huge number of compatibility issues you'd have to fix for each and every game to make sure it runs with the, hum, let's call it the Fenestrator for now. In all honesty, installing a virtual machine and running that in windowed mode would be a lot easier, even if it's a very resource-heavy solution (and you can run into licensing issues). It's challenging and interesting from a technical POV, but it's a big project and one that would likely always remain in beta, since you can never be sure it won't fail on the next game and require updating.
You're much better off looking for a community patch for a specific game and/or a remake that possibly upgrades the textures and overall look as well (Freespace Open does this for Freespace, for example). Many classic games have such projects, especially the early 3D games.
Edit: it occurs to me that a potentially simpler solution is to offer a virtual display device that windows through to a real one. Benefit is that the display technology used by the game no longer matters, but it's only really simpler if you don't offer any 3D acceleration (otherwise you have to emulate that, and you're no better off) and while a lot of old games can probably run fast enough with software acceleration on modern hardware, some won't (and it might look crap even if they do). Plenty will even refuse to run if they don't detect hardware acceleration, though you could fake this. You are still left with the problem of making the games run on your virtual display device rather than the real one, but a lot of games will allow you to select your display adapter regardless, and for those that don't, forcing a display adapter is probably easier than changing all calls. The big counterargument to the "potentially simpler" argument is that developing a stable kernel-mode display driver that can pipe its output to a user-mode window and will run on all the major Windows flavors is not exactly a cakewalk -- and an unstable display driver can crash the entire system pre-Vista, so you better get it right.
Post edited September 15, 2008 by Nudiustertian