Retrograde: Thanks Namur, got mine to work with your hints. If anyone else needs more detailed instructions here's what i did to get it to work under win7 x64:
1. Change the setup utility (realMYSTSetup.exe) in your realMyst folder to run as admin always
2. Set the RealMystStart exe to windows xp sp3 compatibility and check off run as admin.
3. Set your 3D Hardware to Direct3d HAL and
NOT Direct3d T&L HAL
4. Set your resolution to whatever you want (I'm running 1920x1080 32 bit)
5. Set your texture quality to High (that's what I'm running at least)
6. I have Full Scene Anti-Aliasing on and it works fine, ymmv
7. Select ok and run RealMystStart.exe (the exe with the yellow gog icon and not the myst book one).
8. Hopefully play like me.
Hope this helps.
That does work, unfortunately without HW T&L the game lags in places. Still, guess it's better than nothing.