Ok, i had the exact same problem a while ago when i was trying to play my original copy of RealMYST on Win XP SP3, after much searching i found the solution from a well known member of the MYST Community so all props and rep should go to him, Jehon the Scribe, for finding this fix and posting it, i am just sharing it. The original page with the solution has been off line for a while now which was Jehons
website. It is also not indexed by the Wayback Machine. Fortunately i copied the whole text from his website and saved it on my computer.
Anyway, this fix should work for people having troubles on Win XP, Win Vista, and Win 7. Jehon tested it on Win Vista, i have tested it successfully on Win XP, applying it can only be beneficial regardless of the Windows version you are using. I don't even have to use compatibility options anymore, and i can run it on the highest resolution with all the bells and whistles enabled and it runs like a train without any crashes.
MODS: if the following instruction are against the rules, please remove my post, but please forward this fix to the GOG programmers so they can incorporate it into their RealMYST package which will save a lot of people headaches. So without more useless chatter, here are the complete instructions to get your RealMYST fixed:
realMYST and Windows Vista
Foreword: The following article concerns the retail Windows version of realMYST. I know nothing about the GameTap version although I suppose it suffers from the same problem.
the real cause of the problem...
When you start the game realMYST on Windows Vista, it is very likely that it'll briefly show a black screen and then immediately terminate again. The reason why it does that is simple. It has nothing to do with Windows services, drivers, compatibility settings or other things some people suggest. It is because of a stupid mistake in realMYST's programming. That mistake had no severe consequences on previous Windows versions, but even there the game sometimes failed to start.
As you might know, realMYST uses three custom "dni" archive files to hold much of its data – mara.dni, pera.dni and sho.dni. The first two contain only music and sound effects whereas sho.dni contains initalization settings, 3D models and more. If realMYST can't access that data, it obviously won't be able to run, and that's precisely what's happening when it terminates right after starting.
So, what is going wrong, why can't realMYST read from sho.dni? That's where the programming mistake comes in. To understand what's going on you have to know that realMYST employs a technique called memory-mapped file to access those dni archives, by utilizing certain functions the operating system provides. The details aren't important here, just that two of those functions are called MapViewOfFile and UnmapViewOfFile. The former makes a part (or all) of a file visible at a certain memory address while the latter removes that view from memory again. Here's how realMYST does it:
Open the dni file and prepare for memory mapping.
Call MapViewOfFile to map the first 28 bytes into memory (that's the header of the dni file).
For technical reasons, more than just 28 bytes will be mapped, but that is irrelevant.
Remember the memory address at which the file is visible for later use.
Check if the file actually is a dni file by examining the first 8 bytes of the file header.
(dni files start with the letters "Dirt" followed by a version number.)
If it's no dni file then quit.
Call UnmapViewOfFile to close the view.
Call MapViewOfFile again and this time map the entire file into memory, not just its first 28 bytes.
Continue doing other stuff...
Did you spot the error? It's step 3! The programmer who did this either wrongly assumed that the complete file will be mapped to the same memory address in step 7 as its first 28 bytes were in step 2, or he forgot to insert step 3 after step 7. By chance, or by internal design, in older Windows versions MapViewOfFile really did use the same memory address both times. Which is why realMYST worked there, although it probably shouldn't have, because MapViewOfFile was never guaranteed to behave that way.
...and the solution
Knowing all this, devising a fix for the error is relatively simple: make sure that realMYST remembers the correct memory address of sho.dni after its last call to MapViewOfFile. This can be easily done by modifying a few bytes in realMYST.exe's program code.
However, there is a problem – realMYST uses a copy protection scheme that encrypted the program code, namely SafeDisc v1.50.20. The RealMYST.exe file that is about 336 KB in size is really just a loader and decrypter for the actual realMYST program, which is REALMYST.ICD, a file of 1.70 MB in the same folder. Therefore, the first thing you have to do is to obtain the real realMYST.exe, a decrypted version of REALMYST.ICD. A search for "realmyst nocd" or "realmyst fixed exe" with your favorite search engine should turn up some results. Second, you need a hex editor, a program that'll let you view and modify binary files. I recommend HxD, a nice and free program. Then follow these steps:
Search for the byte sequence
00 C2 04 00 5F 5E B8 01 00 00 00 5B 81 C4 00 01 00 00 C2 04 00 90 90 90 90 90 90 90 90 90 90 90
Replace with the byte sequence
00 C2 04 00 8B 07 89 46 08 5F 5E B8 01 00 00 00 5B 81 C4 00 01 00 00 C2 04 00 90 90 90 90 90 90
In case you're not familiar with hex editors and this was too fast for you, here's a walkthrough with HxD:
Open the unencrypted realMYST.exe (should be 1.70 MB in size) with HxD.
Make sure that you have write access to the file, of course. If necessary run HxD as administrator.
Press Ctrl+F to bring up the Find dialog, set data type to hex-values and copy & paste this sequence into the search field:
00 C2 04 00 5F 5E B8 01 00 00 00 5B 81 C4 00 01 00 00 C2 04 00 90 90 90 90 90 90 90 90 90 90 90
Press OK. You should find one occurence of this sequence (at file offset 0014DC40, normally).
If it can't be found, then you don't have the proper unencrypted realMYST.exe or maybe it is already patched.
The file view in hex editors basically consists of three columns: the offset on the left, the hex view in the middle and the text view on the right.
Make sure that the hex view is selected. You can press the tabulator key to jump between hex view and text view columns.
Copy this sequence:
00 C2 04 00 8B 07 89 46 08 5F 5E B8 01 00 00 00 5B 81 C4 00 01 00 00 C2 04 00 90 90 90 90 90 90
and paste it into HxD (Ctrl+V)
If HxD warns you that the file size will change, then you're doing something wrong. The file size must not be changed.
Maybe you have tried to paste while the text view column was active.
Save the changes and move your newly patched realMYST.exe to the game's folder, if it isn't there already. You might also have to enable compatibility settings to run realMYST.exe in Windows 2000 mode. The other settings like "Disable visual themes" etc. should not be necessary, just compatibility mode for Windows 2000 and maybe run as administrator, depending on where you installed the game.
On my system, realMYST runs just fine on Vista x64 (64 bit) without any compatibility settings enabled. On Vista x86 (32 bit) I had to enable compatibility mode for Windows 2000 (Windows XP worked too) because otherwise the game would run very slowly.
Anyway, if you follow these instructions correctly, then you will be able to play realMYST on Vista. Finally. It should work in Windows 7 as well.
Yeah, I could have made it easy for you and just provided a link to a patched realMYST.exe, but aside from the legal issues, where would be the fun in that? ;-) Other people will undoubtedly spread a properly patched realMYST.exe eventually.
— Jehon the Scribe, April 2009 (jehon@gmx.net)