I've isolated the causes of the performance issues I was having with BRUSA (which I did not suffer with my Windows version of Timeshock).
1. The main cause of my nice modern machine being unable to deal with BRUSA above 1024x768 (or during multi-ball frenzies even at that resolution) seems to have been the motion-blur effect. Timeshock doesn't have this feature, of course.
To change this, go to Options -> Graphics -> Detail -> Custom and set "Motion Blur Detail" to "off". I now have smooth play at 1600x1200 at all times.
2. I think the primary reason Win95/98 compatibility mode was useful is because it appears to set the processor affinity to a single CPU core. Timeshock and BRUSA both have big issues running on more than one core. If you have a multi-core CPU and you are still suffering some problems, try manually setting the processor affinity and NOT using compatibility mode. (I think this has resolved the audio popping I was hearing with audio hardware acceleration enabled.)
GOG, perhaps you could look into supplying a utility to set processor affinity automatically for multi-core CPUs, and packaging that with games like this, such that the shortcuts you generate to run the game always ensure that only the first CPU/core is used?

Yes, works for me on a quad core vista 64x. Thank you very much.
I was regretting buying of this game seeing how it lags on a new pc especially on multiball.
Now works like a charm

Weirdly, step 2 (using processor affinity instead of compatibility mode) hasn't worked properly for me since I posted that, and I'm now getting seemingly-random keyboard input issues, which can make the flippers stick, or mess with selecting options in the menus.
WIn95 or 98 compatibility mode fixes this, but then I need to disable audio hardware acceleration to fix the sound.
I'm positive it was fine at the time I last posted, though, so I feel like there must be something I could do to resolve this (if I only had the faintest idea what!).
A bit frustrating, but never mind.
Can anyone tell me know how to script the audio acceleration settings with a batch file?
edit: hmmm... well, I found this list of registry keys. Hardly ideal, but if no one has a better idea, it sounds like it would probably work for XP.
http://social.msdn.microsoft.com/Forums/en-US/csharpgeneral/thread/3a053e43-8aed-4094-ad9b-e1eeaefa0ae8

I'm not absolutely positive about this, but I think the remaining and inconsistent issues I've been having with sticky flippers and the like, may simply be due to other processes running on my machine.
I was having issues with Fantastic Journey today, and I decided to experiment. I have a dual-core CPU, and the game was running on CPU 0. I went to the other processes running under my user, and manually assigned them all to CPU 1 (tedious, but it was an experiment), and then returned to the game.
The result: Aside from some slight occasional glitches with the music, I didn't have any further problems. Certainly no sticky flippers.
I then loaded up BRUSA, and it seemed fine as well.
Now, I didn't play either table for long enough to feel 100% certain about this, but once again it seemed worth posting about.
So now I'd be keen on a utility which sets processor affinity for all my running processes, so I could try this again in future without the tedium...

Here are some processor affinity tools I've found:
http://www.beyondlogic.org/solutions/processutil/processutil.htm
Command-line tool to "view, kill, suspend or set the priority and affinity of processes"
http://technet.microsoft.com/en-us/sysinternals/bb897553.aspx
Command-line tool to execute a process, specifying the desired affinity.
http://www.activeplus.com/us/freeware/runfirst/
An even simpler command-line tool for running a program on your first CPU/core.
I was already using this one, so my Fantastic Journey shortcut specifies the following:
Target: "E:\Program Files\RunFirst\RunFirst.exe" "FantasticJourney.exe q"
Start in: "E:\Games\Pro Pinball\Fantastic Journey"
http://www.tomshardware.com/reviews/bang-dual-processing-buck,815.html
GUI tool from Tom's Hardware Guide.
http://www.softpedia.com/get/System/System-Miscellaneous/Processor-Affinity-Manager.shtml
GUI tool. I haven't tried this one. It sounds a bit more heavyweight, but not in a way that's very useful to me.
http://www.robpol86.com/pages/imagecfg.php
Command-line tool. I think this one actually patches the affinity mask into the executable. I've used it in the past, but the alternatives above are more to my liking.
edit:
http://www.koma-code.de/index.php?option=com_content&task=view&id=88&Itemid=93
(GUI tool, noted by WBGhiro)
edit2: Apparently in Windows Server 2003 and later, the "start" command has an /affinity switch. This would be the simplest method of all, if it's applicable to your OS.
The first utility accepts process names as well as IDs, so a small script could alter the affinity of any known process names that you don't want to run on the same core as the game, then you could launch the game on the core you want, and then finally revert the original changes after the game exits.
I'm not using compatibility mode on Fantastic Journey or Big Race USA any longer. Fingers crossed that this 'solution' continues to work correctly... (I'm still setting compatibility mode on Timeshock, but I've never experienced any problems with that table, so I'm not going to try to fix what isn't broken!)
I don't recommend using this, but here's a quick bash script proof-of-concept. (If that means nothing, this isn't for you.) I'm using cygwin on WinXP.
Process.exe -v | egrep -i '\.exe .+ (COMPUTER)\\(USER)'
| sed -e 's/^ *\(.\+\.exe\) .*$/\1/i' | xargs -r -I{} Process.exe -a {} 10
RunFirst.exe "FantasticJourney.exe q"

Just to follow up, I've played loads of Fantastic Journey and quite a lot of Big Race USA since my last post, using the Process utility to run the game on the first CPU core, and all my other processes on the other one, and both games have been running flawlessly every time -- no glitches, no sound problems, no sticky flippers.
At some point I should automate it a bit more, but the commands are in my shell history, so it's all pretty painless :)
When I'm done playing pinball, I reset the other processes using the same commands with a mask of '11' instead of '10'. If you have more than two cores, the appropriate affinity masks will differ slightly, of course -- AFAIK it's just a bit-mask, so just prefix another '1' for every extra core you have.
I'm inclined to look into doing this for every game I have which doesn't like multi-core CPUs. Makes sense to be able to largely dedicate a core to it in that situation.

Ahh, that makes sense, I might give this a go to see if it improves performance a bit. It seems to be running alright for me personally, but every so often it will go funky. As I recall I did this for Thief 2, since it's Dark Engine doesn't like dual core processors. Hadn't occurred to me to try it with this.

Re: that script for multi-core CPUs to apply processor affinity masks to running processes...
I thought I'd look into an alternative for those who don't use cygwin. You will still want to have some comprehension of Windows/DOS batch files.
If you do not have a multi-core CPU, this is of no use to you. It works for me on Windows XP. I have absolutely no idea whether it will work on Vista.
This is all the equivalent of bringing up the task manager, right-clicking a process, selecting "Set affinity...", un-setting the first CPU, clicking OK, and repeating for every single process running in your user name; then running the game on the CPU you told the other processes not to use; and finally when the game has finished, letting all those processes use all the CPUs again.
Usage of this is entirely at your own risk. It's kinda heavy-handed, and if any of your processes were already running with a custom affinity mask, this will mess with that. The nice thing to do would be to remember all the original process affinity masks, and restore those afterwards, but I'm not going to go as far as that. (If someone else wants to run with this, though, please go ahead!)
Here's what to do:
1. Get http://sourceforge.net/projects/unxutils
Click: "Download", "Download", "UnxUtils.zip"
2. Get http://www.beyondlogic.org/solutions/processutil/processutil.htm
Click the download link (below the usage instructions, and above the revision history).
3. Get http://www.activeplus.com/us/freeware/runfirst/
Click "Download".
4. Unpack UnxUtils into its own folder, somewhere.
5. Unpack Process.exe into the (newly unpacked) usr\local\wbin folder of UnxUtils
6. Unpack RunFirst.exe into the main UnxUtils folder.
7. In the main UnxUtils folder, create the following batch file, and call it "affinity.bat":
@echo off
bin\sh -c "cd usr/local/wbin && PATH=. && Process -v | egrep -i \.exe\ .+\ $USERDOMAIN\\\\$USERNAME | egrep -v '\b(sh|Process|egrep|sed|xargs)\.exe\b' | sed -e 's/^ *\(.\+\.exe\) \+\([0-9]\+\) .*$/\2/i' | xargs -r -i{} Process.exe -a {} %1 | egrep -v '^(Command Line|Copyright|Affinity Mask Successfully Set|$)'"
8. Create a second batch file called "play.bat":
@echo off
set mask1=10
set mask2=11
echo Setting Affinity Mask: %mask1%
call affinity %mask1%
echo Starting program on CPU 0
RunFirst %1
echo Setting Affinity Mask: %mask2%
call affinity %mask2%
That's for a dual-core CPU. If you had a quad-core CPU, the mask values would be 1110 and 1111
9. Create shortcuts to play.bat for each game. Append to the Target the executable path for the game, and any command-line arguments, all within double quotes.
e.g.:
play.bat "E:\Games\Pro Pinball\Fantastic Journey\FantasticJourney.exe q"
In my case, this shifts all of my user processes over to core 1, runs the game on core 0 (with the q argument to skip the loading screens), and then spreads all my user processes back across both cores afterwards.
edit: I changed the affinity.bat file to target the process ID rather than the process name. Multiple instances of the same process weren't caught otherwise.
edit 2: Changed batch files to display the useful output, and only remove the redundant bits. Also filtered out the names of the processes that we're calling here from the list of processes to modify (as some of them have already exited before we try to modify them).

Another possible enhancement to this system would be to increase the process priority of the game's process.
The highest priority is "realtime", but you are warned against using this as it can starve system-critical processes, and it is recommended that you use "high" instead.
What I've been unable to find by searching for the past half-hour or so is whether or not those processes will still starve if the realtime process is running on a different core?
I would sincerely hope that this wasn't the case, but I've simply been unable to find anything which states the answer. Please post here if you know (preferably with a reference).
Even if this works the way I hope, I'm not certain it's a good idea to use "realtime", but "high" should be a no-brainer (that said, I believe the game may already run at "high" by default.
I originally used RunFirst.exe for simplicity, but that has no priority options that I've noticed. For testing this out (albeit not at all rigorously), I'm switching to using the psexec tool from SysInternal's "PsTools" package, which can do both.

A clear answer would require intimate knowledge of the scheduling algorithms of the specific operating system considered, but based on what I know about scheduling and operating systems in general, I doubt it. Most modern scheduling algorithms employ code to guarantee all processes receive a minimum amount of each important resource, or at least CPU time, in order to prevent any process from starvation. So while higher priority processes are favored whenever possible, other processes will not starve because of it.
The kernel itself will never starve because it has a higher priority than any other process, even real-time ones. I would assume the kernel handles the resource distribution to the system processes it deems as critical in turn.

Great explanation stonebro.

Ok, BRUSA's "Motion Blur" effect can make the game very stuttery, with huge FPS variations, so your tip for turning that OFF is really great, thank you for that Shadowcat. Other than that, I have no issues running the game in my E8500, XPSP3 @ 1600x1200. Really smooth at constants 75FPS (my CRT max refresh @ that rez).
No affinity, priority and compat. adjustments necessary on my box, so I'm happy.
ps: my sound acceleration is already set to "medium" to fix issues with Prototype.

I've been playing the Pro Pinball tables again since the recent Kickstarter project, and I've been using the 'realtime' priority to no obvious ill effects, so I'll continue to use that in the hope that it will eliminate the rare instances I still experienced with delayed flippers.

YMMV of course, and I would not attempt to set this priority until you have restricted the process to a single core of your processor (and you would almost certainly not want to try this at all on a processor which was not multi-core).

Ironically, it sounds as if the sticky-flipper problem (which could be restricted to Windows XP) may have been tied to the game elevating its own priority! So my attempts to eliminate the issue by increasing the priority were perhaps both misguided and doomed to failure.

Information gleaned from here:
http://www.pro-pinball.com/forum/viewtopic.php?f=12&t=282#p1445

I understand that the unofficial 1.20b patch attempts to stop the game from using a higher priority.

I believe you'll still need to deal with the processor affinity issue.

(Hopefully soon moot for Timeshock at least -- with the Ultra version now released for iOS, I suspect the PC version will be with us in the not-too-distant future!)