ALPHA TEST BUILD AVAILABLE

Please, check the PreAlpha Test Guide.
If you signed in for the PreAlpha Test, you should have got an email in the address you provided with instructions on how to download XWVM.

Original Post from 2014 starts here:

Hi, all X-Wing fans.

I'm a X-Wing enthusiast and hobbyist game developer.
After trying to find the perfect X-Wing edition and realizing that the closest one to perfection is virtually lost to us (the 1996 Mac PowerPC edition cannot be run at a decent speed on modern x86_64 machines), I started looking for the best way to improve X-Wing to get the best of all the editions.

There is the option of importing X-Wing mission to XvT or XWA, and it has been done already before.
However I have found this approach subject to many compromises:
- You lose the setting context of the game (Ackbar or Dodonna talking to you, the concourse music and animations, Tour of Duty roll texts missing, etc). Playing the campaigns in these games feels almost selecting an entry on a list and hitting Launch. Quite immersion breaking.
- Missions must be reworked because of the different simulation rules in these newer games.
- XvT and XWA are old games by now anyway. In the end we are importing the missions to a 14 years old game engine (18 in the case of XvT) that doesn't work very well without extensive modding. Modding itself involves a lot of hexediting in these games, as of course, they are closed-source and not mod-friendly.

And above everything else, that would require you to own another game. GoG doesn't even sell those games right now. And buying them off Amazon (or even pirating them) still has a lot of problems when installing them on modern systems because of the old DRMs or 16Bit installers they had.

So instead, I propose to build a remake of the fligt engine, to start with, that reads the original XWing resources (music, graphics, missions) in your GoG XWing game folder, and uses them to play the mission, allowing for modern resolutions, input methods, and even usability improvements that were later added to TIE Fighter and XvT.

The task is complex but not impossible. It's not at all that hard considering that I'm using Unity3D for the framework and most of the resource file formats have been described in the past. Also, there exist many 3D models created and freely available for most of the ships in the game. The XWAU project has good material, but we aren't restricted to use XWA OPT model files. Or even better if we don't.

I chose the name XWVM inspired on ScummVM because:
- It will be a free utility that includes no copyrighted material.
- You need to own the orginal game, encouraging sales in GoG or other media.
- I will be multiplatform, as Unity3D targets Windows, Mac, Linux and even Android and iOS.
- It will make use of the original resources, but some improvements can be added where it makes sense.

Also, unless I'm horribly mistaken, I believe ScummVM is open source project and we could reuse some of their work to implement some systems in this one, for example, all concerning the iMuse system.

Some improvements I have been thinking on/have done already:
- Of course, any resolution and screen aspect ratio. That comes for free with the framework.
- Mouse input support, both in X-Wing DOS fashion (absolute mouse movement), and in Freelancer fashion (relative with dead zone).
- Joystick support with ability to map any number of buttons and axes.
- Other kind of input support? Touch screen support for cockpit control toggles (ELS system, throttle, shield angling, power transfer, etc).
- Remove roll/pitch/yaw limitations when flying in particular directions (in special, when going "up" and "down" the vertical axis)
- Retrofit X-Wing with TIE Fighter's and XvT targeting improvements. That is, holographic model of the targeted ship on the cockpit monitor, along with basic stats, cargo and directional indicator. Maybe also the target information screen (Z key in TIE fighter) either as a different screen/window, or integrated into the monitor.
- Goal completion list.
- In-flight voice hints for events and status of critical crafts ("Message from mission critical craft: Their hull is damaged!")
- Directional hits for events and important craft. An arrow-like icon on the border of the screen pointing on the direction of some important craft, or the selected target.
- Virtual cockpit a la XvT/XWA.
- Choice between original cockpit interiors a la XW/TF/XvT or 3D interior like in XWA.
- Choice between original concourse navigation a la XW/TF/XWA or a 3D concourse in the style of the base in XCOM.
- Optional mission extending patches: Little editable files that apply extensions to original X-Wing missions, like in-flight messages (possible with voice), rebalancing passes, enable hangars in allied spaceships where to get repaired or refilled during battle, etc, in order to make X-Wing missions as rich as TIE Fighter's ones without having to rebuild them from scratch. Also making the game more accessible for today's audiences.
- Cooperative play: Up to 3 players can join another and select a ship on his flight group or another allied flight group to attempt some hard mission together. Deaths aren't permanent for coop players. Progression is recorded for all involved players.
- Online leader boards with hiscores.

What is your opinion/interest on something like this?

EDIT:
Latest video
ModDb page

I'd buy that for a dollar!!!

Sounds great, I admire your enthusiasm. :^)

A few things though:

Have you considered to contact any existing mod groups, like XWA upgrade group or the one that is making an Star Wars mod for FSO2?

Maybe wait for Star Citizen or Elite Dangerous to come out and see if they can be used for your mod, might save you a lot of work (or X Rebirth even).
Actually there is the Vega Strike engine which has been succesfully used to make a Wing Commander: Privateer remake.

VM in ScummVM stands for Virtual Machine, from what I understand you want to rebuild the game engine and use assets from the original games which can be confusing to people as it isn't the same as emulating the game in a virtual environment. Scumm was the name of the game engine LucasArts used for many of their succesful adventure games.

Thanks.
Yes, I know that the VM part doesn't fit much, right now. A virtual machine is supposed to be a common layer over which some other code runs. DosBox would be a VM, for example. This would be not, unless you consider the Mission .XWI files to be the code to be run. I admit it's a quite stretched comparison.
The fact is that I am actually not so sure that ScummVM is a VM either. If I am not mistaken, they have had to implement and maintain several different "versions" of the Scumm engine because different Lucas adventures used different versions of SCUMM, incompatible between each other. In the end they aren't running the SCUMM bytecode over a single engine, but over multiple different implemented engines (that obviously share parts in common).
They are including the different Scumm engines (and Sierra's, and others) and then using the resource files. I think that is not so different from what I want.

My greater goal (although ridiculously far fetched at this stage) is to develop a general X-Wing Saga-like engine, able to run all games in the series in the same way that ScummVM allows to run multiple graphical adventures under the same main native program.
We all know the differences between the different games in this series (laser speed, turning physics, whether missions have multiple sectors or just one, multiplayer support...). I'd say, it could be possible to reduce those differences to just some background settings, and alter the mission parser, and the "concourse" part for every game. Or would any of you see imposible that LucasArts would have come out with a single pack of the whole series, where we could run one single game and there choose which career we want to take: Rebel or Imperial. Then the Balance of Power campagins would be just another Tour of Duty or TIE Fighter campaign. I don't see any reason why that couldn't happen. We could have had that in XWA.

I know Freespace 2 is open source and considered the best spacesim engine ever, and I admire the Fate of the Galaxy total conversion. However, believe it or not, I consider that their project is much more laborious than what I propose.
A flight engine is not a game. I mean, we would not be playing this game if the missions would have been crap, the music would have been the red book one from the beginning, and there would not be any immersion building cutscenes to keep our interest picked. The flight engine in XvT is better than in the two previous games, and we all agree that XvT was the low point in the series, right? Then what failed? The design of everything else.

In this project we have the design ready, the gameplay ready, the missions are ready to parsed, the overall story is done. Lucasarts did all of that for us already. The art assets and music, even when with room for improvement to adapt them to HD, are still there for us to grab and use if we feel in a hurry and feel okay with a "faithful" look.

Star Citizen and Elite Dangerous, like Vegastrike, are games more appropiate for attempting a Privateer or Wing Commander total conversion than an X-Wing one. They are also much more ambitious. And that means much more work to get anything finished.
I'm not planning to reinvent X-Wing. I like X-Wing how it is. Not planning to add 100 new ships, 100 new missions, and make up new storylines. The plan is to just give it a new engine that mimicks the old one to the detail to keep the feel of X-Wing but lifting all restrictions that time had over the original one, like resolution, multiplayer, and some minor refinements.

It sounds all too complicated for me but it seems you have given this a lot of thought.
By the way don't forget to include headtracking support and you will get a donation from me. :^)

Good luck though.

This sounds just too good to be true, but if you're able to pull this off, it'll be incredible!

And as the proud owner of an Oculus Rift DK2, it would blow if away, if you'd manage to include VR head tracking for the 3D cockpits (which seems to be rather straightforward in Unity, since by now it's officially supporting the Rift).

Well, OpenXcom is my absolute inspiration for this. It's exactly the model I am going after: a little program that uses the resources of the orginal. The bigger changes I proposed are just wishful thinking.
If I didn't use a name like OpenXwing it's because I don't want to use any trademarked term at all. It's Disney that is behind the game now, famous for their hordes of lawyers and their Cease and Desist letters.

A custom built engine gives you more control over everything, but that is both a good and a bad thing.
The advantage of Unity is that it's already there, ready to be used. It's multiplatform from scratch, and lets me focus on the game stuff rather than having to spend one or two years developing a game engine first.
If tomorrow Microsoft or Apple bring out an update that breaks the game, I would need to keep catching up after them. With Unity, there is already a bunch of engineers keeping all that under control.
Also, Unity can be used to develop a prototype fast. If it works very well, then you can keep it. If it doesn't, then you can think on developing your own. Well, that would be really good, indeed. That would save us work with having to create models for the ships.
I know the W95 versions uses OPTs just like XvT does. But where are the DOS models stored, and what format does it use, I have no idea.
Any insight? Yes, I didn't like that resolution switching between areas in TIE Fighter. It also happens in XvT if you select a different resolution for the flight engine. Very ugly.
The problem is that the aspect ratio would be arbitrary. That is no problem (much) for the inflight engine. But the concourse uses art that is probably 4:3. I guess some vertical black bars will need to be used if we don't want to see a squished game on a 16:9 panoramic monitor.

Sorry, I didn't see the lower part of your response. About your first and last point, TIE Fighter's engine basically is X-Wing's engine refined with some obvious good ideas had during the development. As I see it, the only reason why X-Wing doesn't have those things (useful CMD, match speed, current target's attacker, who is attacking me, Z key, etc) is just that the developers didn't think on it.
As such, I believe that the older brother should be given some facelift and add most of the features of the younger one. Not only those you mentioned. Also the better map, warhead selection, pilot autobackup, etc.
The missions could be left as they are, as X-Wing particular physics for lasers if changing it would mean breaking some balanced. But as I said, those feel more like little configurable tweaks than serious incompatibilities.

About custom stuff, obviously yes. No more meddling with files, hexediting or replacing. Everything must be extensible from the beginning. If this really gets traction, I would say the game should include a mission editor itself, and stop relying on old 16bit third party mission editors.

I'm already fond of projects like OpenRA, DarkXL or the OpenGL reimplementation of Elite II and III.
I find it cool that you plan something similar for X-Wing.
Good luck with that. I'm looking forward to see what you'll come up with. :)

The solution to that is to use a library to handle OS specific stuff, like SDL, just like OpenXcom. In fact, if this project used SDL, it'd already have a leg up on OpenXcom and other projects like DOSBOX and dxx-Rebirth and because XWVM would be new enough to be able to start with SDL2! I am pretty sure it's the .SHIP files in RESOURCE/SPECIES.LFD. Beyond that I do not know what the format is. But it can be researched if enough talented people are interested in the project. Just like OpenXcom, every format will have to be fully reverse-engineered. Chances are, though, it probably has a lot in common with other LucasArts model files (probably even OPT) so it might not be a stretch.

Which reminds me - for modding support, modern image formats such as .pngs would be great, OpenXcom also does this. It allows people to mod the game without having to use the obscure old file formats. Of course it would be supported in conjunction with the originals.

I bet they're much simpler to decipher than OPT, since there is no UV mapping information. Probably nothing more than submodels, vertex coordinates, polys, and what color the poly is.

In the end, I think the art and sound assets are the only ones that REALLY need to be reverse engineered, and most of it already is. Things like stat tables, cockpit layouts, concourse layouts, etc. might be better off getting a new human readable format instead. and just remade by hand.

Beyond that, I think most of the unknown fileformats by now are all for TIE Fighter CD (XACT graphic format, WRK cutscene video format, possibly shared by XvT), I think X-Wing is pretty much totally figured out. My thoughts on that would be to have two aspect ratio options: one for concourse, and one for inflight. Since people might be fine with stretched cockpits but not stretched concourses. The concourse aspect ratio letterboxing would be on be default, while the inflight letterboxing would be off by default, since it would limit game view.

As for texture scaling, I think 320x200 2D assets should be pixel doubled first before scaling further, because it almost never looks good if it's linear filtered. 640x480 assets however are more suited to linear scaling and should not be prescaled. All should be user configurable, of course, but those would be sensible default values, I think.

--------------------------

Anyway, my main concern for using Unity is that I have no idea how it works with external libraries or not. What if we want to bundle MUNT or an OPL emulator to emulate the Adlib and MT-32 music? In fact, it would be required if it is ever to support the floppy version of X-Wing because pretty much all of its sounds are soundcard instruments and not digital samples.

Also, I just saw this: That would be nice, but what I would REALLY love to see is a combination of both - specific, handmade instruments rendered into the 3D model itself using render-to-texture gauges and displays. That way it wouldn't have to use a generic 2D overlay like XWA does! Plus it would open the door for things like TrackIR and Oculus Rift without straying from the original aesthetic, especially if cockpit models were made using the DOS cockpit art as a texture base. No offense, but I think the XWAUP cockpits are kinda ugly compared to the DOS ones...

I agree it would be nice to use SDL2. Alas, I'm not that well versed on that library. Last time I played with SDL was back at the University, in C. And it has been more than 10 years since.
The problem is that I'm not a graphical programmer. I don't have the training for wrapping my head around all that about culling planes, shading and all the algebra involved. Or I had it, but I have forgotten since. :) In the end, I would spend several months of my scarce free time just to end with something that looks like crap next to any of the readily available 3D frameworks. Please, have a look at this short discussion to understand my point.
I want this project to become a reality and I want it sooner than later.

Most of the work will be, anyway, figuring out the original data format (can we say reverse-engineering in this forum without summoning industrial property lawyers?), and the rules of the movement and AI behavior of the ships. Once that knowledge base is set, implementing something that runs it is almost trivial.

For example, I have been researching the kind of AI that the spaceships seem to use for flying around and avoiding obstacles or crashing against each other, or stay in formations. I think I have it cornered as Potential Fields movement planning. I just need to refine the constants and the exceptions. But the results I am getting in the Unity prototype are quite convincing compared to how a battles used to look like when witnessed from the XvT realtime map.
My point is that, once this is figured out, implementing it in Unity or SDL is just the last step, and not necessarily the one that is most time consuming.
I'll go for Unity now, because it's what I know and feel confortable with. I have built some momentum with it that would be halted if I need to go back to the drawing board to implement a 3D framework from scratch. Later on, I would be totally open to transfer all of it to SDL when the rest of the work is done, though.

Reverse engineering isn't something that can be done overnight. That is why I have started by implementing the flight engine, rather than the whole menu/concourse system. The .XWI, and .PLT file formats, and stats for the ships were found out in the 90s. That has allowed me to build some barebones flight sim with stuff that knows what to do and where to go. Far away from playable, but something at least.

I think it's important to flow around roadblocks instead of trying to drill thru them. I don't know the SHIP format? Then I will just paste there a free 3d model from Devianart until something better is found later. One good thing of Unity is that it can use many formats without having to transform everything to a particular one. I can use 3DS, Collada, Blender, or many other models, and use many kinds of audio and image formats without any problem. That will allow us to reach a prototype faster.

X-Wing is now, TIE Fighter is in the future. TIE Fighter already has 640x480 resolution and so it's in less urgent need for a facelift. Also, TIE Fighter is in all aspects an evolution of X-Wing, with much more complex... well... everything. That's why I start with X-Wing.

The iMuse system seems to be one of the most complex assets. We could use the General MIDI music files and figure out what to do with them. I am not so sure what would be the point of emulating Adlib or MT-32 or the GUS at this point. I have listened to the X-Wing soundtrack running over those different hardware sound cards and even when some of them sound better than other, it all sounds quite, let's say, MIDI to modern ears.

It would be awesome if we could get the guy that made the Reorchestrated TIE Fighter soundtracks to do the same with every iMuse chunk of music, then glue them togheter at real time as the iMuse system did. If that would be possible, I could live perfectly without Adlib, MT-32 or GUS emulation.

I know that the iMuse system was more complex than that. It could silence some instruments or make them sound louder, or alter the tempo and all of that (It easy to spot on the X-Wing main concourse, the soundtrack adds a trumpet if you keep any of the shuttle gates open).
That would be impossible to do with digital recordings. But it's a better compromise than nothing or the awful Redbook music.

Well, I never really saw the point of the 3D cockpit in XWA. In XW, TF and XvT, the cockpits were actually showing your instruments. But in XWA, they are there just to limit your field of view. You had the virtual cockpit instruments around the screen, so the real cockpit actually had no value.
The XWAU ones were prettier but still useless.

The problem of the 2D bitmap cockpits is that some people feel constricted by them. A friend couldn't play these games, he told me, because the 2D X-Wing cockpit makes him feel like if he would be a flying Hannibal Lecter's head with his famous asylum psycho mask on. :D Also it would be impossible to use Oculus or TrackIR with those.
What you say about mixing 3D surfaces with the 2D textures intrigues me.
I guess the idea would be to build the interiors in 3D, then UVmapping them so that the original 2D cockpits are laid over them like a decal?
Maybe the 640x480 cockpits from the W95 or Mac editions would look okay. But what about the 320x200 cockpits from the DOS editions?
Damn, I didn't want to start meddling with Blender this early but now you have picked my interest. :D I suspect it will look kind of retro-3D. Like Fez or a Nintendo DS game.
Then I need to figure out where the cockpit art is stored and what palette it uses.
If you have any information about it, please share it with me. :)

Well, I meant more like literal 3D versions of the DOS cockpits, trying to fit to the shape shown in the art. Not just projecting it onto a flat surface! But the most important thing to get out of it render texture gauges, so that instead of the XWA HUD overlay, you can see all the laser meters and shield bars and all that stuff animate on the model itself. I think that feeling of interactivity is the difference between feeling like you're flying a cool spaceship VS playing pretend in a cardboard box

All the cockpits are in the CP folder. Here's a guide for getting/editing the cockpit graphics for XvT, but I imagine X-Wing is the same due to the same filenames and file extensions. Worst case scenario is you can just copy the art from screenshots for now! Actually, it wouldn't be too difficult. All it would need would that specific trumpet track to be split off into its own file and played alongside the main part accordingly. What would be nearly impossible is something I noticed in one of the inflight tracks:

If you listen closely during one of the inflight 'waiting' tracks (not in dogfight) there's a trumpet part that plays more and more rapid notes the closer enemies get until they're close enough to trigger the dogfight music. It wouldn't be impossible, it could at least be simplified to 4 states with their respective separate trumpet tracks, but there's a certain point where that becomes more trouble than it's worth.

Anyway, even if all that subtle stuff was gone, frankly just having dynamic music at all would indeed be refreshing.

The first step to getting re-recorded music would be to get the .midi files available to give to anybody willing to do it. I think that's what's stopped people so far. But, I've been experimenting and I've got all the inflight music segments into regular MIDI files, along with notes on what I think are their cues, and a couple loop tables(see below) I've extracted so far. Not totally clean, they all have a bar of silence at the beginning. I am not sure if that is by design or not.

I'll comb through the concourse music and see if there's any other specific subtleties other than the 3 big doors playing different songs in the background. Beyond those subtleties, the iMuse system is fairly simple. I won't post the whole thing but in BFLIGHT.OVL (big binary file that stores a ton of hardcoded values for the game) there are several tables that dictate how song segments are queued.

For example, take the dogfighting music, which is broken up into MIDI files DG01, DG02, DG11, DG12, DG13, DG21, and so on... a few dozen. Anyway, the table is laid out like this (not exactly, this is an example. I can go dig up the real tables later)

DG01 - DG02 DG12 DG21
DG02 - DG01 DG22 DG32 DG03
DG12 - DG02 DG03

What it means is: When DG01 is played by the game (cued by being within a certain distance of hostiles, I think) it will then randomly choose between DG02, DG12, or DG21 to play next. If it picks DG02, then it will randomly play DG01, DG22, DG32, and DG03 next, and so on. This will continue forever until interrupted by another cue (going back to calm tracks, shot down an enemy, ship arrived, got an enemy in sights, and so on...) The real trick is just figuring out which events specifically cause which track "group" to play, and then recreating that in our own engine.

So you have actually dug into that already? Wow!
Well, I will have to write down all of this later but there are a bunch of event that quite evident, like the ones you mentioned.
Out of my mind, right now, I could say:
In flight themes:
- Quiet waiting theme.
- Enemies withing 6 km battle theme.
- Enemies around you battle theme.
- Dogfighting (enemy on screen, at close distance) battle theme.
- Death Star surface battle theme.
Events tunes:
- Player kills enemy.
- Allied ship down.
- Neutral ship down.
- Allied ship comes into the sector.
- Imperial ship comes into the sector.
- Neutral ship comes into the sector.

I cannot think on any else right now. It seems to be much simpler than TIE Fighter's, at least.

By reading the patent document on iMuse, it seems that the best approach would be to keep a finite state automaton that models these different tunes as states, and the different in-game events as transitions. When several transitions are eligible for the same event, then the automaton picks one at random.
So, the flight sim should feed the automaton with events, maybe throttling them down to a certain limit so that not too many transitions are happening if several different events happen at the same time (two enemies down and incoming of a new allied ship at the same time should produce one single transition, instead of 3).

Here's the link to the extraxted inflight MIDI files and my notes There are only a couple bits I can't quite figure out what causes them. Read through my text files and see if you have any ideas that I'm missing. For example, "FL" seems like the obvious failure track, yet I often seem to hear it even when the mission is going just fine.

Also I haven't gleaned every loop table yet, there's one for every prefix and most of them are quite long...

As for picking a transition at random, I think the game might have a priority system or some kind of queue for them. I'll have to play around with that later and see how it happens. It could just be that anything that happens while an "event" bit is playing is simply ignored