It seems that you're using an outdated browser. Some things may not work as they should (or don't work at all).
We suggest you upgrade newer and better browser like: Chrome, Firefox, Internet Explorer or Opera

×
Community
General discussionOffline Game Installers - why use C:\Temp?(54 posts)(54 posts)
(54 posts)
Pages:
1
2
3
4
This is my favourite topic
Posted April 01, 2021
low rated
avatar
coffeecup: The main reason why they are doing this is to "obfuscate" the installer data and to make it harder for everyone else (including people who legally obtained the games, but just use Linux) to get to the data inside, but this is in vain. Innosetup now has a special --gog switch specific to deal with these hacked up data.
I'm sorry, but that makes no sense.
What is it exactly that they are obfuscating?
And I think you are mixing up InnoSetup with InnoExtract.
And that --gog switch does not do what you are thinking it does.

If you don't want to install the game and just want to extract the data, then you can do that, --gog switch or not, though that might be needed for the content of any BIN files associated to the EXE file. Though the BIN files can be extracted with UnRAR or 7-Zip in any case.

In reality though, just installing the game extracts everything.

I'm not sure why a TEMP location is needed, but perhaps because of further extractions. Otherwise selective removal would be all that was needed for unwanted content (language files, etc).
Posted April 01, 2021
high rated
I do know what I am saying and doing.

avatar
Timboli: I'm sorry, but that makes no sense.
What is it exactly that they are obfuscating?
GOG throws all files into a blender, so if you unpack it with innoextract (without) the special switch or with innounp, you will get a big fat {tmp} folder with many many itty bitty chunks, but not the game itself. In case of Terraria 1.4.2.0 you will end with about 14000 chunks (files) there.

These chunks will be reassembled during installation by a special compiled Pascal script (I nearly reverse engineered it fully out of frustration, until I got the memo with the switch). This is also the main reason of the disk thrashing, disk wearing and delay in the installation, because this compiled script has to reassemble the files.

There is a reason, why this switch exists with innoextract, it does that transparently for you.

avatar
Timboli: And I think you are mixing up InnoSetup with InnoExtract.
No. InnoSetup is the installer system used by GOG; InnoExtract can extract the data and support files from the executable installer; but discards the embedded script. Innounp cannot deal with the provided garbage, but does not discard the embedded script and all the other "invisible" stuff if you want to have it.

avatar
Timboli: And that --gog switch does not do what you are thinking it does.

If you don't want to install the game and just want to extract the data, then you can do that, --gog switch or not, though that might be needed for the content of any BIN files associated to the EXE file. Though the BIN files can be extracted with UnRAR or 7-Zip in any case.
I may quote the homepage of InnoExtract (I said the same, but in more layman's terms:
-- snip --
Other newer GOG.com installers don't include the raw files directly but instead store them in GOG Galaxy format: split into small parts which are then individually compressed. These files are named after their MD5 hash and stored in the tmp directory, for example "tmp/ab/d7/abd72c0dddc45f2ce6098ce3a286066a". innoextract 1.7 or newer will automatically re-assemble these parts and extract the original files unless the --no-gog-galaxy option is used.
-- snip --

avatar
Timboli: In reality though, just installing the game extracts everything.
Yes, if you are using Windows.

If you are using Linux, you really don't wanna fire up a WINE environment or a full-blown VM just to install the game. Prior to the addition to this nonsense, it was true, that you could decompress the installer with RAR or 7z, but if you do this with the "newer" ones, you end with a mountain of chunks.


For reference with Terraria 1.4.2.0
installed size: 494 MB
files: ~ 13500

GOG Original installer
Size: 418 MB
files: ~ 13000 files (tmp folder is alone 13008)
installation time (excluded redistributable): 2m 22s

GOG rebuilt installer
- adjusted the setup.iss to accomodate the original files and threw out the custom code by GOG
- lzma2 solid maximum compression
Size: 395 MB
files: same as installation, plus some script files and picture files
installation time (excluded redistributable): 42s

Granted, most time the original installer is busy decompressing chunks and reassembling them for not really anything valuable.
Posted April 02, 2021
avatar
coffeecup: I do know what I am saying and doing.
My apologies, as it does indeed look like you do.
I've never gone to the effort you clearly have, just seen things through the eyes of InnoExtract, 7-Zip and UnRAR.
Thanks for the detailed information.

avatar
coffeecup: GOG throws all files into a blender, so if you unpack it with innoextract (without) the special switch or with innounp, you will get a big fat {tmp} folder with many many itty bitty chunks, but not the game itself. In case of Terraria 1.4.2.0 you will end with about 14000 chunks (files) there.

These chunks will be reassembled during installation by a special compiled Pascal script (I nearly reverse engineered it fully out of frustration, until I got the memo with the switch). This is also the main reason of the disk thrashing, disk wearing and delay in the installation, because this compiled script has to reassemble the files.
Why I wonder, does GOG do this?
It's not very user friendly or open ... and it has more in common with DRM behavior.

avatar
coffeecup: There is a reason, why this switch exists with innoextract, it does that transparently for you.
Thank GOG for that.

avatar
coffeecup: No. InnoSetup is the installer system used by GOG; InnoExtract can extract the data and support files from the executable installer; but discards the embedded script. Innounp cannot deal with the provided garbage, but does not discard the embedded script and all the other "invisible" stuff if you want to have it.
Okay, I knew some of that. As they say, sometimes a little bit of knowledge can be dangerous, and lead to assumptions ... oops.

avatar
coffeecup: I may quote the homepage of InnoExtract (I said the same, but in more layman's terms:
-- snip --
Other newer GOG.com installers don't include the raw files directly but instead store them in GOG Galaxy format: split into small parts which are then individually compressed. These files are named after their MD5 hash and stored in the tmp directory, for example "tmp/ab/d7/abd72c0dddc45f2ce6098ce3a286066a". innoextract 1.7 or newer will automatically re-assemble these parts and extract the original files unless the --no-gog-galaxy option is used.
-- snip --
Obfuscated indeed.

avatar
coffeecup: Yes, if you are using Windows.

If you are using Linux, you really don't wanna fire up a WINE environment or a full-blown VM just to install the game. Prior to the addition to this nonsense, it was true, that you could decompress the installer with RAR or 7z, but if you do this with the "newer" ones, you end with a mountain of chunks.

For reference with Terraria 1.4.2.0
installed size: 494 MB
files: ~ 13500

GOG Original installer
Size: 418 MB
files: ~ 13000 files (tmp folder is alone 13008)
installation time (excluded redistributable): 2m 22s

GOG rebuilt installer
- adjusted the setup.iss to accomodate the original files and threw out the custom code by GOG
- lzma2 solid maximum compression
Size: 395 MB
files: same as installation, plus some script files and picture files
installation time (excluded redistributable): 42s

Granted, most time the original installer is busy decompressing chunks and reassembling them for not really anything valuable.
Great information. My apology once again, and thanks for setting me straight ... and politely. :)

Am I right in presuming, that the GOG rebuilt installer is actually a rebuilt GOG installer by yourself?
Posted April 02, 2021
avatar
Timboli: Why I wonder, does GOG do this?
It's not very user friendly or open ... and it has more in common with DRM behavior.
Who knows. My personal guess is that the GOG Galaxy game data on their content servers is also chunked in the same fashion, suggested by the InnoExtract page. So, it is easier in their build queue for the offline installers to put this data and a custom decoder script into the installer and call it da day. This is just an assumption.

avatar
Timboli: Am I right in presuming, that the GOG rebuilt installer is actually a rebuilt GOG installer by yourself?
Yes, I rebuilt it with the extracted decoded data and installation script..

If you want to try it for yourself, extract a GOG installer with innoextract for the game files and innounp for the script and rebuild it with innosetup. You will have to remove the chunk files, add the real game files and adjust some things which depend on the removed compiled custom script blob.

The interface wizard will be bog standard innosetup, not GOG, for the testing I did not bother to recreate it.

As already mentioned, I cannot and will not redistribute any generated results from that, sorry.
Post edited April 02, 2021 by coffeecup
Posted April 02, 2021
avatar
Timboli: Why I wonder, does GOG do this?
It's not very user friendly or open ... and it has more in common with DRM behavior.
avatar
coffeecup: Who knows. My personal guess is that the GOG Galaxy game data on their content servers is also chunked in the same fashion, suggested by the InnoExtract page. So, it is easier in their build queue for the offline installers to put this data and a custom decoder script into the installer and call it da day. This is just an assumption.
Makes sense if the offline installers really are generated automatically nowadays, using the Galaxy files.

avatar
coffeecup: Yes, I rebuilt it with the extracted decoded data and installation script..

If you want to try it for yourself, extract a GOG installer with innoextract for the game files and innounp for the script and rebuild it with innosetup. You will have to remove the chunk files, add the real game files and adjust some things which depend on the removed compiled custom script blob.

The interface wizard will be bog standard innosetup, not GOG, for the testing I did not bother to recreate it.
Can that process be automated with a script, or will there always be manual steps (the bolded part above)? While I can live with the current installers, I guess it would be nice to rebuild the stabilized (= not getting updates anymore) installers at some point, if the new installers really are both smaller and faster.
Post edited April 02, 2021 by timppu
Posted April 02, 2021
avatar
coffeecup: If you want to try it for yourself, extract a GOG installer with innoextract for the game files and innounp for the script and rebuild it with innosetup. You will have to remove the chunk files, add the real game files and adjust some things which depend on the removed compiled custom script blob.
+1 for testing this stuff. I think the efficiency of offline installers can be improved a lot for both speed & space. Eg, Prey's installers are 24.5GB but that includes all languages. Install the game, delete the unwanted language files from the game folder, 7zip it up and the "installer" size instantly shrinks to just 16.8GB. Cyberpunk 2077 is apparently the worst offender resulting in a 100GB installer to install a 60GB game for same reason (every language is included instead of sensibly being split). Whether this is accidental or done deliberately to discourage the use of offline installers and make Galaxy look relatively better remains to be seen...

As for recompiling vs effort, I think it depends on the game. 90% of DRM-Free games here (especially newer ones) are "portable" in that simply copying the game folder across to a PC that never had it installed would work (eg, the registry entries that Unity Engine saves to HKEY_CURRENT_USER\Software\ are created by the game on first run rather than the installer). I tend to divide registry entries into "Internal" (required by the game, eg, Operation Flashpoint's 'serial key') vs "External" (stuff usually found under HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\GOG.com that's unused by the games themselves and exists mostly for Galaxy metadata or perhaps 3rd party utilities like GameSave Manager looking up just the "path" value for old games whose saves are placed in the install folder). So if you don't use Galaxy at all, then it might not even be worth going to the extent of recreating an InnoSetup but instead just zipping them up and using a batch file to automate the unzip and add the (very few actually required) registry entries.

Either way, removing unneeded language packs alone saves literally dozens of GB's (and for highly moddable games, zipping it up with mods pre-installed saves a LOT of time when reinstalling for games like Morrowind & Oblivion). Same goes for games that use source-ports, eg, I have Doom 1-2, Heretic, Hexen, Hedon, Strife & Chex Quest + pre-configured GZDoom all backed up as one portable 7z file. "Re-Installing" that = everything gets reinstalled including custom key-bindings, settings, HD texture packs, other WAD files, etc, saving much more time than just faster extraction time alone.
Posted April 02, 2021
avatar
AB2012: So if you don't use Galaxy at all, then it might not even be worth going to the extent of recreating an InnoSetup but instead just zipping them up and using a batch file to automate the unzip and add the (very few actually required) registry entries.
You mean by installing each game and then compressing the installation?

I'm trying to figure out which would be the way that could be automated with a suitable script, just telling it to go over your whole local installer library and converting each and every installer to some better format, one by one. But I guess that can be quite hard to achieve if it depends on the game if there are certain (language) files to remove etc.

Anyway, I am thinking this is more relevant for the future, 10-20 years from now when gog.com doesn't necessarily exist anymore, and I am thinking it would be nice to convert my whole local GOG game installer library to a smaller and more efficient format. But maybe then we have already moved to 500 petabyte SSSDs (Super-SSD format) where I couldn't care rats ass if I can shave off a few terabytes from the GOG installers... Could be still nice to just try it out.
Post edited April 02, 2021 by timppu
Posted April 02, 2021
Am afraid they are not going to give a FCK. It’s just another thing on the huge list of issues. The way to move people away from piracy is to provide a product which is superior. However gog have simply followed all the other stores down the road of more money less use. Even something simple like leaving older versions available is impossible:
https://www.gog.com/forum/general/provide_a_full_and_complete_changelogged_download_system

You can take cyberpunk (although why anyone would want it is beyond me), gog installer is huge 100gb plus, and wasn’t available for offline preorders. Pirates had it pre order, language packs split out, 30% the size of the gog one. Why would anyone bother with an inferior product? It’s the same thing for drm, drm destroys the quality of the product for its users, so they go for a better product. It used to be different, the offline installers were once something worth paying for, now they are outdated, poorly made attempts to push galaxy.
Posted April 02, 2021
Preserving a game includes having everything from it, also other "unwanted" languages.

I made a decision to base my games primarily on GOG installers, mostly due that I have not to deal with cracks or other stuff, supplemented by my original disc images where I know that GOG omitted the audio tracks (Realms of Arkania series, that one French 3D Diablo Clone, Kyrandia games), provided just a floppy version or the games GOG sells, are ripped from a games collection (Might & Magic series, Space Quest series, Kings Quest series, Quest for Glory series, Leisure Suit Larry games, their (now owned) SSI games like Krynn saga, Pools saga, Ravenloft games, Ultima games). Lastly, there are the zipped games.

I would not do compression with 7Zip, as this archive format - unlike RAR and ZIP - is still in state of flux. You may encounter the problem that it will be undecipherable in some time of the future, because the authors tweaked it too much.

My educated guess is that GOG internally already has said goodbye to offline installers and the current ones are just a byproduct of how they do storage with Galaxy, but just don't admit it in order not endanger their last promise they did not break. This is just an educated guess which crossed my mind after reading the section "GOG.com installers" on the InnoExtract page https://constexpr.org/innoextract/.

-- snip --
Other newer GOG.com installers don't include the raw files directly but instead store them in GOG Galaxy format: split into small parts which are then individually compressed. These files are named after their MD5 hash and stored in the tmp directory, for example "tmp/ab/d7/abd72c0dddc45f2ce6098ce3a286066a". innoextract 1.7 or newer will automatically re-assemble these parts and extract the original files unless the --no-gog-galaxy option is used.
-- snip --

I hope my gut feeling is wrong.
Posted April 02, 2021
avatar
coffeecup: Preserving a game includes having everything from it, also other "unwanted" languages.
Depends to whom you are preserving them: to yourself (and possibly your family, kids etc.), or to the world.

I am mainly interested just in preserving them to myself, and my closest family. I have never been into this "it must have all possible languages, including swahili and ancient hutu-indian languages!". Learn English, people! The only languages (audio) I am interested in my games are:

- English

- If the game is e.g. Japanese, maybe also that... but not a biggie.

- For some children games, maybe Finnish would be nice (for the kids, you know?), but hardly any games have a Finnish language option, and at least that is a good incentive for the kids to learn English! I learned English a lot from games when I was a kid.

For instance, as I am now downloading my GOG game installers with the gogrepoc.py tool, I've restricted the downloads only to English Windows versions of games (excluding all non-English versions of games, and also Mac/Linux versions). Preserving also the Linux versions might be in my plans at some point, but if most of the "Linux versions" are merely the Windows version running in preconfigured WINE, I might just as well save just the Windows version, and set it up with WINE myself.

avatar
coffeecup: My educated guess is that GOG internally already has said goodbye to offline installers and the current ones are just a byproduct of how they do storage with Galaxy, but just don't admit it in order not endanger their last promise they did not break. This is just an educated guess which crossed my mind after reading the section "GOG.com installers" on the InnoExtract page https://constexpr.org/innoextract/.
To me it sounds more logical that they have changed the offline installer format to be "Galaxy-like" because that makes it easier for them to automate the creation of the offline installers, straight from the "Galaxy files".

So when a GOG game is updated (or added), those Galaxy-downloadable, md5-checksummed partial files are generated and stored to the download servers. And then the offline installer creation kicks in, creating the offline installer from those files which are already available on the server for Galaxy users.

Or then it is something else, how would I know for sure...
Post edited April 02, 2021 by timppu
Posted April 02, 2021
avatar
timppu: You mean by installing each game and then compressing the installation?
Yes. Like how some people do that for games on Steam / Epic that have no DRM and are "portable". I've done this with around 300x DOS / ScummVM games (some are on GOG, many others came on disc) and the whole lot batch installs in 90 seconds with one double click of an install_all.bat file using one central DOSBox / ScummVM installation (vs hours and thousands of clicks manually installing each game one at a time via 300x InnoSetup installers). Same goes with making "portable" zips of source-ports, eg, Doom 1-2, Heretic, Hexen, Strife, Hedon + GZDoom. Aside from being much faster to install, it also preserves stuff like custom settings, key-bindings, etc, between installs.

avatar
coffeecup: Preserving a game includes having everything from it, also other "unwanted" languages.
As timppu said, it depends on the objective. If I were running a game preservation website, I wouldn't delete anything. Nor would I do stuff like delete the original .exe just because they run under ScummVM (GOG actually does this for some games so certain installers already suffer from this anyway). But if I'm just preserving my own files for myself, I'm quite happy to lose stuff I'll never use like 12x foreign languages in WAV format to debloat them somewhat. Some games here can be literally halved in size.

avatar
coffeecup: would not do compression with 7Zip, as this archive format - unlike RAR and ZIP - is still in state of flux. You may encounter the problem that it will be undecipherable in some time of the future, because the authors tweaked it too much.
Whatever format gets used that problem is easily solved just by backing up a copy of whatever version was used to create them. I mean WinRAR & 7zip combined take up less than 5MB space.

avatar
coffeecup: My educated guess is that GOG internally already has said goodbye to offline installers and the current ones are just a byproduct of how they do storage with Galaxy, but just don't admit it in order not endanger their last promise they did not break. This is just an educated guess which crossed my mind after reading the section "GOG.com installers" on the InnoExtract page https://constexpr.org/innoextract/. I hope my gut feeling is wrong.
It would definitely be nice to be wrong about this stuff, but that's been my gut-feeling too ever since the offline installers were Galaxified.
Posted April 02, 2021
Standard practice
Posted April 03, 2021
avatar
AB2012: So if you don't use Galaxy at all, then it might not even be worth going to the extent of recreating an InnoSetup but instead just zipping them up and using a batch file to automate the unzip and add the (very few actually required) registry entries.
avatar
timppu: You mean by installing each game and then compressing the installation?
Off the top of my head, the only possible issue I can see with this could be that first run Registry entry that was mentioned (for games that have that). Does the game check again for that after initial creation, and recreate if needed on a different PC. Or would you need to install again from scratch. Obviously there is a flag somewhere that is checked each time the game starts.

EDIT - Well not the only issue maybe, as there could be other dependencies installed outside the game folder.

The safer bet probably, would be to do as coffeecup suggested.

avatar
coffeecup: Preserving a game includes having everything from it, also other "unwanted" languages.
I guess that depends on who you are preserving it for, or likely to be (copyright issue aside).

avatar
coffeecup: My educated guess is that GOG internally already has said goodbye to offline installers and the current ones are just a byproduct of how they do storage with Galaxy, but just don't admit it in order not endanger their last promise they did not break. This is just an educated guess which crossed my mind after reading the section "GOG.com installers" on the InnoExtract page https://constexpr.org/innoextract/.

I hope my gut feeling is wrong.
I do too, but it certainly is troubling.
Post edited April 03, 2021 by Timboli
Posted April 19, 2021
Sorry to jump in the discussion like this but isn't there a sort of feature tracker for GoG?

Can't we start a request for them to look at this? Prefer to use the GoG Installers instead of GoG Galaxy (which I don't even know if it uses a different solution to be honest) and I'd like not to have to throw my SSDs in the bin earlier than needed because of this absurdly ill-engineered solution that they are using.
Posted April 19, 2021
avatar
thecrius: Sorry to jump in the discussion like this but isn't there a sort of feature tracker for GoG?

Can't we start a request for them to look at this? Prefer to use the GoG Installers instead of GoG Galaxy (which I don't even know if it uses a different solution to be honest) and I'd like not to have to throw my SSDs in the bin earlier than needed because of this absurdly ill-engineered solution that they are using.
Unless you are some kind of game tester who constantly installs and removes GOG games every day, several times a day, the effect of GOG installers decompressing installer files twice in the process has a negligible effect to your SSD's lifespan.

I think e.g. a swap file or your web browser cache and offline data on your SSD causes several orders of magnitude bigger, constant, stress to your SSD, and even their effect is probably quite negligible.

Plus, as long as you don't own a M1 Mac (where you can't replace the SSD, poor bastards), I don't think it matters that much either way: when your SSD finally starts dying 10 years from now, you just replace it, if you haven't replaced it to a bigger one already before, or even replaced the whole computer.

Either way, I don't really see GOG changing the way the offline installers work right now. The current method probably offers them so big advantages, like being able to automate how offline installers are made from the existing "Galaxy files", instead of having to unpack and repack them, just so that they would be optimized for offline installer use.

Anyway, in some other discussion someone else claimed that some pirate lady has been able to re-compress GOG installers to something like 30% of the size of the original GOG installers. I'd just like to hear about the technology there, how does she achieve that, or does she leave some data out? Or was it just a rumor and hearsay, not true at all?

I recall from the optical drive times that there was some kind of "ecm tool" that would rip data out of ISO images that it considered "unneeded", like cyclic-redundancy-check stuff, and sometimes being able to shring the ISO images quite much. Is this something similar, or?

EDIT: So this was or is the ecm tool: https://archive.org/details/ECMToolsV1.0
Post edited April 19, 2021 by timppu
Pages:
1
2
3
4
This is my favourite topic
Community
General discussionOffline Game Installers - why use C:\Temp?(54 posts)(54 posts)
(54 posts)