Thank you Sude, I’m adding the .torrent file to my server right now and updating the opening post in a couple minutes.

Thanks. Creating the delta, despite the messages it showed, hasn't changed the sums.
I'm testing now creating the new file form the old installer+delta.

xdelta3 -d -s gog_pillars_of_eternity_1.0.0.1.tar.gz gog_pillars_of_eternity_1.1.0.2.tar.gz.delta gog_pillars_of_eternity_1.1.0.2.tar.gz

After it's finished, I'll uncompress it and check if it works properly.

Interesting. I didn't think of applying a delta to archives, I thought they'd differ way too much. Applying it to unpacked files and then compressing would be my first approach.

It may be a bit more memory/cpu intensive, so it takes a bit, but the resulting diff is smaller.
Creating the new archive/recompressing, does indeed change the sums for the resulting file, so in the end it may have been useless.

xdelta3: externally compressed input: gzip -dc < gog_pillars_of_eternity_1.0.0.1.tar.gz
xdelta3: externally compressed input: gzip -dc < gog_pillars_of_eternity_1.1.0.2.tar.gz
xdelta3: WARNING: the encoder is automatically decompressing the input file;
xdelta3: WARNING: the decoder will automatically recompress the output file;
xdelta3: WARNING: this may result in different compressed data and checksums
xdelta3: WARNING: despite being identical data; if this is an issue, use -D
xdelta3: WARNING: to avoid decompression and/or use -R to avoid recompression
xdelta3: WARNING: and/or manually decompress the input file; if you know the
xdelta3: WARNING: compression settings that will produce identical output
xdelta3: WARNING: you may set those flags using the environment (e.g., GZIP=-9)

md5sum gog_pillars_of_eternity_1.1.0.2.tar.gz
8e240d11206c33f1e57ca06142136b4b gog_pillars_of_eternity_1.1.0.2.tar.gz

Maybe using the --rsyncable flag for gzip? Without knowing the original compression options, it might not be possible get the same sum for the archive.

I've just uncompressed it and I'll check if it works properly.

awesome job... so. does it work?

edit: can the mods make this a sticky please?

Seeing than no one sent me death threats yet, I assume it works quite well ;)
Anyway, I’m a bit like GOG on this one: taking the time to test my stuff before releasing it.

Bugs can (and will) happen with exotic setups on the game, but on a vanilla installation I’m sure at 99.99% it’ll work.

Yes works.

Have you considered using something like Mega (mega.co.nz) to host the files? Heck, you could even put the source for your tool on GitHub and provide the binary/package under the "Releases" tab.

May help for future cases.

Source of what tool?

To be precise: the script wrapping (in this case) rdiffdir or (otherwise) xdelta. Which would effectively be a couple of lines, but enough to justify putting a binary blob (the actual patch) up on GitHub.

Else just host the binary diff on Mega.

I'm not sure if GitHub allows hosting such blobs without explicit permission, which will have to be obtained from Obsidian I guess. So the best option is to use P2P until GOG will start offering such patches.

No way I’m going to host anything on this website (or any of its kind).
My reasons for this are well explained here:
https://mega.co.nz/#terms
and there:
https://mega.co.nz/#privacy

If you’re interested by the "sources" of this project, the whole process (three lines) is described by tonurics on the inXile forums:
https://forums.inxile-entertainment.com/viewtopic.php?f=34&amp;t=11081&amp;start=20#p143470

About the hosting of these patches, I think I’m going to stay with bittorrent for the most part, with a fallback direct download from my server for those who can’t use .torrent files. Bittorrent has the advantage to let anyone help me to share these patches just by staying as seeders after the download.
Of course, any offer for a mirror will be warmly welcomed!

Okay. I tried acouple of times more, but the xdelta3 generated gog_pillars_of_eternity_1.1.0.2.tar.gz
doesn't match the one downloaded from gog because of the whole decompression/compression.
Not even with something like GZIP="--rsyncable"

But, the file generated from the delta works fine, and all of it's contents match.
So, if anyone want's it to be able create the compressed archive, I'll try to uploadit somewhere, create a torrent, or something.

For security, someone should also independetly check the extracted files.
The easiest way I've found would be to generate and audit the hashes of all the files with something like hashdeep.
It's in the md5deep package in debian (an related I supposed) distros.

cd $GAMEDIR
hashdeep -rl . |& tee pillars_hashdeep_v1.1.0.2.txt

The whole process would be somethig like:

1. Download delta file

2. Generate gog_pillars_of_eternity_1.1.0.2.tar.gz from gog_pillars_of_eternity_1.0.0.1.tar.gz + delta

xdelta3 -d -s gog_pillars_of_eternity_1.0.0.1.tar.gz gog_pillars_of_eternity_1.1.0.2.tar.gz.delta gog_pillars_of_eternity_1.1.0.2.tar.gz

3. Extract to the desired directory and check if the files match with the above.

mkdir test
tar xvfz gog_pillars_of_eternity_1.1.0.2.tar.gz -C test
cd test\Pillars\ of\ Eternity
hashdeep -r -a -k pillars_hashdeep_v1.1.0.2.txt . |& tee hashdeep_audit.log

If everything is ok, then copy over the files to $GAMEDIR

By the way, @v221, as you only run the rdiffdir against the game subdir, there's a mismatch in the start.sh script, but it's jut the variables specifying the version numbers.

sdiff -s Pillars_of_Eternity_rdiffdir/start.sh Pillars_of_Eternity_gog/start.sh
declare -r VERSION="1.0.0.1" | declare -r VERSION="1.1.0.2"
declare -r VERSION_DEV="1.0.2.0508" | declare -r VERSION_DEV="1.0.3-503"

There's another, faster, way to check the modified files with rsync, but it gives less information.
rsync -avcn Pillars_of_Eternity_rdiffdir/ Pillars_of_Eternity_gog/

Or checking only size instead of checksum
rsync -avn Pillars_of_Eternity_rdiffdir/ Pillars_of_Eternity_gog/

And then copying over the files
rsync -av Pillars_of_Eternity_rdiffdir/ Pillars_of_Eternity_gog/

If you want I could upload the delta somewhere, but I think anyone could replicate the process and upload it/create a torrent of it.



My, that was long :P. Sorry

Right, thank you.
I’ll keep that in mind for the next iterations of the patch.

About your method with xdelta3, I’m going to give it a try on my end and see if I can control that everything is in the state it should be after the patching process.

Hi,

Can I ask... would it be very hard for you to make this as a .deb installer that would handle all of it? I am a relatively new Linux user, and I just don't understand why Linux games from GOG are always in a tarball, or something similar and require the user to either not have the game actually installed as an application, or have to delve into the terminal, or fiddle with making special .desktop files to have things look right, or run right. It's frustrating to me. I've learned a lot, but I don't know if I'll ever know enough to get passed the frustrating of this.

GOG staff didn't seem to respond when I asked about it, and neither has Obsidian on their forums... it just bugs me. But I like using Linux because of the fact that it's mine, and I can do what I want with it - and not have a company watch everything I do and prevent me from accessing some things in my own operating system.

I might sound ridiculous to some people, and I'm sorry for that. But I can't change what matters to me, and what doesn't for fear of being a liar.