It seems that you're using an outdated browser. Some things may not work as they should (or don't work at all).
We suggest you upgrade newer and better browser like: Chrome, Firefox, Internet Explorer or Opera

×
Although it may not make a huge difference depending on block and sector sizes, perhaps taking advantage of quiet times to run all known zips (and those going into installers) through zopfli to get a little extra data compression and space; Course the data inside is still the same just brute-forced to be smaller.

To note I'm referring to anything that's technically a zip container, including some dat files, as well as png files. Since the decompression doesn't take more resources or change the programs, the combined efforts to maximize space may help those with more limited bandwitdth, or just making it better in the long term.

As examples I've managed to drop the Pak.zip in Torchlight I think some 11Mb. Something to consider. Here's an example of repacking Reus since it only has a few files to consider, it may only be 60k or so but that's still 60k saved.

[code]
user@pc /n/Games/PC/GoG/Reus
$ advzip -z -4 *.zip
1533033 1526688 99% reus_artworks.zip
61117 60387 98% reus_avatars.zip
9922490 9883393 99% reus_soundtrack_sample.zip
3246878 3234868 99% reus_wallpaper.zip
14763518 14705336 99%
[/code]
Well I've compacted one of the primary files in torchlight, the Pak.zip went from 300+Mb to 292Mb (with no loss to my knowledge). Although maybe I can get it smaller I'm not sure if the 8+ megs in saved space is worth it. Sure running the game from a ramdrive might help, but unless I can replace the file in the installer then it takes up 292Mb extra space on my external right next to the main install files. (Plus the editor is another duplicate of that data)
So an update, I've Zopflied all my zips that were 800Mb and under (since if you go much bigger it's too much and the program just dies), and probably saved something like 100Mb. A note every single file had some of it's contents/size shaved, although some were just 30 bytes and some were a few megs.

SO... If GoG admins want the smaller files I can upload them and they can confirm and update the files. Course there were a couple zips were broke (CRC's didn't check out) and had to be rebuilt, but they looked okay. If not uploading the extras this way to other people is probably a waste of time for my slow bandwidth...
If you want to save a lot of space, try using the xz or lzma archive formats, supported by 7zip et al. Doing the compression on a machine with lots of RAM can yield better compression results as the encoder is capable of better compression when given unbounded gobs of RAM. Just an idea to ponder.
Wow, the last time I actively struggled to reduce the size of files was about 12 years ago when I had an 1.2GB hard drive. Eversince the Terrabyte era I don't really care about size. I keep only about a dozen games installed and I usually don't keep the installers around as it takes about just as much time to d/l them as to copy them from my NAS.
Yeah, I don't really bother doing that anymore either but xz or lzma would be the way to do it if one wanted to. :) I'd rather spend $50 for a new hard disk than waste hours of my time. :)
avatar
skeletonbow: If you want to save a lot of space, try using the xz or lzma archive formats, supported by 7zip et al. Doing the compression on a machine with lots of RAM can yield better compression results as the encoder is capable of better compression when given unbounded gobs of RAM. Just an idea to ponder.
Didn't notice i got replies to this.

I'm not against 7zip (LZMA), actually usually i prefer it. However the zip file is the standard for non unix-systems, and that's how it's being delivered via GoG. Windows has built-in zip support as well.

Zopfli is truthfully less effective than 7zip, but if I recompact all my zips and then upload them again, then everyone benefits from my work. If they release the files as 7z instead then this question is totally redundant and I'd never have brought it up.


This is also one of those things where you can recompress a whole bunch of files in the background by putting it as a low priority, you're not exactly wasting your time since you aren't really waiting for it to get done before you can do anything else.... Just give it a few hours or days while you're busy away doing other things.

There's also a few cases where 7zip isn't really going to help. An example being Torchlight's Pak.zip file, I've recompacted it so it's some 12 Megs smaller by recompressing the png's and the entire zip archive. It certainly makes it easier to throw onto a ram-drive and just play without having to have the harddrive humming away for the main files.

But it's apparent interest in taking advantage of this is rather low priority.
avatar
skeletonbow: If you want to save a lot of space, try using the xz or lzma archive formats, supported by 7zip et al. Doing the compression on a machine with lots of RAM can yield better compression results as the encoder is capable of better compression when given unbounded gobs of RAM. Just an idea to ponder.
avatar
rtcvb32: But it's apparent interest in taking advantage of this is rather low priority.
Another highly contributing factor is that for example I own approximately 1/3 of the GOG current game catalogue (242 out of 700 games), and that consumes approximately 340GB as-downloaded (not installed), including all bonus materials for each game. With hard disks going for around $60 or less for 1TB, and cheaper per TB as you increase the size to 2/3TB or possibly larger, the cost per TB is super cheap and there is a strong argument to be had for buying a new (or used) hard disk large enough to just store all the games outright than to spend too much effort trying to compact them for very small benefit of 1-3%.

If someone at GOG can toggle a flag in a compression tool or even use a similar tool without much effort at all and without risk to quality assurance or other problems and gain a non-negligible amount of reduction in size compared to the going price per terabyte of disk space, then hey - why not! But I wouldn't want to see them spend too many man hours of effort doing such a thing for ever diminishing benefit as hard disks and other data storage devices increase at sizes far more exponentially than old games increase in size for example. ;o)

I suspect this is a rather niche thing in that the number of people who could benefit from it is likely relatively small percentagewise of the customer base, unmeasureable without some form of data gathering tool or hardware survey like Steam uses, and of those customers that could benefit from it on old hardware with small storage space only a fraction of them are even affected and only a fraction of them actually care and only a fraction of them who care would end up benefitting.

That's not suggesting to do or not do anything, just trying to rationalize that even if it is beneficial to some subset of users who would actually benefit and care, the man hours to do it out of the man hours of finite resources available are possibly better spent bringing new games to the catalogue, or fixing bugs or something and low impact changes like this could get thrown in a tickler file as low priority for a future time. Kind of like how they just updated all the installers in November, which was a change that was lingering low priority forever but eventually happened. ;o)

I used to experiment very heavily with various compressors and algorithms in the past, ZIP, ARJ, LHA, RAR, UC2 and many others both in DOS, Linux and Windows over time. Eventually I settled on RAR for years and then eventually on zip in Windows and bz2 in Linux even though there were others out there that gave better results for particular types of data etc. The reason was largely that improvements in compression technology were largely being dwarfed significantly by the doubling in size of hard disks etc. every 18 months. Compression wasn't halving in size every 18 months for example, so you end up with a law of diminishing returns and the time and effort put into it was mostly academic than much real world benefit for most intents and purposes.

Now that 7zip is very popular in Windows and available everywhere I might need it, I use that with the highest settings by default, and if they're files to pass to others they always have 7zip already anyway or I can send them the URL. That works for personal usage anyway (although not necessarily for product offerings from a business per se.) And in Linux I mostly use bzip2 still but transitioning to XZ for larger scope things in which the savings are actually beneficial to me or others (ie: creating rpm packages, zipping huge archives of files that benefit from compression in the multi-gigabyte range etc.)

Anyhow, it is still interesting stuff and worth experimenting with over time whether for academic reasons or real world benefits. If nobody ever does it, we might be missing out on something good! :) XZ is definitely the hugest breakthrough in general purpose compression I've seen since Bzip2 though, and where my personal experiments would begin if I were to get the spark. :)
avatar
skeletonbow: Another highly contributing factor is that for example I own approximately 1/3 of the GOG current game catalogue (242 out of 700 games), and that consumes approximately 340GB as-downloaded (not installed), including all bonus materials for each game. With hard disks going for around $60 or less for 1TB, and cheaper per TB as you increase the size to 2/3TB or possibly larger, the cost per TB is super cheap and there is a strong argument to be had for buying a new (or used) hard disk large enough to just store all the games outright than to spend too much effort trying to compact them for very small benefit of 1-3%.

If someone at GOG can toggle a flag in a compression tool or even use a similar tool without much effort at all and without risk to quality assurance or other problems and gain a non-negligible amount of reduction in size compared to the going price per terabyte of disk space, then hey - why not! But I wouldn't want to see them spend too many man hours of effort doing such a thing for ever diminishing benefit as hard disks and other data storage devices increase at sizes far more exponentially than old games increase in size for example. ;o)
I'm glad to see someone with similar interests on me regarding compression, and I agree the improved hard drive space for price has greatly increased. I remember when 512Mb thumb drives were $80, now you can't get anything under a Gig and a Gig would be a couple dollars, while the newest ones of 80Gig or something is about $25-$30. When the 2Gig for $8 was up, you knew it was time to get a few thumb drives for every day use; Of course it's so easy to lose those.

A lot more compression is being seen with advancements regarding video and h.264, but the downside is the processing power needed to encode and decode such videos compared to regular Mpeg4. True it's just harder to compress lossless data since patterns, are harder to locate within certain lengths, but more importantly just the sheer amount of processing time we allow for them to work. Kinda why I brought up Zopfli is although it's highly intensive, it's 'compress once, decompress lots of times' type of situation. Quite often with more games I've been purchasing I've got the extras first, then download the actual games while I recompress the zips, by the time they are downloaded I have everything re-compressed, so it's not really that large of man-hours of wasted space. Course doing all the png's in a set of files does require a little more work.

But some of the re-compression isn't diminutive. TotalBiscuit recommended the Captain's Edition as a mod, and I've re-compressed most of the PNG's to be 10% smaller on average. Course recompressing them with a high iterative (of 65k) to try other combinations takes considerably longer with very little return compared to the initial (ranging from 5 to 30 bytes usually).

I love 7Z/XZ format, and like you I prefer to go with the highest setting usually (even if it's a lot more than necessary). Oh well, maybe we'll see some big improvements to come.
avatar
rtcvb32: I'm glad to see someone with similar interests on me regarding compression, and I agree the improved hard drive space for price has greatly increased. I remember when 512Mb thumb drives were $80, now you can't get anything under a Gig and a Gig would be a couple dollars, while the newest ones of 80Gig or something is about $25-$30. When the 2Gig for $8 was up, you knew it was time to get a few thumb drives for every day use; Of course it's so easy to lose those.

A lot more compression is being seen with advancements regarding video and h.264, but the downside is the processing power needed to encode and decode such videos compared to regular Mpeg4. True it's just harder to compress lossless data since patterns, are harder to locate within certain lengths, but more importantly just the sheer amount of processing time we allow for them to work. Kinda why I brought up Zopfli is although it's highly intensive, it's 'compress once, decompress lots of times' type of situation. Quite often with more games I've been purchasing I've got the extras first, then download the actual games while I recompress the zips, by the time they are downloaded I have everything re-compressed, so it's not really that large of man-hours of wasted space. Course doing all the png's in a set of files does require a little more work.

But some of the re-compression isn't diminutive. TotalBiscuit recommended the Captain's Edition as a mod, and I've re-compressed most of the PNG's to be 10% smaller on average. Course recompressing them with a high iterative (of 65k) to try other combinations takes considerably longer with very little return compared to the initial (ranging from 5 to 30 bytes usually).

I love 7Z/XZ format, and like you I prefer to go with the highest setting usually (even if it's a lot more than necessary). Oh well, maybe we'll see some big improvements to come.
Yeah, this sort of thing is done every so many years in the world of Linux to crunch all the software down to fit more things on a CD/DVD ISO image. I remember when Red Hat shifted from everything using gzip as the standard to trans-compressing all the main tarballs and whatnot in the distribution from gz, Z and other formats to using bzip2 everywhere possible, and later on some or all switched to XZ also. For many usage scenarios the gains are not necessarily big enough to justify it but one thing where it makes a useful differences is that crunching the files to fit everything on one or two DVDs or similar. It can be the difference between it going to 3 discs or staying on 2. But even if one tinkers with these things and it turns out to not be worth it, it's worth it to find out and there are always corner case or special scenarios where the extra effort might be worth it. My current collection is about 450GB at the moment. It'd be interesting to move all of that onto my Linux VM and script up a transcompression to XZ with options cranked and even toy with altering the compression of other things as you have done. Sometimes it's just academic fun to see how far one can push things like this too, and that's all the justification we need. :)

Been ages since I did such an experiment, but now would be the time to do it as this machine has a lot of hamsters in the wheel so to speak. :)
avatar
skeletonbow: Been ages since I did such an experiment, but now would be the time to do it as this machine has a lot of hamsters in the wheel so to speak. :)
About 25 years ago, I spend a week testing all the compression algorithms I could find (about 50 I think). My 2 GB hard drive was getting full, and I had set up a menu-driven system to keep all my games compressed, and decompress only the one I was currently playing. I needed to know which archiver to use for this system.

Surprisingly, the best compression was achieved by ACB, though its long compression/decompression times made it impractical to use in the setup I had built. It would be interesting to see how the algorithm fares today, but its size limitations probably make it impossible to test on real-world data.
avatar
skeletonbow: Yeah, this sort of thing is done every so many years in the world of Linux to crunch all the software down to fit more things on a CD/DVD ISO image.

<snip>

It'd be interesting to move all of that onto my Linux VM and script up a transcompression to XZ with options cranked and even toy with altering the compression of other things as you have done. Sometimes it's just academic fun to see how far one can push things like this too, and that's all the justification we need. :)

Been ages since I did such an experiment, but now would be the time to do it as this machine has a lot of hamsters in the wheel so to speak. :)
Well a favorite distro I've used a few times is Slax, Slax is based on being a LiveCD, using SquashFS combined with LZMA compression. The core ISO is 210Mb (including 6 modules) ready for most basic needs, including programming, Xwindows, internet/browser, office, etc. The whole system is based on using a Layered FUSE filesystem (I think) where you can just attach a module that includes everything needed for a program set, and if you unmount it, it's as though it was never on the system in the first place (Causes some issues trying to get the VirtualBox video drivers working on it...).

Documentation and Modules are slim pickings for now since the newest version ended up throwing out a bunch of old stuff but it has a huge amount of potential.



edit: filesystem was more union based, but this is mostly off memory...
Post edited April 20, 2014 by rtcvb32
According to benchmarks done on a Worms 2 demo, the best compression rates were achieved by RK. Apparently they didn't test LZMA though, which is a bit odd. Source
avatar
Psyringe: According to benchmarks done on a Worms 2 demo, the best compression rates were achieved by RK. Apparently they didn't test LZMA though, which is a bit odd. <a href="http://www.gog.com/forum/general_archive/zopfli_on_extras_and_zipspngs/post13" class="link_arrow"></a></div> If they have a preference for <span class="bold">RK</span>, then they could conveniently leave 7z off in order to give another method more praise. It could also be based on memory useage or a time limit, although if [url=http://en.wikipedia.org/wiki/PPMD]PPMD is then that shouldn't be applicable since PPMD is one of the more intensive ones. Some algorithms also do better than others, but LZMA is by far one of the best I've ever used.

I guess it really comes down to decompressing and speed. Example: A number of security systems could use h.264 for encoding and storage, however in a centralized setup the amount of CPU power required for encoding/decoding could be too high and in cases have caused skipped frames or worse, at which case a lower less powerful but reliable and less intensive algorithm is used, like MPEG-2 or MPEG-4. More importantly if it's on something like an integrated system or something more limited (example, Android devices or smart phones) the memory required just to decode it may not be an option and so it has to be stripped off. Course with technology as it is, MPEG encoding can be done via a dedicated chip for that intent purpose. This as an example, the Raspberry Pi which runs at 700Mhz can decode full HD Blueray video and still use less than 1% of it's CPU power because it's GPU does the decoding job. Lastly the most recent 7zip program that was stable was in 2010, so they may not want to base testing on a buggy newer version (but seems unlikely since I've read those are apparently pretty stable).

I'm not going to sift through the entire benchmark, but what they were concentrating on could gleam a hint of why they chose certain algorithms. LZOP for example is a real-time encoder/decoder, great for a basic compression filter while not degrading any performance what-so-ever (as far as you'd notice anyways).

Post edited April 21, 2014 by rtcvb32