It seems that you're using an outdated browser. Some things may not work as they should (or don't work at all).
We suggest you upgrade newer and better browser like: Chrome, Firefox, Internet Explorer or Opera

×
avatar
immi101: which is why the installer doesn't do that :)
I haven't looked into it myself, but I read some people noticed installers at one point which were using zlib compressed files within the installer, which was still being compressed with lzma2 as far as inno setup was concerned (can't remember the thread right now).

Is it "official" that they've switched only to zlib based compression on the new installers that are showing up these days?

Edit: Nevermind, found your original reply on this matter :).

To share my views on it, yes zlib is faster, but I think lzma is as fast today as zlib/deflate was back when everyone appreciated its speed. We have a lot more processing power under the hood these days, and since lzma2 now is multi-threading capable on decompression as well, I think the better compression ratio it provides would be an offer hard to refuse. Think of all the people with limited bandwidth that would benefit immensely from this - I don't think they'd mind an extra 30 seconds of installation time if they save up 10-20 minutes of download time.
Post edited May 12, 2018 by WinterSnowfall
avatar
rtcvb32: I wonder if it's more the ram to decompress that's an issue more than the speed... While i don't see anyone today using one of the installers with say less than 1Gb of ram, i've done compression jobs where it takes 2Gb to compress (single thread) and then it wants to decompress with 300Mb, or larger, simply the decompressor taking as much memory as it can.

But it does seem unlikely...
The estimates I've seen in terms of memory usage during 7-Zip decompression alone did not go above 100MBs even on Ultra. This was before multi-threaded decompression was added mind you, but I expect it to scale more or less linearly. Even with some memory overhead from the installer itself, it shouldn't be a problem for any system that can run Windows 7 :).

The decompression routine will "take as much as it can" to run as fast as it can, I think, but it will work even in situations where limited RAM is available.
Post edited May 12, 2018 by WinterSnowfall
avatar
WinterSnowfall: The estimates I've seen in terms of memory usage during 7-Zip decompression alone did not go above 100MBs even on Ultra.
Raise the Dictionary size... The 'Ultra' is just a preset.
Attachments:
dict.png (21 Kb)
avatar
marees: I should mention, in case it's relevant, that I'm running on 32-bit Windows OS.
Is it still supported? I know official support for Windows 7 ended a while back. No bueno being on an OS that isn't officially supported, and Win10 is free, so...
avatar
StingingVelvet: I know official support for Windows 7 ended a while back. No bueno being on an OS that isn't officially supported, and Win10 is free, so...
Unofficially still free to upgrade, but Win10 is a spyware hell with less user control, so.. no thanks.
Btw, Win7 is still receiving patches (even if buggy like all the others released by MS this year).
Post edited May 12, 2018 by phaolo
avatar
WinterSnowfall: To share my views on it, yes zlib is faster, but I think lzma is as fast today as zlib/deflate was back when everyone appreciated its speed. We have a lot more processing power under the hood these days, and since lzma2 now is multi-threading capable on decompression as well, I think the better compression ratio it provides would be an offer hard to refuse. Think of all the people with limited bandwidth that would benefit immensely from this - I don't think they'd mind an extra 30 seconds of installation time if they save up 10-20 minutes of download time.
i would be really surprised if the speed improvements for lzma allow it to come even somewhat close to zlib. zlib is just faster by a lot, especially if you take decompression and compression speed into account.
though there are other new modern codecs that aim at the same application profile as zlib: fast and moderately well compression.
e.g.: brotli, zstd and lz5
these probably would make a better alternative.

that being said, personally I absolutely agree with you. I don't care that much about slower installation. Most of the time I spend on downloading, smaller download sizes would be always preferable to me.
But we had that discussion already a few times about the linux installers. And it seems GOG has already made up their mind to favour speed over size. not sure they are going to change that now *shrug*

the other thing is that games not necessarily compress very well to begin with. Tightly compressed assets will usually give you better loading times (cpu power beats disc IO). Trying to compress these assets further for distribution just means a lot of wasted cpu cycles.
I tried in the past to recompress some linux games for storage and came across a few examples where lzma2 would give nearly zero improvements (iirc trine games for example). In the end when looking at the space saved by recompressing games where it does make a difference vs. the time wasted for games where it does not - I abandoned the whole project because it seemed just not worth it.
(obviously I only tested part of my library, who knows how that would look when testing the whole catalogue)


edit:
there is a fork of 7zip which includes support for the codecs I mentioned above:
https://mcmilk.de/projects/7-Zip-zstd/
it also has some fancy performance graphs for comparison
Post edited May 12, 2018 by immi101
avatar
StingingVelvet: Is it still supported? I know official support for Windows 7 ended a while back. No bueno being on an OS that isn't officially supported, and Win10 is free, so...
What do you mean "official"?

Mainstream support for Windows 7 ended in Jan 13th 2015. This means no new features, no free online or phone support etc.

However, MS is delivering security updates to Windows 7 until Jan 14th 2020. So still little over 1½ years of life left.
avatar
rtcvb32: Raise the Dictionary size... The 'Ultra' is just a preset.
Fair enough. But it's a preset for a reason - larger dictionaries will not necessarily bring any noticeable improvement in the compression ratio, except in very specific cases, while bumping the memory requirements quite a lot (especially for compression).
avatar
immi101: i would be really surprised if the speed improvements for lzma allow it to come even somewhat close to zlib. zlib is just faster by a lot, especially if you take decompression and compression speed into account.
No, that's totally true. zlib will always be faster than lzma2 on compression and decompression - it's not even a competition. When I tar tens/hundred of gigabytes worth of data I always go with gzip compression to get it done on the same day, while bzip2 or lzma2 (xz) would take ages longer.

My point was that lzma2, for common usage, is today as fast as zlib was back in the day on a Pentium III, when people used rar or ace if they needed something with a better compression ration. There's a general impatience these days - if an installer is not done in under a second then it's crap, because you have the best 16-core processor at your fingertips.

avatar
immi101: i would be really surprised if the speed improvements for lzma allow it to come even somewhat close to zlib. zlib is just faster by a lot, especially if you take decompression and compression speed into account.
True. In which case you could go with a more optimized preset even in lzma2. It's simply a matter of trying out the more efficient preset first, checking out the compression ration, if it's below 5%, go with the faster option. Sure, there's some wasted time here, but if you look at the overall gain in space this would lead up to, I believe we're talking tens if not hundreds of GB at least for the entire GOG catalogue.

Anyway, it's not like GOG won't go with zlib in the end, so guess we'll need bigger hard drives :).
Post edited May 12, 2018 by WinterSnowfall
avatar
timppu: What do you mean "official"?

Mainstream support for Windows 7 ended in Jan 13th 2015. This means no new features, no free online or phone support etc.

However, MS is delivering security updates to Windows 7 until Jan 14th 2020. So still little over 1½ years of life left.
Hmm well as long as people bail before security support ends I guess.
avatar
rtcvb32: Raise the Dictionary size... The 'Ultra' is just a preset.
avatar
WinterSnowfall: Fair enough. But it's a preset for a reason - larger dictionaries will not necessarily bring any noticeable improvement in the compression ratio, except in very specific cases, while bumping the memory requirements quite a lot (especially for compression).
I've seen an improvement from 5%-20%, but your mileage may vary depending on content.
I rarely use 7z since WinRAR introduced bigger than 4 MB dictionary sizes in version 5.

I've seen WinRAR to be around 2-3 times faster when compressing and 4-5 times faster when decompressing, compared to 7z LZMA2, both using 256 MB dictionary and word size (and ofc solid archives), while the compression ratio is about 3-5% worse for WinRAR.

Of course, this can depend quite a lot on the content, as I've seen in few occasions where the pre-WinRAR 5 version achieved better compression ratio, with its 4 MB dictionary size, than 7z & WinRAR 5 with 256 MB dictionary size.
avatar
StingingVelvet: Hmm well as long as people bail before security support ends I guess.
Yeah. I'll probably do with this Windows 7 laptop like I've done with my former Windows XP laptops (and a desktop): the original Windows (7) will remain in this, but strictly in offline mode, ie. internet connectivity is disabled. So this will be my "Windows 7 era gaming PC" as long as it runs, running those older games that have issues on Windows 10.

On the side, I might install Linux (Mint) in multiboot mode, in case I or someone else wants to go online with this PC. I have couple such older PCs where I am running both Windows XP (offline) and Linux Mint side by side, in fact they even have Windows 98SE as the third installed OS, but I very rarely need it, ie. that a game wouldn't run ok on XP.

At some point I will buy a newer gaming laptop anyway, and it will have Windows 10 preloaded anyway. Hence I don't necessarily need to upgrade this particular laptop to Windows 10, even though I already have that option too (I registered the free Windows 10 Pro update for this while it was free).
A lot of this is over my head, but from what I can follow, you're saying that GOG is using a different compression method for faster install times. Is that a fair summary? And have any of you actually verified this? Not their motives, I mean, but that they have in fact switched to a different method.
avatar
WinterSnowfall: When I tar tens/hundred of gigabytes worth of data I always go with gzip compression to get it done on the same day, while bzip2 or lzma2 (xz) would take ages longer.
While bzip2 is a bit slower than gzip, it is much faster than xz (at least with the settings I tend to use). And there is a parallel version of it (pbzip2) which is super fast.
Look at lrzip with ZPAQ compression algorithm for top compression (and wait time!).
I wish GOG would use that.