Writing an explanation for my decision took quite a lot of attempts because I myself wasn't sure what the most important factors were. I'd say relational databases are not really made for big scale updates of data but mostly smaller incremental updates where consistency is important. This means a lot of synchronization between the application and DB to keep relationships intact. The overhead of the constant select and update statements is massive for the current updater. Keeping things in memory is not really an option as well because it doesn't scale well. What I really want to do is fetch all the new data, merge it with the old one, compare the two versions, dump everything out again and not worry about any previous state. NoSQL lets me do that.

Writing code to abstract the data in a way that a relational database can deal with also wasn't really fun. I'd have a Pythonic data representation and one that the DB could deal with and a lot of code to convert between the two. With the new architecture very little data conversion needs to happen.

Of course, querying becomes a problem when there's no database engine taking care of the optimization. My current plan is to either generate index-like data structures manually and store them as JSON or insert relevant values into a disposable SQLite DB that runs the query and references the full version in the NoSQL storage. Depends on if it's really worth it pulling in another dependency.

With the discontinuation of the Fair Price Package program (please no discussion about that in this thread) this became something I am actually interested in working on. The new updater has been designed in a way to support this from the beginning, so there's a very high chance I will be able to implement it without many obstacles.

A changelog changelog is a great idea, even if it may sound a bit absurd. Associating file changes with the changelog would probably require a ton of special workarounds to work reliably with, so unfortunately that's not something I plan to do.

Thank you for your suggestions!

Interesting. Nice to know that you are not I/O limited by the GOG servers more than anything else...

Fair point, it's one of the strong points of NoSQL after all.

Sounds like you're shifting overhead around :), but if it gets you the performance increase you need, I guess it's worth it.

Good luck with everything!

P.S.: You may also want to check out stuff like python compilers if you're looking to get some "free" performance for your code. I'm quite fond of Nuitka, as it's a breeze to use. I/O stuff will still bear the brunt of the load, of course, but 3x speedups in other areas are not unheard of.

Destroy All Humans! is listed as: Product Type - Package and no mention of updates.
Product Type should be Game, does it have a wrong ID since its not correctly updated?

Thanks! To me this is what programming is really about, finding the best compromises and sometimes even giving unconventional solutions a try.

Seems like an interesting project, but not something I would use. Almost all applications are limited by a specific bottleneck and usually running that same bottlenecked code slightly faster won't do very much. It certainly doesn't give me the same rush a proper performance driven redesign does ;).

While I agree with your point, sometimes the bottleneck can be the language itself, and that's certainly true for interpreted languages like Python. Getting it "magically" translated to C, even not that optimized C, can work wonders for throughput. That doesn't mean you shouldn't aim for optimized code, of course, but compilers like nuitka strap it to a rocket after you've squeezed all you could out of it.

Thank you very much for your work, Yepoleb. The site is very useful, for us GOGers, and I greatly appreciate it.

A suggestion I have is on the build pages with manifest IDs, which contain a list for the game's files and their checksums, if we could save that list as a checksum file, that would be awesome.

A simple textfile that's compatible with the md5sum command?

sounds like it. however, the md5 checksums from the database don't always coincide with the real game installation for some reason. Perhaps because someone like me uses only offline installers rather than Galaxy? For example, I downloaded md5 checksums from the database for an old Witcher 3 install I had for about a year to check it out and there were some checksum errors. Same thing with Homeworld.

Probably. They have to work for the Galaxy versions, because that's what the client uses for integrity checks, but using them for anything else is not guaranteed to work. There's also nothing I can do, I just display the existing data.

Yup! It will make things easier for some of those that don't use Galaxy.

Galaxy can download two types of installers - Galaxy based one for support with Galaxy, and what they call the offline installers which are essentially the same as the installers you can download from browser links in your GOG library. Both types have different MD5 checksum values.

If you use the Python script gogrepo.py to get a manifest of your GOG games, you can see there are different MD5 value listings for the various files you can download, zip and some other file types excepted.

You mind unpacking this statement because I didn't get it lol

There's a script called gogrepo.py that you can use to download all of your gog games. First it creates a text file (called a manifest) that lists all of the files and their MD5 sums. MD5s seem to only be available for the game files though, not the other types of extras files (soundtracks etc). I'm guessing that Timboli means you could check the difference between the MD5s for the offline installers in the manifest and the ones for the Galaxy installers from the database.

Those md5sum were provided by GOG. You can grab them manually, or use tools like gogrepo.py.
Basically, GOG only provide md5sum for installers, but not for goodies.