It seems that you're using an outdated browser. Some things may not work as they should (or don't work at all).
We suggest you upgrade newer and better browser like: Chrome, Firefox, Internet Explorer or Opera

×
Community
General discussionGOG Database - A website that collects data on GOG games(775 posts)(775 posts)
(775 posts)
Pages:
1
10
20
30
33
34
35
36
37
52
This is my favourite topic
Posted November 08, 2020
avatar
WinterSnowfall: That's quite the paradigm shift. Is it for performance reasons or just more optimal coding?
Writing an explanation for my decision took quite a lot of attempts because I myself wasn't sure what the most important factors were. I'd say relational databases are not really made for big scale updates of data but mostly smaller incremental updates where consistency is important. This means a lot of synchronization between the application and DB to keep relationships intact. The overhead of the constant select and update statements is massive for the current updater. Keeping things in memory is not really an option as well because it doesn't scale well. What I really want to do is fetch all the new data, merge it with the old one, compare the two versions, dump everything out again and not worry about any previous state. NoSQL lets me do that.

Writing code to abstract the data in a way that a relational database can deal with also wasn't really fun. I'd have a Pythonic data representation and one that the DB could deal with and a lot of code to convert between the two. With the new architecture very little data conversion needs to happen.

Of course, querying becomes a problem when there's no database engine taking care of the optimization. My current plan is to either generate index-like data structures manually and store them as JSON or insert relevant values into a disposable SQLite DB that runs the query and references the full version in the NoSQL storage. Depends on if it's really worth it pulling in another dependency.
Posted November 08, 2020
high rated
avatar
Cavalary: That said, my main request stays the same, listing of regional prices (and restrictions) as MaGOG does.
With the discontinuation of the Fair Price Package program (please no discussion about that in this thread) this became something I am actually interested in working on. The new updater has been designed in a way to support this from the beginning, so there's a very high chance I will be able to implement it without many obstacles.

avatar
Cavalary: Other than that, any work on changelogs to keep as much track as possible of changes since GOG often does not will be welcome. Might there be a way to associate official changelog updates with file changes, to list (or link to) the changelog where available and specify there is none where it's not?
A changelog changelog is a great idea, even if it may sound a bit absurd. Associating file changes with the changelog would probably require a ton of special workarounds to work reliably with, so unfortunately that's not something I plan to do.

Thank you for your suggestions!
Posted November 09, 2020
avatar
Yepoleb: The overhead of the constant select and update statements is massive for the current updater.
Interesting. Nice to know that you are not I/O limited by the GOG servers more than anything else...

avatar
Yepoleb: With the new architecture very little data conversion needs to happen.
Fair point, it's one of the strong points of NoSQL after all.

avatar
Yepoleb: My current plan is to either generate index-like data structures manually and store them as JSON or insert relevant values into a disposable SQLite DB that runs the query and references the full version in the NoSQL storage. Depends on if it's really worth it pulling in another dependency.
Sounds like you're shifting overhead around :), but if it gets you the performance increase you need, I guess it's worth it.

Good luck with everything!

P.S.: You may also want to check out stuff like python compilers if you're looking to get some "free" performance for your code. I'm quite fond of Nuitka, as it's a breeze to use. I/O stuff will still bear the brunt of the load, of course, but 3x speedups in other areas are not unheard of.
Post edited November 09, 2020 by WinterSnowfall
Posted November 09, 2020
Destroy All Humans! is listed as: Product Type - Package and no mention of updates.
Product Type should be Game, does it have a wrong ID since its not correctly updated?
Post edited November 10, 2020 by rasay
Posted November 10, 2020
avatar
WinterSnowfall: Sounds like you're shifting overhead around :), but if it gets you the performance increase you need, I guess it's worth it.

Good luck with everything!
Thanks! To me this is what programming is really about, finding the best compromises and sometimes even giving unconventional solutions a try.

avatar
WinterSnowfall: P.S.: You may also want to check out stuff like python compilers if you're looking to get some "free" performance for your code. I'm quite fond of Nuitka, as it's a breeze to use. I/O stuff will still bear the brunt of the load, of course, but 3x speedups in other areas are not unheard of.
Seems like an interesting project, but not something I would use. Almost all applications are limited by a specific bottleneck and usually running that same bottlenecked code slightly faster won't do very much. It certainly doesn't give me the same rush a proper performance driven redesign does ;).
Posted November 10, 2020
avatar
Yepoleb: Almost all applications are limited by a specific bottleneck and usually running that same bottlenecked code slightly faster won't do very much.
While I agree with your point, sometimes the bottleneck can be the language itself, and that's certainly true for interpreted languages like Python. Getting it "magically" translated to C, even not that optimized C, can work wonders for throughput. That doesn't mean you shouldn't aim for optimized code, of course, but compilers like nuitka strap it to a rocket after you've squeezed all you could out of it.
Post edited November 10, 2020 by WinterSnowfall
Posted November 16, 2020
Thank you very much for your work, Yepoleb. The site is very useful, for us GOGers, and I greatly appreciate it.

A suggestion I have is on the build pages with manifest IDs, which contain a list for the game's files and their checksums, if we could save that list as a checksum file, that would be awesome.
Posted November 17, 2020
avatar
ariaspi: A suggestion I have is on the build pages with manifest IDs, which contain a list for the game's files and their checksums, if we could save that list as a checksum file, that would be awesome.
A simple textfile that's compatible with the md5sum command?
Posted November 17, 2020
avatar
ariaspi: A suggestion I have is on the build pages with manifest IDs, which contain a list for the game's files and their checksums, if we could save that list as a checksum file, that would be awesome.
avatar
Yepoleb: A simple textfile that's compatible with the md5sum command?
sounds like it. however, the md5 checksums from the database don't always coincide with the real game installation for some reason. Perhaps because someone like me uses only offline installers rather than Galaxy? For example, I downloaded md5 checksums from the database for an old Witcher 3 install I had for about a year to check it out and there were some checksum errors. Same thing with Homeworld.
Post edited November 17, 2020 by anzial
Posted November 17, 2020
avatar
anzial: the md5 checksums from the database don't always coincide with the real game installation for some reason. Perhaps because someone like me uses only offline installers rather than Galaxy?
Probably. They have to work for the Galaxy versions, because that's what the client uses for integrity checks, but using them for anything else is not guaranteed to work. There's also nothing I can do, I just display the existing data.
Posted November 19, 2020
avatar
Yepoleb: A simple textfile that's compatible with the md5sum command?
Yup! It will make things easier for some of those that don't use Galaxy.
Posted November 20, 2020
avatar
Yepoleb: A simple textfile that's compatible with the md5sum command?
avatar
anzial: sounds like it. however, the md5 checksums from the database don't always coincide with the real game installation for some reason. Perhaps because someone like me uses only offline installers rather than Galaxy? For example, I downloaded md5 checksums from the database for an old Witcher 3 install I had for about a year to check it out and there were some checksum errors. Same thing with Homeworld.
Galaxy can download two types of installers - Galaxy based one for support with Galaxy, and what they call the offline installers which are essentially the same as the installers you can download from browser links in your GOG library. Both types have different MD5 checksum values.

If you use the Python script gogrepo.py to get a manifest of your GOG games, you can see there are different MD5 value listings for the various files you can download, zip and some other file types excepted.
Post edited November 20, 2020 by Timboli
Posted November 20, 2020
avatar
Timboli: If you use the Python script gogrepo.py to get a manifest of your GOG games, you can see there are different MD5 value listings for the various files you can download, zip and some other file types excepted.
You mind unpacking this statement because I didn't get it lol
Posted November 20, 2020
avatar
Timboli: If you use the Python script gogrepo.py to get a manifest of your GOG games, you can see there are different MD5 value listings for the various files you can download, zip and some other file types excepted.
avatar
anzial: You mind unpacking this statement because I didn't get it lol
There's a script called gogrepo.py that you can use to download all of your gog games. First it creates a text file (called a manifest) that lists all of the files and their MD5 sums. MD5s seem to only be available for the game files though, not the other types of extras files (soundtracks etc). I'm guessing that Timboli means you could check the difference between the MD5s for the offline installers in the manifest and the ones for the Galaxy installers from the database.
Posted November 21, 2020
avatar
HappyPunkPotato: There's a script called gogrepo.py that you can use to download all of your gog games. First it creates a text file (called a manifest) that lists all of the files and their MD5 sums. MD5s seem to only be available for the game files though, not the other types of extras files (soundtracks etc). I'm guessing that Timboli means you could check the difference between the MD5s for the offline installers in the manifest and the ones for the Galaxy installers from the database.
Those md5sum were provided by GOG. You can grab them manually, or use tools like gogrepo.py.
Basically, GOG only provide md5sum for installers, but not for goodies.
Pages:
1
10
20
30
33
34
35
36
37
52
This is my favourite topic
Community
General discussionGOG Database - A website that collects data on GOG games(775 posts)(775 posts)
(775 posts)