It seems that you're using an outdated browser. Some things may not work as they should (or don't work at all).
We suggest you upgrade newer and better browser like: Chrome, Firefox, Internet Explorer or Opera

×
Community
General discussionGOG Database - A website that collects data on GOG games(791 posts)(791 posts)
(791 posts)
Pages:
1
10
20
30
36
37
38
39
40
50
53
This is my favourite topic
Posted April 08, 2021
Thank you Yepoleb.

Btw could you maybe differentiate the Bonus downloads tags from the Intallers or Patch ones?
Also, I cannot search them in the Extended Changelog, only product titles are considered.
If it's not possible it's not a big deal tho, since I still have my method.
Posted April 08, 2021
Nice work. There goes a couple of feature suggestions, that might be interesting for some users:

- Date when it was announced "Coming soon".
- Date when it was published in GOG "In development", in case it was.
- Date when it was published in full.
- Dates for updates to the files.

Then, derived data would be: Time between announcement and publication. Time it has been "in development". Time since last patch...

These might be useful for people who are thinking about purchasing a game or not: "How long has it been in development already?" "How long since this game was announced?" "How often is this game updated? Is this a finished game? Is it receiving attention from the developers?" You get the idea.

The dates would provide the whole picture for people who care to mill the numbers and find out trends.

Be free to not doing any of this, though :D Just giving ideas
Posted April 08, 2021
avatar
Yepoleb: After a long night https://new.gogdb.org is finally running. Please go there, look around, but don't share the links outside the forum because they will become dead once the old site gets replaced. Bug reports are appreciated. Sorry if your requested feature is not included, after 6 months of putting all my free time into this project I just had to draw a line somewhere. I'm really looking forward to doing more exciting things now. A few small things are still to do, but nothing that impacts functionality. Now off to bed, have fun but don't break anything while I sleep <3.
If / when you have a moment, the Price graph only goes back to April 7th, so can you fix that so it goes back to previous years too, if possible? Otherwise, great website you have there.
Posted April 08, 2021
high rated
avatar
phaolo: Also, I cannot search them in the Extended Changelog, only product titles are considered.
If it's not possible it's not a big deal tho, since I still have my method.
I really wanted to make a search feature for you, but it would have taken too much time. But there's a very simple solution for you that requires no scraping. The SQLite database at https://new.gogdb.org/data/index.sqlite3 is the main index for the site, so you can download that and run an SQL query locally, for example `SELECT * FROM changelog WHERE "action" = 'del' AND "dl_type" = 'bonus';`. All the details are in the serialized_record column with the structure defined in https://github.com/Yepoleb/gogdb/blob/master/gogdb/core/model.py, the root structure being ChangeRecord. I hope that works, if you need help feel free to message me.
Posted April 08, 2021
avatar
Yepoleb: [...]
Prices are back :). I'd rather look at what GOG DB is showing rather than what my scripts collect, so that's saying something :P.

avatar
TheBigCore: If / when you have a moment, the Price graph only goes back to April 7th, so can you fix that so it goes back to previous years too, if possible? Otherwise, great website you have there.
I think Yepoleb just re-implemented pricing collection. The APIs don't provide historical data, so you can ever only report on what you actively collect at certain points in time. Hope it makes sense.

avatar
Yepoleb: [...]
By "Store API" do you mean the external_releases endpoint?
Post edited April 08, 2021 by WinterSnowfall
Posted April 08, 2021
avatar
Yepoleb: After a long night https://new.gogdb.org is finally running. Please go there, look around, but don't share the links outside the forum because they will become dead once the old site gets replaced. Bug reports are appreciated. Sorry if your requested feature is not included, after 6 months of putting all my free time into this project I just had to draw a line somewhere. I'm really looking forward to doing more exciting things now. A few small things are still to do, but nothing that impacts functionality. Now off to bed, have fun but don't break anything while I sleep <3.
Awesome, I didn't know you were working on an updated version of GOGDB.

I know you consider the project almost complete, but if I may ask a question, would it be possible to add tracking for achievements if they are available for a game, or is there an alternative way to extrapolate the data? GOG's tags are increasingly unreliable (only now I count dozens of errors), and other dedicated sites also seem to have some issues with some GOG games.
Posted April 08, 2021
avatar
Yepoleb: After a long night https://new.gogdb.org is finally running. Please go there, look around, but don't share the links outside the forum because they will become dead once the old site gets replaced. Bug reports are appreciated. Sorry if your requested feature is not included, after 6 months of putting all my free time into this project I just had to draw a line somewhere. I'm really looking forward to doing more exciting things now. A few small things are still to do, but nothing that impacts functionality. Now off to bed, have fun but don't break anything while I sleep <3.
avatar
Alexim: Awesome, I didn't know you were working on an updated version of GOGDB.

I know you consider the project almost complete, but if I may ask a question, would it be possible to add tracking for achievements if they are available for a game, or is there an alternative way to extrapolate the data? GOG's tags are increasingly unreliable (only now I count dozens of errors), and other dedicated sites also seem to have some issues with some GOG games.
Since I already did the research for MaGog, I will just point out here that there are two major ways of knowing if a game has achievements.

The obvious one:
- the "Achievements" information listed under "Game Features" on the game page (and identical information found elsewhere in the API).

The less obvious one:
- the "window.profilesData.achievements" under https://www.gog.com/u/[username]/game/[game-id] (any valid username would do, e.g. Alexim, regardless of whether he or she owns the game or not), which lists all the achievements in the game (name, description, percentage)
- a small subset of this information is displayed under "Popular Achievements" on the game page

This being GOG, it goes without saying that these two sources sometimes contradict each other. The second tends to be more accurate.
Post edited April 08, 2021 by mrkgnao
Posted April 08, 2021
avatar
Alexim: Awesome, I didn't know you were working on an updated version of GOGDB.

I know you consider the project almost complete, but if I may ask a question, would it be possible to add tracking for achievements if they are available for a game, or is there an alternative way to extrapolate the data? GOG's tags are increasingly unreliable (only now I count dozens of errors), and other dedicated sites also seem to have some issues with some GOG games.
avatar
mrkgnao: Since I already did the research for MaGog, I will just point out here that there are two major ways of knowing if a game has achievements.

The obvious one:
- the "Achievements" information listed under "Game Features" on the game page (and identical information found elsewhere in the API).

The less obvious one:
- the "window.profilesData.achievements" under https://www.gog.com/u/[username]/game/[game-id] (any valid username would do, e.g. Alexim, regardless of whether he or she owns the game or not), which lists all the achievements in the game (name, description, percentage)
- a small subset of this information is displayed under "Popular Achievements" on the game page

This being GOG, it goes without saying that these two sources sometimes contradict each other. The second tends to be more accurate.
Thank you. Yes the obvious method has been proven wrong many times, and even if I report the inaccuracy to GOG, it usually takes months for it to be corrected. Another annoying thing is that GOG never indicates the total number of achievements of a game, and it would be very useful to make comparisons.

The second method is very useful thanks!
Posted April 08, 2021
high rated
avatar
TheBigCore: If / when you have a moment, the Price graph only goes back to April 7th, so can you fix that so it goes back to previous years too, if possible? Otherwise, great website you have there.
I don't have a way to get historic data and I decided to hide the old values from pre 2019 because they're so outdated and leave a huge gap in the graph :(.

avatar
WinterSnowfall: By "Store API" do you mean the external_releases endpoint?
I mean /v2/games/, external_releases was discovered too late by me to include it.

avatar
Alexim: I know you consider the project almost complete, but if I may ask a question, would it be possible to add tracking for achievements if they are available for a game, or is there an alternative way to extrapolate the data?
I don't consider the project complete by any means, there's still a lot that would be nice to add, I just want to take a break. Tracking achievements would probably be possible by requesting galaxy APIs documented here:
https://gogapidocs.readthedocs.io/en/latest/galaxy.html#get--clients-(int-product_id)-users-(int-user_id)-achievements
There's a good chance Galaxy 2 introduced even newer better APIs, but I have not looked into that.

Thanks everyone for the feedback so far!

Edit: I just noticed using the profiles it's possible to list the achievements of any game, even those not owned by the user, by changing the product ID in the URL like this: https://www.gog.com/u/Yepoleb/game/1721001406
Post edited April 09, 2021 by Yepoleb
Posted April 08, 2021
avatar
Yepoleb: I mean /v2/games/, external_releases was discovered too late by me to include it.
Gotcha. Makes sense - I don't think external_releases would have brought anything new to the table, at least not as far as game tracking is concerned, so no loss.
Posted April 08, 2021
avatar
WinterSnowfall: Gotcha. Makes sense - I don't think external_releases would have brought anything new to the table, at least not as far as game tracking is concerned, so no loss.
It would add info to a handfull of games that are inaccessible unless you own them (and they can't be bought anymore).
Post edited April 08, 2021 by Yepoleb
Posted April 09, 2021
Oh, this is nice news, and unexpected... Though I see that the regional pricing info didn't make it, though you were saying you were interested in adding it after GOG ended the FPP... But I guess the main purpose of GOGDB is file and change info. On that note though, now that I learned they can be obtained from GOG servers, albeit with a fair bit of trickery, checksums would be a nice thing to also list.
Posted April 09, 2021
Are the games' files checksums from the Manifest IDs completely gone? I can't seem to find them. That's a huge bummer for me. Personally, they were the second most important function of the database, after the Changelog.
Posted April 09, 2021
high rated
avatar
Cavalary: Though I see that the regional pricing info didn't make it, though you were saying you were interested in adding it after GOG ended the FPP...
avatar
ariaspi: Are the games' files checksums from the Manifest IDs completely gone? I can't seem to find them.
I guess it's time to explain why certain features weren't implemented or even removed.

# Changelog search

Doing this properly would have required implementing a complex filtering engine that either translates a number of URL arguments into an SQL query or sanitizes an SQL query string to prevent attacks. I projected it would take me about 20 hours to program that, so I decided to not implement it for now. The required columns were added to the changelog index though, so anyone can write a simple script to run on their own PC.

# Regional prices

I spent a significant time researching how to do this. The general problem was that there are no strict pricing regions on GOG, every country could have its own regional price in theory. Figuring out how to pick the countries to monitor and how to display them in a nice way turned out to be too challenging.

# Checksums

GOG introduced SHA256 checksums in newer builds. These checksums either are in addition to the MD5 values or replace them. Making them machine readable would require two files, one for SHA256 and one for MD5. I considered this too bad of a solution to implement. The old manifest view was also bad because it was limited to a certain number of files, which made searching difficult. I decided a locally run script or program would be the only sensible choice to verify file integrity, so I didn't bother to implement a new checksum view. People manually verifying checksums never appeared to me as a possibility, which is an assumption I now regret.

# Writing bad code

Now there are probably a few of you wondering why I didn't just write a shitty solution to improve on later. The most obvious point is that I don't enjoy making shitty things. I also would rather not show data than show incorrect or misleading data. It undermines my credibility and leads to a lot of annoying support requests and complaints. There's also the problem that once a feature is implemented, no matter how badly done it is, users are going to expect it to be continuously improved and maintained. If you don't believe me, just look at how GOG does things (winking smiley). In all the time I've been running GOG DB, the number of bug reports was negligible and the number of feature requests high. This is in my opinion the best way a software project can be run, because users will always want more features.

# Personal thoughts

Something I've talked about before is my lack of interest in GOG. My last purchase was in December 2018, that's the year Valve released Proton. I buy my games on Steam, Humblebundle or itch.io now. I started developing the current update because I felt like it would make me happy to finally fix and improve a project that has brought value to so many people. But over the months I began to realize that getting there took much longer and was much less fun than anticipated. I started to hate working on GOG DB, because I knew I could instead work on something much more personally fulfilling, but I kept going to not lose all the progress. So all non essential features got cut and I just tried to get a functional product as soon as possible. And once again the future of GOG DB is unclear, because I remembered how frustrating it is to develop.

Thanks for reading another one of my hopefully insightful technical personal posts.
Posted April 09, 2021
avatar
Yepoleb: I also would rather not show data than show incorrect or misleading data.
If only GOG held itself to such high standards. But then we'd be out of a hobby :).

avatar
Yepoleb: It would add info to a handful of games that are inaccessible unless you own them (and they can't be bought anymore).
Fair enough, but the info listed there seems very... what's the right word for it... the complete opposite of normalized.
Post edited April 09, 2021 by WinterSnowfall
Pages:
1
10
20
30
36
37
38
39
40
50
53
This is my favourite topic
Community
General discussionGOG Database - A website that collects data on GOG games(791 posts)(791 posts)
(791 posts)