It seems that you're using an outdated browser. Some things may not work as they should (or don't work at all).
We suggest you upgrade newer and better browser like: Chrome, Firefox, Internet Explorer or Opera

×
Community
General discussionGOG Database - A website that collects data on GOG games(775 posts)(775 posts)
(775 posts)
Pages:
1
10
20
22
23
24
25
26
40
52
This is my favourite topic
Posted August 15, 2018
avatar
immi101: you can avoid the regional blocking for all website APIs by setting the cookie 'gog_lc' to something different than 'DE_EUR_de-DE', like: 'US_USD_en-US'.
The IP-based blocking only kicks in if you actually attempt to buy a game or if you don't provide the cookie in your request.
The galaxy APIs for getting product/build information don't do any regional blocking afaik.
avatar
Yepoleb: I'm already doing that to get USD prices. It's not working anymore.
hmm, seems to be working here: (looking for the Dying Light entries)

$ curl -s 'https://www.gog.com/games/ajax/filtered?mediaType=game&page=1&search=dying&sort=popularity' -H 'Cookie: gog_lc=DE_EUR_en-US' | grep -o '"totalGamesFound":[0-9]*'
"totalGamesFound":0

$ curl -s 'https://www.gog.com/games/ajax/filtered?mediaType=game&page=1&search=dying&sort=popularity' -H 'Cookie: gog_lc=US_USD_en-US' | grep -o '"totalGamesFound":[0-9]*'
"totalGamesFound":6
Posted August 15, 2018
avatar
immi101: hmm, seems to be working here: (looking for the Dying Light entries)
You're right, I forgot to set the locale on the global API instance. Should be fixed now.
Post edited August 16, 2018 by Yepoleb
Posted August 17, 2018
avatar
WinterSnowfall: If you're using the content-system API endpoints to get the build info, I think you're going to run into an unpleasant kind of surprise. It is probably the most throttled API out there and you'll start getting 509 HTTP errors once you exceed what the GOGBear deems to be an acceptable quota. I hope you have a wait and retry loop in place...
avatar
Yepoleb: Not a problem if you provide the appropriate headers and an auth token. My biggest enemy is the Python garbage collection at the moment.
Have you checked whether using the jemalloc allocator helps?

https://zapier.com/engineering/celery-python-jemalloc/

Failing that, I'd suggest rewriting the part of your application which causes the most GC churn in Rust. I've been using the rust-cpython crate and setuptools-rust module to write the backend for one of my projects in Rust while the frontend uses PyQt and they make it very easy to safely and comfortably write a compiled Python module in Rust and integrate it into your build process and Rust is famous for how it can give you low and rock-steady memory consumption while working on a task.
Posted August 17, 2018
avatar
Yepoleb: Not a problem if you provide the appropriate headers and an auth token. My biggest enemy is the Python garbage collection at the moment.
avatar
ssokolow: Have you checked whether using the jemalloc allocator helps?
Thanks for the suggestion but this was only a problem with the initial update of the database. Now with only a couple of new builds per scan the memory doesn't fill up near as fast.
Posted August 19, 2018
avatar
ssokolow: Have you checked whether using the jemalloc allocator helps?
avatar
Yepoleb: Thanks for the suggestion but this was only a problem with the initial update of the database. Now with only a couple of new builds per scan the memory doesn't fill up near as fast.
Ahh. Good to hear. :)
Posted August 21, 2018
avatar
Yepoleb: ...
Seems theres been another batch of public->unlisted->public changes to a bunch of games.
Posted August 24, 2018
avatar
Cusith: Seems theres been another batch of public->unlisted->public changes to a bunch of games.
Thanks for the hint! I wasn't able to find the source of the problem right now, but I'll look into it when I'm back from vacation.
Posted August 28, 2018
Thank you for implementing this!
Post edited August 28, 2018 by Darkzoid
Posted August 28, 2018
Oh, the database still exists! I was looking under old url recently and was sure the project is dead... Perhaps it should redirect to the new address?
Posted August 28, 2018
high rated
avatar
ciemnogrodzianin: Oh, the database still exists! I was looking under old url recently and was sure the project is dead... Perhaps it should redirect to the new address?
It did that for half a year before I shut the old domain down.

Edit: The incorrect entries from the 21st have been removed and I've adjusted the logging to be able to better diagnose the issue in the future.
Post edited August 28, 2018 by Yepoleb
Posted September 01, 2018
I've just blacklisted two IPs from Belgium from accessing GOG DB because they have made over 450,000 requests combined in the past 2 weeks which is absolutely crazy. If you own 193.191.223.81 or 81.247.104.169 please consider using the backups instead of crawling the website and contact me if you want to get unblocked.
Posted September 01, 2018
avatar
Yepoleb: I've just blacklisted two IPs from Belgium from accessing GOG DB because they have made over 450,000 requests combined in the past 2 weeks which is absolutely crazy. If you own 193.191.223.81 or 81.247.104.169 please consider using the backups instead of crawling the website and contact me if you want to get unblocked.
I guess they are 3rd party websites. Normal user (or geek) will not do crazy things like this.
Posted September 01, 2018
avatar
Yepoleb: I've just blacklisted two IPs from Belgium from accessing GOG DB because they have made over 450,000 requests combined in the past 2 weeks which is absolutely crazy.
Holy s**t..
Btw, isn't it possible to automatically limit daily and monthly connections from every IP?
Post edited September 01, 2018 by phaolo
Posted September 01, 2018
avatar
phaolo: Btw, isn't it possible to automatically limit daily and monthly connections from every IP?
It is, but setting that up would take a lot more time than just manually banning them.
Posted September 01, 2018
avatar
phaolo: Btw, isn't it possible to automatically limit daily and monthly connections from every IP?
avatar
Yepoleb: It is, but setting that up would take a lot more time than just manually banning them.
Ah.. I naively thought it could have been some simple option O_o
Pages:
1
10
20
22
23
24
25
26
40
52
This is my favourite topic
Community
General discussionGOG Database - A website that collects data on GOG games(775 posts)(775 posts)
(775 posts)