mrkgnao: two years ago, I was able to collect information about ~150 games without being blocked (roughly 2500 accesses).
phaolo: Thanks for the explanation, even if now.. the number is above what I expected.
I imagined something like 1 or 5 max, not an average of 16 O_o
I thought that Magog downloaded each game page and then processed the data offline.
Does it collect such information "by type" directly online through various passes?
The information I quoted is just for downloaded file information (installer title/name/date/size, patch title/name/date/size, bonus title/name/date/size, etc.). These (and forum) are the limited accesses. Accessing the catalogue, game pages, etc. is not limited AFAIK, or I haven't hit the limit.
To collect all the information about one game, MaGog needs to:
- read and process the catalogue page containing that game (sorted by title; 50 games per page)
- read and process the catalogue page containing that game (filtered by indie; 50 games per page)
- read and process the catalogue page containing that game (sorted by bestselling; 50 games per page)
- read and process a cart with all the GOG games in it to deduce bundle discounts (all games at once, but requires up to 40 attempts to get past GOG's server errors)
- read and process the catalogue page containing that game for each of the 25 different regions (50 games per page, repeated 25 times)
- read and process the movie catalogue so these could be filtered out later on from wishlists (50 movies per page)
- read and process 100+ wishlists, some with multiple pages (100 games per page)
- read and process the game page (1 game per page)
- read and verify the game forum link (1 game per link)
- read and process the catalogue page containing that game for each of the 3 different languages (50 games per page, repeated 3 times)
- read and process 8 GOG mixes
- read and process local information about series and packs (manually maintained by me)
- read and process public file information (1 game per page)
- read and process files in my library (1-2 accesses for every link in the library, dependent on the number of files per game; note that every OS and every language is a different link; so there are games (e.g. Witcher 2/3) that require many hundreds of accesses)
Once a week, also:
- read and process the catalogue page containing that game for each of the 200+ different countries (50 games per page, repeated 200+ times)
- read and process game length information from HowLongToBeat.com (one or more pages per game)