It seems that you're using an outdated browser. Some things may not work as they should (or don't work at all).
We suggest you upgrade newer and better browser like: Chrome, Firefox, Internet Explorer or Opera

×
Community
General discussiongogcli - Gog Client(295 posts)(295 posts)
(295 posts)
Pages:
1
2
3
4
5
6
10
15
20
This is my favourite topic
Posted February 26, 2021
avatar
Timboli: Ha ha ha. Don't mind me, do what you feel you have to. I guess I was just stating the obvious anyway.
Also another reason from the coder's perspective to only deal with one downloading thread. ;)
Yeah, thinking about it, it might have simplified some things, oh well.

avatar
Timboli: I found my old code from 2016, which I hope will be sufficient to this purpose.
Not had a chance to check it yet, and so with the aim of making that easier, I have tentatively started work on GOGcli GUI. Spent the morning getting the basics done on that, pinching code from various other programs of mine, especially GOGRepo GUI. I decided to use a ListView instead of ListBox this time around for Game Titles, with a hidden Game ID column, just to simplify an aspect or two ... if you can call it simplifying using a ListView rather than a simpler ListBox. :)
ListView... vaguely reminds me of when I was doing Windows GUI in C#. You're bringing back memories.

Unless something goes horribly wrong, I should be able to do my next release (zip validation & all file having verified size info in manifest) today or tomorrow (working has been hectic this week, insane rampup to deploy a robust etcd cluster with all the bells and whistles, lots of learning on my off time).

After that, I'll cram json/xml file output in the api and a single list for owned games in the release after (which will be about patchfiles for new games and game updates), hopefully this weekend as well, but we'll see.
Post edited February 26, 2021 by Magnitus
Posted February 27, 2021
avatar
Magnitus: ListView... vaguely reminds me of when I was doing Windows GUI in C#. You're bringing back memories.

Unless something goes horribly wrong, I should be able to do my next release (zip validation & all file having verified size info in manifest) today or tomorrow (working has been hectic this week, insane rampup to deploy a robust etcd cluster with all the bells and whistles, lots of learning on my off time).

After that, I'll cram json/xml file output in the api and a single list for owned games in the release after (which will be about patchfiles for new games and game updates), hopefully this weekend as well, but we'll see.
Sounds good.

Did a bit more work on the GUI this morning, and got some things working beyond the visual, but no downloading yet.

Created a repository for it at GitHub - https://github.com/Twombs/GOGcli-GUI

Here's the main window screenshot of what I have done so far .... of course it will be subject to change.

gogcli_main.png

NOTE - A red row indicates that game has an update.

Just wondering. Is the DLC code not complete yet, as I don't appear to have any games showing a value that isn't '0', and of course many of my games on that list have DLC?
Post edited March 07, 2021 by Timboli
Posted February 27, 2021
avatar
Magnitus: [...]
So if I'm understanding you correctly the data collection including the md5 checksums is already finished, right?

How long does it take with you program? Can you give some data like for number of games = X and settings 'collect info on installers, patches and extras for windows in the languages A and B' resulting in number of files = Y, the time needed to get all the data is X:XX?

Since I'm also writing a downloader of my own I'm interested in such data to know if there is a possibility for optimization.
Posted February 27, 2021
avatar
Geralt_of_Rivia: So if I'm understanding you correctly the data collection including the md5 checksums is already finished, right?

How long does it take with you program? Can you give some data like for number of games = X and settings 'collect info on installers, patches and extras for windows in the languages A and B' resulting in number of files = Y, the time needed to get all the data is X:XX?

Since I'm also writing a downloader of my own I'm interested in such data to know if there is a possibility for optimization.
I haven't done it for my entire list of games yet. I don't want to get rid of the "updated" and "new" indicators just yet in my gaming library, because I haven't implemented patch files yet for updates and new games (and I want some sample data for that).

Anyways, the real bottleneck here shouldn't be the client implementation, its the gog api. In my case, I set a variable number of concurrent workers (10 by default) and a pause between fetches (200ms by default), because if I retrieve my entire game collection, which would entail thousands of requests, and all those requests fly at once... my client would probably handle it no problem, but GOG.com might flag it as a DOS attack and I don't want to get blocked.
Posted February 27, 2021
Version 0.4.0 is out. See first post for details.

avatar
Timboli: Sounds good.

Did a bit more work on the GUI this morning, and got some things working beyond the visual, but no downloading yet.

Created a repository for it at GitHub - https://github.com/Twombs/GOGcli-GUI

Here's the main window screenshot of what I have done so far .... of course it will be subject to change.

https://user-images.githubusercontent.com/19166041/109370993-af18f800-78f2-11eb-933b-51910c74648a.png

NOTE - A red row indicates that game has an update.
Looks nice.

avatar
Timboli: Just wondering. Is the DLC code not complete yet, as I don't appear to have any games showing a value that isn't '0', and of course many of my games on that list have DLC?
I'm not sure, I'll have to take a look. Unless I have a bug, I'd say the dlc flag is whatever is returned by the gog api. I don't do any fancy logic there.

FYI, I have long term plans to add more metadata (whatever I can scrape from the GOG api really) to eventually have a LAN web dashboard to download your games from a backup.

However, I don't want to couple that with the main backup. All that info will be generated and stored in a separate metadata directory structure.
Post edited February 27, 2021 by Magnitus
Posted February 28, 2021
avatar
Magnitus: Version 0.4.0 is out. See first post for details.
Excellent.

avatar
Magnitus: Looks nice.
Thanks.

Done a bit more work on it and got a few more things working, mostly based on my GOGRepo GUI code, but making a few changes or adaptions. You can now view and download the cover image, as well as visit the online game page. As always it is a matter of structure building for me first, before I get down to real specifics, as that often dictates what I have to account for, and sometimes nice ideas come up, like the red row for updated games, which occurred to me after deciding to use a ListView instead. At the moment I am focusing on the practical for testing too, so some of what is shown may change or disappear ... perhaps going to another window in the process.

Here's the latest screenshot of my work in progress.

gogcli_main_2.png

avatar
Magnitus: I'm not sure, I'll have to take a look. Unless I have a bug, I'd say the dlc flag is whatever is returned by the gog api. I don't do any fancy logic there.
No worries. Be a shame if they haven't supported it properly, but it ain't a big deal and I get to reclaim some window real estate. Not that what I am currently showing is the final visual. It's all rather like a jigsaw + juggling act.
Post edited March 07, 2021 by Timboli
Posted February 28, 2021
Bloody hell, they return malformed xml for some files and I need to deal with that >< :

<chunk id="117" from="1226833920" to="1237319679" method="md5">61af8627a00387f14dc7158a3921ab24</chunk>
<chunk id="118" from="1237319680"to="1247805439" method="md5">75b9e15c1b7091137e7c94e5abeb805e</chunk>
<chunk id="119" from="1247805440" to="1258291199" method="md5">562371bea2fe4593983b3c7fdd6f8a60</chunk>

Edit:

This is pretty bad, the listed english patch for craft the world also returns a 404 (also validated from the library page):

{
"Errors": [
"GetDownloadInfo(downloadPath=/downloads/craft_the_world/en1patch1) -\u003e Expected response status code of 200, but got 404"
]
}
Post edited February 28, 2021 by Magnitus
Posted February 28, 2021
Released v0.5.0 and v0.5.1

Will probably have to add some kind of fault tolerance that will require user review once manifests are generated as gog seems to have 404 links for some download files (I guess if that's the way gog operates, the manifest should still get generated, but at the same time, its probably a bad idea to just overlook this without warning the user).

Edit: Just realized that I removed the filters in manifest generation in patch v0.5.1 I believe. I'll do a v0.5.2 release to add them back. I'll actually improve the game title search as I find that the search provided by the gog api is so so.

Edit2: Republique (a game that appears to be no longer in the game store) has a manual extra with "(coming soon)" in the title that returns a 403 in the download url :...(

Edit3: Ok, finally got a manifest with "all" my Windows and Linux games, though I'm missing about 70 games in the manifest or so. will have to investigate why.

v0.6.0 released.
Post edited March 01, 2021 by Magnitus
Posted March 02, 2021
v0.7.0 is out.

Starting to look like a stable tool.

Probably won't have all the bells and whistles everyone was hoping for right out the gate (that will take more time), but I think odds are very good I'll be able to use it to start my backup in Minio this weekend.

I successfully generated my full manifest and the downloads I've attempted so far (all modest few GBs of files at most so far) worked well.
Post edited March 02, 2021 by Magnitus
Posted March 02, 2021
avatar
Magnitus: v0.7.0 is out.

Starting to look like a stable tool.
Excellent. I need to grab your latest ... been sitting on version 0.4 for my testing.

---------------------------------------------------------------

For those who might like to test (play with) my GUI as it is, all but the DOWNLOAD button currently works.
If you can manage to get the needed cookie data, you are good to go, and I have provided an EXE for the GUI to make testing easier.
I recommend either following the instructions that Magnitus provides for cookie data or using a browser addon that saves to a text file, from which you can copy the required line(s) to paste into a Cookie.txt file.

https://github.com/Twombs/GOGcli-GUI

Here's a screenshot of the SETUP window. Not much to it at this point ... mostly just text.

gogcli_setup.png

Things to note.

I have only tested with gogcli.exe version 0.4 and earlier.

Minimal testing so far.

ADD TO MANIFEST has other CTRL and SHIFT key options when clicked.
Those relate to viewing Manifest.txt and manifest.json with Wordpad.

For the moment, Manifest.txt file is a compilation of every created manifest.json file. The manifest.json file gets created every time you click the ADD TO MANIFEST button for the selected game.

If you only want to see the status of current process in the Cover preview field, and not see a console window then select the minimize option on the SETUP window.

Cover images (thumbnail) are either downloaded on demand if you have the 'Show' option enabled, or if you have the save option selected on the SETUP window they get stored for future use and offline access.

The 'Download Cover' button downloads a large image file for each selected game.

Nothing yet writes to the Log.txt file.

All pretty basic right now, but I will be working on some kind of game download support next. I also need to check out the more recent improvements to gogcli.exe.
Post edited March 07, 2021 by Timboli
Posted March 02, 2021
avatar
Timboli: For those who might like to test (play with) my GUI as it is, all but the DOWNLOAD button currently works.
If you can manage to get the needed cookie data, you are good to go, and I have provided an EXE for the GUI to make testing easier.
I recommend either following the instructions that Magnitus provides for cookie data or using a browser addon that saves to a text file, from which you can copy the required line(s) to paste into a Cookie.txt file.
I'll tackle this in increments. kohlrak suggested (after you've logged in to GOG.com) setting up a proxy: having a command that listens on port 8080, setting localhost:8080 as your internet proxy, then going to [url=http://www.gog.com:8080]http://www.gog.com:8080[/url] which would generate the cookie file and then undoing the proxy. I could implement this quickly as its fairly easy to do.

Still not as user-friendly as putting a wrapper around the login, but some people might find is simpler than manually grabbing values from their cookie (I'm not sure).

avatar
Timboli: I have only tested with gogcli.exe version 0.4 and earlier.
Then, of particular interest to you:

Most api commands will now return optional json format which you might find easier to digest. It can also, optionally, output to a file as opposed to the terminal.

There's also a manifest summary command that doesn't compute anything you couldn't have computed yourself from the summary, but you might find it conveninent...

The filter that was used to generate your manifest is now stored inside your manifest which I think is very sound for bookkeeping (if ever you are unsure what filter was used for your backup and also during upgrades which I'll implement soon, the same filter will be re-applied when adding new games and updates without the user having remember and re-input it).

And finally, the manifest generation title filter has been overhauled. It is a tad slower as it doesn't use the search feature from the gog api (which only works with prefixes and not perfectly I find), but in exchange, you get:
- The ability to put more than a single title term to check for
- Your search terms matching titles that contain the term, but not necessarily at the beginning of the title
- Something that behaves as you would expect all the time as opposed to most of the time
Post edited March 02, 2021 by Magnitus
Posted March 02, 2021
I haven't read your code yet but since you were talking about getting file sizes for extras using HEAD requests I got the impression you don't know about this part of the API
https://api.gog.com/products/PRODUCT_ID_HERE?expand=downloads,expanded_dlcs,description,screenshots,videos,related_products,changelog&amp;locale=en-US
for example Witcher 3 GOTY

["downloads"]["bonus_content"][i]["files"][j]["size"]
Shows file size for extra in bytes

From that same json you can follow the downlink url to get links to the file (and xml data for installers)
["downloads"]["installers"][i]["files"][j]["downlink"]

Lots of stuff about login and getting cookies (hopefully GOG forum software doesn't mess up the formatting too badly)

Login form has stayed the same for over 4 years
So far the only change that has mattered for login flow has been that GOG added galaxy login to it

Get login form
https://auth.gog.com/auth?client_id=46899977096215655&amp;redirect_uri=https://embed.gog.com/on_login_success?origin=client&amp;response_type=code&amp;layout=default&amp;brand=gog

Check for recaptcha. If login form contains recaptcha do the login using some web engine and monitor "load finished" events for https://embed.gog.com/on_login_success
Then extract the auth code from it.
Extract and save cookies from web engine.
Get and save galaxy tokens json
https://auth.gog.com/token?client_id=46899977096215655&amp;client_secret=9d85c43b1482497dbbce61f6e4aa173a433796eeae2ca8c5f6129f2dc4de46d9&amp;grant_type=authorization_code&amp;code=AUTH_CODE_HERE&amp;redirect_uri=https://embed.gog.com/on_login_success?origin=client

If no recaptcha was found then do the following

Get login token value from input tag with id "login__token", also get the name of the tag
POST these values to https://login.gog.com/login_check
login[username]=USERNAME_HERE&login[password]=USER_PASSWORD_HERE&login[login]=&TOKEN_TAG_NAME_HERE =LOGIN_TOKEN _HERE

Check but don't follow the redirect uri you get from login_check
If it contains "two_step" then ask user to input the 4 character code
Get the two step form using the redirect uri you got earlier and get two step authentication token and its tag name (input tag with id "second_step_authentication__token")

Then POST these values to https://login.gog.com/login/two_step
second_step_authentication[token][letter_1]=CHAR_1_HERE&second_step_authentication[token][letter_2]=CHAR_2 _HER E&second_step_authentication[token][letter_3]=CHAR_3_HERE&second_step_authentication[token][letter_4]= CHAR_4_H ERE&second_step_authentication[send]=&AUTH_TOKEN_TAG_NAME_HERE=AUTH_TOKEN_HERE

Extract auth code from redirect uri and follow redirects until finished

Check that login was successful by trying to access something that requires login like account page https://www.gog.com/account
Save cookies.
Get and save galaxy tokens json
https://auth.gog.com/token?client_id=46899977096215655&amp;client_secret=9d85c43b1482497dbbce61f6e4aa173a433796eeae2ca8c5f6129f2dc4de46d9&amp;grant_type=authorization_code&amp;code=AUTH_CODE_HERE&amp;redirect_uri=https://embed.gog.com/on_login_success?origin=client

See GuiLogin class and Website::Login from lgogdownloader for details
https://github.com/Sude-/lgogdownloader/blob/e43d1c4/src/gui_login.cpp
https://github.com/Sude-/lgogdownloader/blob/e43d1c4/src/website.cpp#L254

Useful links
Yepoleb's documentation about Galaxy API
Comet - Yepoleb's minimal Galaxy client to allow games to use the Galaxy multiplayer service on Linux
Wyvern and gog-rs - Rust implementation of downloader using Galaxy API
Wyvern
gog-rs
lgogdownloader - my implementation of downloader using Galaxy API written in C++

edit: well GOG forum software butchered the formatting as expected :(
check lgogdownloader's login implementation for details

edit2: tried fixing the formatting but GOG forum software just doesn't want to cooperate
Post edited March 02, 2021 by Sude
Posted March 02, 2021
avatar
Sude: ...
Thanks for all that great info! I'll peruse through it tonight and how I can improve the design and/or user experience.

For the login, the tricky part for me is the user-interface. Maybe I'm mistaken, but I wasn't under the impression that you can just resolve it with a web form in a browser using something like a frame.

If you need to load the login form into some kind of framework (like Electron does), then platform compatibility comes into play. Something I like very much about the tool right now is the wide range of platform it supports (Linux amd64 & arm64, MacOS, Windows adm64 and even Windows 386). I have a feeling that if I integrate some kind of UI framework into the tool, that might shrink that compatibility list so I want to be somewhat careful there.

One thing I though about is making a separate electron app just to manage the login (on platforms that can't run it, you can still fall back to manually creating a cookie file).

But I might be mistaken, maybe I can spin up a small web server (something that is very platform agnostic) that serves the GOG login in a frame and successfully get the info.

I need to spend more time looking at it honestly (and digest some of the great info you provided here). So far, the bulk of my energy was spent on what I consider the core functionality of the tool and I haven't given the login the attention that it needs. I need to implement a proper poc there and see what works.
Post edited March 02, 2021 by Magnitus
Posted March 02, 2021
avatar
Magnitus: I'll tackle this in increments...........

Still not as user-friendly as putting a wrapper around the login, but some people might find is simpler than manually grabbing values from their cookie (I'm not sure).

Then, of particular interest to you .........
Sounds good. Thanks.
Posted March 03, 2021
avatar
Geralt_of_Rivia: So if I'm understanding you correctly the data collection including the md5 checksums is already finished, right?

How long does it take with you program? Can you give some data like for number of games = X and settings 'collect info on installers, patches and extras for windows in the languages A and B' resulting in number of files = Y, the time needed to get all the data is X:XX?

Since I'm also writing a downloader of my own I'm interested in such data to know if there is a possibility for optimization.
avatar
Magnitus: I haven't done it for my entire list of games yet. I don't want to get rid of the "updated" and "new" indicators just yet in my gaming library, because I haven't implemented patch files yet for updates and new games (and I want some sample data for that).

Anyways, the real bottleneck here shouldn't be the client implementation, its the gog api. In my case, I set a variable number of concurrent workers (10 by default) and a pause between fetches (200ms by default), because if I retrieve my entire game collection, which would entail thousands of requests, and all those requests fly at once... my client would probably handle it no problem, but GOG.com might flag it as a DOS attack and I don't want to get blocked.
The last time I checked (though that was years ago with a browser) the GOG servers only allowed 6 connections at a time so 10 workers might be a bit too much. It doesn't hurt though the connections over the limit will simply get delayed until one of the earlier connections has ended.

And while the GOG API may be a bottleneck the question is always how close you can get to the restrictions set by the bottleneck with your implementation. My data collection is a bit more than 4 times faster than gogrepo and I haven't even done any optimization yet. So obviously I'm interested to know if there is still room for improvement.

Edit: I just checked and now the servers allow more than 6 connections.
Post edited March 03, 2021 by Geralt_of_Rivia
Pages:
1
2
3
4
5
6
10
15
20
This is my favourite topic
Community
General discussiongogcli - Gog Client(295 posts)(295 posts)
(295 posts)