Yeah, thinking about it, it might have simplified some things, oh well.

ListView... vaguely reminds me of when I was doing Windows GUI in C#. You're bringing back memories.

Unless something goes horribly wrong, I should be able to do my next release (zip validation & all file having verified size info in manifest) today or tomorrow (working has been hectic this week, insane rampup to deploy a robust etcd cluster with all the bells and whistles, lots of learning on my off time).

After that, I'll cram json/xml file output in the api and a single list for owned games in the release after (which will be about patchfiles for new games and game updates), hopefully this weekend as well, but we'll see.

Sounds good.

Did a bit more work on the GUI this morning, and got some things working beyond the visual, but no downloading yet.

Created a repository for it at GitHub - https://github.com/Twombs/GOGcli-GUI

Here's the main window screenshot of what I have done so far .... of course it will be subject to change.

gogcli_main.png

NOTE - A red row indicates that game has an update.

Just wondering. Is the DLC code not complete yet, as I don't appear to have any games showing a value that isn't '0', and of course many of my games on that list have DLC?

So if I'm understanding you correctly the data collection including the md5 checksums is already finished, right?

How long does it take with you program? Can you give some data like for number of games = X and settings 'collect info on installers, patches and extras for windows in the languages A and B' resulting in number of files = Y, the time needed to get all the data is X:XX?

Since I'm also writing a downloader of my own I'm interested in such data to know if there is a possibility for optimization.

I haven't done it for my entire list of games yet. I don't want to get rid of the "updated" and "new" indicators just yet in my gaming library, because I haven't implemented patch files yet for updates and new games (and I want some sample data for that).

Anyways, the real bottleneck here shouldn't be the client implementation, its the gog api. In my case, I set a variable number of concurrent workers (10 by default) and a pause between fetches (200ms by default), because if I retrieve my entire game collection, which would entail thousands of requests, and all those requests fly at once... my client would probably handle it no problem, but GOG.com might flag it as a DOS attack and I don't want to get blocked.

Version 0.4.0 is out. See first post for details.

Looks nice.

I'm not sure, I'll have to take a look. Unless I have a bug, I'd say the dlc flag is whatever is returned by the gog api. I don't do any fancy logic there.

FYI, I have long term plans to add more metadata (whatever I can scrape from the GOG api really) to eventually have a LAN web dashboard to download your games from a backup.

However, I don't want to couple that with the main backup. All that info will be generated and stored in a separate metadata directory structure.

Excellent.

Thanks.

Done a bit more work on it and got a few more things working, mostly based on my GOGRepo GUI code, but making a few changes or adaptions. You can now view and download the cover image, as well as visit the online game page. As always it is a matter of structure building for me first, before I get down to real specifics, as that often dictates what I have to account for, and sometimes nice ideas come up, like the red row for updated games, which occurred to me after deciding to use a ListView instead. At the moment I am focusing on the practical for testing too, so some of what is shown may change or disappear ... perhaps going to another window in the process.

Here's the latest screenshot of my work in progress.

gogcli_main_2.png

No worries. Be a shame if they haven't supported it properly, but it ain't a big deal and I get to reclaim some window real estate. Not that what I am currently showing is the final visual. It's all rather like a jigsaw + juggling act.

Bloody hell, they return malformed xml for some files and I need to deal with that >< :

<chunk id="117" from="1226833920" to="1237319679" method="md5">61af8627a00387f14dc7158a3921ab24</chunk>
<chunk id="118" from="1237319680"to="1247805439" method="md5">75b9e15c1b7091137e7c94e5abeb805e</chunk>
<chunk id="119" from="1247805440" to="1258291199" method="md5">562371bea2fe4593983b3c7fdd6f8a60</chunk>

Edit:

This is pretty bad, the listed english patch for craft the world also returns a 404 (also validated from the library page):

{
"Errors": [
"GetDownloadInfo(downloadPath=/downloads/craft_the_world/en1patch1) -\u003e Expected response status code of 200, but got 404"
]
}

Released v0.5.0 and v0.5.1

Will probably have to add some kind of fault tolerance that will require user review once manifests are generated as gog seems to have 404 links for some download files (I guess if that's the way gog operates, the manifest should still get generated, but at the same time, its probably a bad idea to just overlook this without warning the user).

Edit: Just realized that I removed the filters in manifest generation in patch v0.5.1 I believe. I'll do a v0.5.2 release to add them back. I'll actually improve the game title search as I find that the search provided by the gog api is so so.

Edit2: Republique (a game that appears to be no longer in the game store) has a manual extra with "(coming soon)" in the title that returns a 403 in the download url :...(

Edit3: Ok, finally got a manifest with "all" my Windows and Linux games, though I'm missing about 70 games in the manifest or so. will have to investigate why.

v0.6.0 released.

v0.7.0 is out.

Starting to look like a stable tool.

Probably won't have all the bells and whistles everyone was hoping for right out the gate (that will take more time), but I think odds are very good I'll be able to use it to start my backup in Minio this weekend.

I successfully generated my full manifest and the downloads I've attempted so far (all modest few GBs of files at most so far) worked well.

Excellent. I need to grab your latest ... been sitting on version 0.4 for my testing.

---------------------------------------------------------------

For those who might like to test (play with) my GUI as it is, all but the DOWNLOAD button currently works.
If you can manage to get the needed cookie data, you are good to go, and I have provided an EXE for the GUI to make testing easier.
I recommend either following the instructions that Magnitus provides for cookie data or using a browser addon that saves to a text file, from which you can copy the required line(s) to paste into a Cookie.txt file.

https://github.com/Twombs/GOGcli-GUI

Here's a screenshot of the SETUP window. Not much to it at this point ... mostly just text.

gogcli_setup.png

Things to note.

I have only tested with gogcli.exe version 0.4 and earlier.

Minimal testing so far.

ADD TO MANIFEST has other CTRL and SHIFT key options when clicked.
Those relate to viewing Manifest.txt and manifest.json with Wordpad.

For the moment, Manifest.txt file is a compilation of every created manifest.json file. The manifest.json file gets created every time you click the ADD TO MANIFEST button for the selected game.

If you only want to see the status of current process in the Cover preview field, and not see a console window then select the minimize option on the SETUP window.

Cover images (thumbnail) are either downloaded on demand if you have the 'Show' option enabled, or if you have the save option selected on the SETUP window they get stored for future use and offline access.

The 'Download Cover' button downloads a large image file for each selected game.

Nothing yet writes to the Log.txt file.

All pretty basic right now, but I will be working on some kind of game download support next. I also need to check out the more recent improvements to gogcli.exe.

I'll tackle this in increments. kohlrak suggested (after you've logged in to GOG.com) setting up a proxy: having a command that listens on port 8080, setting localhost:8080 as your internet proxy, then going to [url=http://www.gog.com:8080]http://www.gog.com:8080[/url] which would generate the cookie file and then undoing the proxy. I could implement this quickly as its fairly easy to do.

Still not as user-friendly as putting a wrapper around the login, but some people might find is simpler than manually grabbing values from their cookie (I'm not sure).

Then, of particular interest to you:

Most api commands will now return optional json format which you might find easier to digest. It can also, optionally, output to a file as opposed to the terminal.

There's also a manifest summary command that doesn't compute anything you couldn't have computed yourself from the summary, but you might find it conveninent...

The filter that was used to generate your manifest is now stored inside your manifest which I think is very sound for bookkeeping (if ever you are unsure what filter was used for your backup and also during upgrades which I'll implement soon, the same filter will be re-applied when adding new games and updates without the user having remember and re-input it).

And finally, the manifest generation title filter has been overhauled. It is a tad slower as it doesn't use the search feature from the gog api (which only works with prefixes and not perfectly I find), but in exchange, you get:
- The ability to put more than a single title term to check for
- Your search terms matching titles that contain the term, but not necessarily at the beginning of the title
- Something that behaves as you would expect all the time as opposed to most of the time

I haven't read your code yet but since you were talking about getting file sizes for extras using HEAD requests I got the impression you don't know about this part of the API
https://api.gog.com/products/PRODUCT_ID_HERE?expand=downloads,expanded_dlcs,description,screenshots,videos,related_products,changelog&amp;locale=en-US
for example Witcher 3 GOTY

["downloads"]["bonus_content"][i]["files"][j]["size"]
Shows file size for extra in bytes

From that same json you can follow the downlink url to get links to the file (and xml data for installers)
["downloads"]["installers"][i]["files"][j]["downlink"]

Lots of stuff about login and getting cookies (hopefully GOG forum software doesn't mess up the formatting too badly)

Login form has stayed the same for over 4 years
So far the only change that has mattered for login flow has been that GOG added galaxy login to it

Get login form
https://auth.gog.com/auth?client_id=46899977096215655&amp;redirect_uri=https://embed.gog.com/on_login_success?origin=client&amp;response_type=code&amp;layout=default&amp;brand=gog

Check for recaptcha. If login form contains recaptcha do the login using some web engine and monitor "load finished" events for https://embed.gog.com/on_login_success
Then extract the auth code from it.
Extract and save cookies from web engine.
Get and save galaxy tokens json
https://auth.gog.com/token?client_id=46899977096215655&amp;client_secret=9d85c43b1482497dbbce61f6e4aa173a433796eeae2ca8c5f6129f2dc4de46d9&amp;grant_type=authorization_code&amp;code=AUTH_CODE_HERE&amp;redirect_uri=https://embed.gog.com/on_login_success?origin=client

If no recaptcha was found then do the following

Get login token value from input tag with id "login__token", also get the name of the tag
POST these values to https://login.gog.com/login_check
login[username]=USERNAME_HERE&login[password]=USER_PASSWORD_HERE&login[login]=&TOKEN_TAG_NAME_HERE =LOGIN_TOKEN _HERE

Check but don't follow the redirect uri you get from login_check
If it contains "two_step" then ask user to input the 4 character code
Get the two step form using the redirect uri you got earlier and get two step authentication token and its tag name (input tag with id "second_step_authentication__token")

Then POST these values to https://login.gog.com/login/two_step
second_step_authentication[token][letter_1]=CHAR_1_HERE&second_step_authentication[token][letter_2]=CHAR_2 _HER E&second_step_authentication[token][letter_3]=CHAR_3_HERE&second_step_authentication[token][letter_4]= CHAR_4_H ERE&second_step_authentication[send]=&AUTH_TOKEN_TAG_NAME_HERE=AUTH_TOKEN_HERE

Extract auth code from redirect uri and follow redirects until finished

Check that login was successful by trying to access something that requires login like account page https://www.gog.com/account
Save cookies.
Get and save galaxy tokens json
https://auth.gog.com/token?client_id=46899977096215655&amp;client_secret=9d85c43b1482497dbbce61f6e4aa173a433796eeae2ca8c5f6129f2dc4de46d9&amp;grant_type=authorization_code&amp;code=AUTH_CODE_HERE&amp;redirect_uri=https://embed.gog.com/on_login_success?origin=client

See GuiLogin class and Website::Login from lgogdownloader for details
https://github.com/Sude-/lgogdownloader/blob/e43d1c4/src/gui_login.cpp
https://github.com/Sude-/lgogdownloader/blob/e43d1c4/src/website.cpp#L254

Useful links
Yepoleb's documentation about Galaxy API
Comet - Yepoleb's minimal Galaxy client to allow games to use the Galaxy multiplayer service on Linux
Wyvern and gog-rs - Rust implementation of downloader using Galaxy API
Wyvern
gog-rs
lgogdownloader - my implementation of downloader using Galaxy API written in C++

edit: well GOG forum software butchered the formatting as expected :(
check lgogdownloader's login implementation for details

edit2: tried fixing the formatting but GOG forum software just doesn't want to cooperate

Thanks for all that great info! I'll peruse through it tonight and how I can improve the design and/or user experience.

For the login, the tricky part for me is the user-interface. Maybe I'm mistaken, but I wasn't under the impression that you can just resolve it with a web form in a browser using something like a frame.

If you need to load the login form into some kind of framework (like Electron does), then platform compatibility comes into play. Something I like very much about the tool right now is the wide range of platform it supports (Linux amd64 & arm64, MacOS, Windows adm64 and even Windows 386). I have a feeling that if I integrate some kind of UI framework into the tool, that might shrink that compatibility list so I want to be somewhat careful there.

One thing I though about is making a separate electron app just to manage the login (on platforms that can't run it, you can still fall back to manually creating a cookie file).

But I might be mistaken, maybe I can spin up a small web server (something that is very platform agnostic) that serves the GOG login in a frame and successfully get the info.

I need to spend more time looking at it honestly (and digest some of the great info you provided here). So far, the bulk of my energy was spent on what I consider the core functionality of the tool and I haven't given the login the attention that it needs. I need to implement a proper poc there and see what works.

Sounds good. Thanks.

The last time I checked (though that was years ago with a browser) the GOG servers only allowed 6 connections at a time so 10 workers might be a bit too much. It doesn't hurt though the connections over the limit will simply get delayed until one of the earlier connections has ended.

And while the GOG API may be a bottleneck the question is always how close you can get to the restrictions set by the bottleneck with your implementation. My data collection is a bit more than 4 times faster than gogrepo and I haven't even done any optimization yet. So obviously I'm interested to know if there is still room for improvement.

Edit: I just checked and now the servers allow more than 6 connections.