For gogcli, I added a bunch of examples at the end of the README.

At this point, once I've added the "storage actions-update" command to deal with gog updates occuring in the middle of a multi-steps upload (I'll explain it in the README) and uploaded my entire collection on my home servers, I'll consider it battle ready enough to for prime time and remove the WIP notice.

After that, it will just be polish (adding something for the login, improve the quality of the user feedback and in the longer term, probably add metadata and a web dashboard to view and download your games).

This should be my last detailed update post in this thread for GOGcli GUI, as I will be creating my own thread next and adding to the Wiki for it.

You can download the Second Beta Release of GOGcli GUI in the link below. Ignore the trailing 3, that is just for my records.

https://github.com/Twombs/GOGcli-GUI/releases/tag/v0.1_b03

ZIP file checking is now supported. This is fairly basic, and works on getting a count of files or folders in the root of the zip file. At this point, until I find a differently corrupted zip file to double check with, incomplete downloads have only been tested and they report '0' content with my check, even though a program like 7-Zip will report more content while also showing an error. In fact, my last check was with complete and incomplete downloads of the Blood 2 Soundtrack, and I got '0' content for the incomplete and '17' for the complete. Whereas 7-Zip reported '11' for the incomplete and '17' for the complete. So my zip check method seems fairly reasonable at this point. NOTE - Many zip files from GOG just have a single folder in the root of it. Blood 2 Soundtrack doesn't, so a good test. Please report any issues.

gogcli_selector_7.png

Log file writing has been improved. Visual improvements in the text file with more gaps, and a few more report instances.

Multiple manifest retrieval (batch of 10 or less) is now supported. Basically you select a game title on the list, then click the ADD TO MANIFEST button while holding down the CTRL key, and the game title and ID will be added to a manifests retrieval list. The title of the button will also change to MANIFEST LIST and stay that way even after a program restart until cleared. See right-click list menu options to VIEW or CLEAR. At the moment the list is limited to 10 game titles at one time. While the button is named MANIFEST LIST, you can click it to process the list ... all successful retrievals are removed from the list.

gogcli_main_7.png

Support for gogcli.exe (v0.8.0) in regard to the new download destination parameter. This means your GAMES folder can be on another drive now without a relocation penalty. Please note, that game files are initially downloaded to the root of your main GAMES folder, then relocated to the game title sub-folder(s), where size and checksum and zip testing is done. The checksum and zip testing is only done after all selected files have been downloaded.

Added a gogcli.exe version check (checksum comparison). Because of the new download destination parameter in v0.8.0 of gogcli.exe, it is important for the GUI program to know which version of gogcli.exe is being used, so that it uses the right commands for that version. The program has a checksum record (Win32 and Win64) of gogcli.exe versions up to v0.8.0. Any unrecognized version at this point is deemed v.0.9.0, so treated as having the features of v.0.8.0 at least. For those compiling their own version of gogcli.exe, they will be seen as v.0.9.0 for now, so ensure features are compatible or there could potentially be an issue.

-------------------------------------------------------

See the first post in this thread for a link to download required gogcli.exe and information about it.

-------------------------------------------------------

See the following for source and information.

https://github.com/Twombs/GOGcli-GUI

See the following and other posts here (for now) for further information.

https://www.gog.com/forum/general/gogcli_gog_client/post77

I have made a start on the Wiki at GitHub, but nothing much in it yet.

https://github.com/Twombs/GOGcli-GUI/wiki

I considered adding a "version" command before that returns the version, but thought it was a bit premature.

As I see it would be useful, I'll add it in the next release.

Great, thanks.

-----------------------------------------------------------------

While not a video tutorial, I have now added two Usage pages so far, to the Wiki for GOGcli GUI.

https://github.com/Twombs/GOGcli-GUI/wiki

That should get people started, at least until I do the rest.

Thanks .

Ok, I finally started the big download tonight and experienced some problems:

Apply stopped after:
FS: storage-s3-store.go:427: UploadFile(source=..., gameId=1590012242, kind=installer, name=setup_battle_brothers_1.4.0.47_(44153).exe) -> Uploaded file
addFileAction(gameId=1207658738, fileInfo={Kind=installer, Name=setup_earth_2140_1.0_(19593).exe, ...}, ...) -> Download file checksum of 38e4ab868b7d7e179db21938f2688b83-3 does not match expected file checksum of 5a0d53a1ed9c7e373dd527835d0ae7dc
addFileAction(gameId=1207658738, fileInfo={Kind=extra, Name=setup_earth_2140_dos_2.0.0.16.exe, ...}, ...) -> Download file checksum of a8c31896fc6007464f47e780d9a503b9-2 does not match expected file checksum of f0bba8240fada41a6e6f78eaf49b5432
addFileAction(gameId=1207660283, fileInfo={Kind=installer, Name=setup_heretic_kingdoms_1.5_(12276)-1.bin, ...}, ...) -> Download file checksum of a6f91fcd5fce83b4b4c553db0f7bb935-8 does not match expected file checksum of 554724f4420450724443d1ea24c47619
addFileAction(gameId=1590012242, fileInfo={Kind=installer, Name=setup_battle_brothers_1.4.0.47_(44153).exe, ...}, ...) -> Download file checksum of a93be7078602c59a1a621505a644e05d-10 does not match expected file checksum of 7dd67b1b1bb6feb3a1a013a0d99fd349

Resume stopped after:
addFileAction(gameId=1207658897, fileInfo={Kind=installer, Name=setup_moto_racer3_2.0.0.4.exe, ...}, ...) -> Download file checksum of ca7420ba72012c56b41dcef38fd85cd9-5 does not match expected file checksum of 00d16f1ccb49e0f5e2ec957f3a945344
addFileAction(gameId=1207659072, fileInfo={Kind=installer, Name=to_the_moon_1389114090_lang_25959.sh, ...}, ...) -> Download file checksum of ac04c107762f18f68d16dc06dc71882f-2 does not match expected file checksum of 8774a72bdb768d9c5b3c817198363a04
addFileAction(gameId=1207658815, fileInfo={Kind=installer, Name=setup_ufo_aftermath_1.4_(french)_(19448)-1.bin, ...}, ...) -> Download file checksum of f694e0c3468cc408645c5b6ef2de45ea-10 does not match expected file checksum of ca686f1a90e34ea29cb31d2d4543295c
addFileAction(gameId=1207658815, fileInfo={Kind=installer, Name=setup_ufo_aftermath_1.4_(19448)-1.bin, ...}, ...) -> Download file checksum of 77b1a8e20924d833ddef8422b09c49fd-10 does not match expected file checksum of 53eefd9b1e788cb1fdb068b0f34e9115

I strongly suspect its my relatively slow internet connection (~3.1 MB/s) coupled with having 10 concurrent downloads on large files. I'll try reducing the number of concurrent downloads. If I still experience a lot of halting after that, I'll probably have to implement retries.

Edit: Actually, I just verified against one of the large file (it seems to only fail on larger files) and the file is correct, but when the file exceeds a certain size, the computed checksum by the tool is wrong. I'm probably hitting some buffer size limit or something of the kind. I'll look at it tonight. Should be a quick fix.

Ok, the problem is resolved. It was strictly with the s3 storage, because I was using Minio's etag to get the md5 checksum.

Given that Minio follows the s3 protocol set by AWS, the checksum seems to behave differently once uploads are done in multiple parts (I think you essentially get the checsum for the last uploaded part only), which happens once the file size is bigger than 64MB, hence the error.

~70GBs of games uploaded last night without a problem.

The fix is in the main branch and will be in the 0.9.0 release.

I'll take a peek. FYI, if you add the '-d' flag to the command, you'll see the body of the reply returned by gog in the logs. If the info is in the body, it's me and I can fix it. If the info is not in the body, its gog and there is not much I can do.

Edit:

Ok, I took 10 minutes to look at it and GOG's legendary api consistency is clearly hard at work here.

The library page indicates the game runs on MacOS, but the owned games api returns this:

"
"worksOn": {
"Windows": true,
"Mac": false,
"Linux": true
},
"

Thankfully, I'm not using it to generate/filter manifests (from the owned games api, I just get the title and ids of games, I use the game details api for everything else) so it won't really affect other functionality in the tool, just this api command (which is technically faithfully returning what gog says).

I guess if that info is not accurate, I should remove it from the returned values in that command though, just so that somebody building on top of the tool is misled into thinking its reliable info (it clearly isn't).

Thanks for reporting it.

No worries, thanks for the clarity.

It's not a big issue for me, as it's what's in the manifest that really counts, and I'd probably rather you leave the data in, which is surely much more often right, than wrong ... at least I hope so ... haven't noticed any others ... yet.

If you start removing GOG information because it's inaccurate, you will quickly remain with no information at all. Trust me on this. The fact that you find a discrepancy in one game in one direction, does not mean that it does not exist for another game in the opposite direction. From my experience, it's best to program defensively when it comes to GOG. If you have two sources of the same information, use both and implement a sensible algorithm for the case where they disagree.

Just my 2p.

I just found out about this project. I'm glad it exists. Best of luck for you.

Well, in that particular case, one source of information (game details) is used by the bookshelf (likely validated by thousands of people) and the other source of information is used... I don't know where (probably nowhere, so its essentially validated by nobody, at least until now :P).

In my experience, there are 2 types of assurances in software:
1. What is tested
2. What is used

The first assurance, if done very well, tends to be the most reliable, but its only as good as the test coverage (its very much like the validity of a statistical experiment which is only as good as the sample used in the experiment)

The second assurance, by itself, is horrible to protect against regression bugs, but still has value for long term stability if combined with stable releases and if used broadly by a lot of people, can complete the first category well (especially if edge case bugs that are reported by users are integrated into the test suite).

Anything that is not validated by at least one of those two things has a tendency to decay and become less and less usable over time.

I experienced this at a previous shop I worked in where 200 employees were contributing to a codebase and I was the only one using certain features to validate on my local machine that some things still worked as intended in production (always better to find out on your local environment then to find out when you deploy in production). Because nobody else was using it by me, other people broke it all the time.

Thanks :).

I've now started my own thread for GOGcli GUI, which can be found at the following link.

https://www.gog.com/forum/general/gogcli_gui_frontend_downloader_and_validator

I've also done another upload, Fourth Beta Release, which has some great new features and a minor bugfix.

https://github.com/Twombs/GOGcli-GUI/releases/tag/v0.1_b08

I should be continuing work on the multiple game downloading ability next.

Version 0.9.0 is out. See the first post for details. There are also examples at the end of the README that should cover most common use-cases.

At this point, I'm well underway with my backup (~205GB of games files for 300+ games backed up so far) which I'm doing across several iterations (usually at night when I go to sleep as I try not to clog the household bandwidth too much during the day) and am satisfied that the tool is stable.

There are still rough edges that will need smoothening (see whats left to do). However, these will all be backward compatible addons.

Should I ever have to do a backward-incompatible change on the storage format, I'll provide a migration command in the tool as I have my own backup to manage and I won't redownload terabytes of game files.