I'm not so sure. If you compare the checksum on two separate downloads, to get a corrupted file, you'd need to:
- Get an error that goes undetected by the GOG servers and the client (ie, returns a 200 error code and no lower level connection error)
- The error should also not alter the size of the transmitted file (that would get detected right away)
- The error should occur twice in such a way that both times, the download have the exact same checksum

Nothing is certain in life, its about how much risk you're willing to tolerate and personally, I can live with those odds.

Main drawback would be that downloads would take twice as long and we might not be able to use the checksum as liberally as a base of comparison as we do now (might have to fallback to detecting size and file name differences which is not so bad, I'm already doing that for extras as they don't have a checksum).

Well maybe I mistook what you were saying then, and you are more the expert than me, so I will go by your judgment in that regard ... it just seemed like you were mentioning the slow onset of a slippery slope, which of course eventually speeds up.

I think this particular issue, I'll be able to dance around. It will degrade the experience for sure when you can't get the xml data (downloading the entire file just to get a redundant copy of the checksum to compare against takes time), but the end result should still be usable.

But yes, already, things are not exactly optimized to use a client efficiently. Depending on what they do (or stop maintaining well as the case may be), it could become very difficult to maintain a decent end-user experience with a client.

I hope it doesn't reach the point where mass-updating your games becomes a full-time engineering hacking job.

Finally had a break from work.

I've implemented the workaround of just getting what I need from the file download handle when the xml metadata is bad.

I'm testing it now by updating my games that need updating. Unless there are unexpected problems, I'll do a release with the fix later today.

I'll investigate the multi-region idea for Digital Ocean over the coming weeks. That will take longer. Assuming it makes a difference and I decide to proceed with that, I might take a shortcut and require a Terraform binary to be existent on the path of the system to provision on Digital Ocean (saving me the time required to learn how to use the Digital Ocean libraries directly).

Released the fix under version 0.18.0.

There is a new flag "--tolerate-bad-metadata" (or "-b" for the short version) that was added to the manifest generation and update commands.

It will add entries that had to apply the workaround to the warnings file. It will also output them on the command prompt like so:
[sdk] Bad metadata for /downloads/trash_sailors/en1patch0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/star_renegades/en1installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/ghostrunner/en1installer1: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/death_trash/en3installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/cultist_simulator_perpetual_edition_content/en3installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/source_of_madness/en1installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/cultist_simulator_the_exile/en1installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/cultist_simulator_the_priest/en1installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/cultist_simulator_the_ghoul/en1installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/cultist_simulator/en1installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/stellaris/en1patch0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/master_of_magic_caster_of_magic_for_windows/en1installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad download link for /downloads/dying_light_the_following_enhanced_edition/88945: File was not added to manifest.
[sdk] Bad download link for /downloads/dying_light_the_following_enhanced_edition/88954: File was not added to manifest.
[sdk] Bad download link for /downloads/dying_light_the_following_enhanced_edition/88951: File was not added to manifest.
[sdk] Bad download link for /downloads/dying_light_the_following_enhanced_edition/88948: File was not added to manifest.
[sdk] Bad download link for /downloads/dying_light_the_following_enhanced_edition/88891: File was not added to manifest.

As indicated by the messages, it will add significantly to manifest generation/update time as it will do an extra download of the files without metadata during that step to get the checksum, but unless the metadata gets fixed, it is what it is.

I'm glad you didn't encounter those issues. I think they don't occur on the majority of games, though when they do, they are disruptive to the "canonical" (given that we're relying on something no longer used by GOG and not officially documented, the quotes are appropriate here) way of doing things hence the workaround.

Whenever you need it, the fix is here in the last version.

I am guessing I may have now experienced the issue, as I cannot get a manifest entry for Card Shark Demo.

https://www.gog.com/en/game/card_shark_demo

When I investigate the manifest.json file, it shows the following. The original (last version) of gogrepo.py also fails.

It took several goes to get the download via the offline installer browser link, with it always going to an error page. In fact, it seemed to get trapped in a loop doing that when it finally worked.

I had to verify the file using my GOGPlus Download Checker program, with it's slower alternate method.

That's the one. You can use the latest version with the -b flag in the command to generate/update manifests and you should be good.

In retrospect, I'm thinking I should make that flag the default. I'll think about it a bit more when I next work on the tool, but I'll probably do that for the next release.

In the case of my tool, it does an empty download during manifest generation to infer the md5 when that error is encountered. Given that the md5 also computed and verified during file downloads, it ensures that the md5 is computed twice independently which, along with size verification and the integrity check of zip files, is probably the greatest assurance you'll possibly get that your file is correct in a situation like that.

No worries and thanks, I will give it a try, and I agree that making it automatic would likely be good, as in my case I would not need to update my GUI for your program, for other users.

Hello Magnitus.

You may have seen this query of mine or this one or you will at some point.

I want you to know, that I am not aiming at replacing what your update will do, just that while working on another project (which is a test of concept for yet another project), I realized I had nearly all the data I needed to create a manifest like the one gogcli.exe creates, so of course I could not resist the opportunity to create another tool for the toolbox.

Concerning what I saw in your posts:

If you use the flag I outlined (-b) for your manifest generation/update, it should work. I don't have any errors with the manifest generation/update when I use that flag. The manifest generation/update just takes longer, because the checksum of game files with bad metadata needs to be computed (you get notified about it in the output), but that's it.

I'll make that flag (-b) the default in version 0.19.0, but there is a regression bug I introduced in the main branch that I want to stabilize first before I release again. Ideally, I'd also like to cram in custom/external storage support before my next release so that I can completely decouple future storage plugings (ex: backblaze b2, aws glacier) from gogcli. I guess I could alternatively do a separate release for that.

Otherwise:

Of course I'm happy that my tool is of use to you even if it is only to make api calls or generate manifests as opposed to a full solution with storage and everything (as a devops, empowering other devs is part of what I do). However, if you find it becomes limiting and you want to do away with that dependency, that's fine too. The gui layer you made is your project and you should make the calls that you feel are best. You are in no way beholden to me to use my tool as a dependency. It is there for your convenience if you need it.

Added notes for release v0.18.0 and v0.19.0.

I think next release will finally be the one where I add external storage plugins support (via a grpc protocol) from which I'll support backblaze b2 and aws glacier with a separate s3 plugin.

The main adjustment I'll have to make to the tool (especially as it relates to the current internal s3 implementation) is limit file downloads from the storage (in order to mitigate egress costs in the cloud) particularly during file integrity checks and I'll also take the opportunity to integrate more customisable file paths.

Goodo. I've been so busy and side tracked with various projects lately that I haven't even tried your update yet, but I will shortly.

No worries, I am quite happily using gogcli.exe, I just saw a chance to add a possible alternate method to my toolbox, always good to do that for that possible rainy day ... and the joy of the experiment.

And let me thank you again, for all your great work on gogcli. :)

Released version 0.20.0.

See first post for details.