BIG THANKS. Grabbed it. :)

Fyi, for those who had Elderscroll Arena & Daggerfall, just noticed that they were removed from my collection.

From I plan file I created:

"1435828982": {
"Title": "The Elder Scrolls: Arena",
"Slug": "the_elder_scrolls_arena",
"Id": 1435828982,
"Action": "remove",
"InstallerActions": {
"setup_the_elder_scrolls_arena_1.07_(28043).exe": {
"Title": "The Elder Scrolls: Arena",
"Name": "setup_the_elder_scrolls_arena_1.07_(28043).exe",
"Url": "/downloads/the_elder_scrolls_arena/en1installer0",
"Kind": "installer",
"Action": "remove"
}
},
"ExtraActions": {
"arena_codex_scientia.zip": {
"Title": "Codex Scientia",
"Name": "arena_codex_scientia.zip",
"Url": "/downloads/the_elder_scrolls_arena/59153",
"Kind": "extra",
"Action": "remove"
},
"arena_manual.zip": {
"Title": "manual",
"Name": "arena_manual.zip",
"Url": "/downloads/the_elder_scrolls_arena/59123",
"Kind": "extra",
"Action": "remove"
},
"arena_player_guide.zip": {
"Title": "playguide",
"Name": "arena_player_guide.zip",
"Url": "/downloads/the_elder_scrolls_arena/59133",
"Kind": "extra",
"Action": "remove"
},
"arena_quick_ref.zip": {
"Title": "quick reference card",
"Name": "arena_quick_ref.zip",
"Url": "/downloads/the_elder_scrolls_arena/59143",
"Kind": "extra",
"Action": "remove"
}
}
},
"1435829353": {
"Title": "The Elder Scrolls II: Daggerfall",
"Slug": "the_elder_scrolls_chapter_ii_daggerfall",
"Id": 1435829353,
"Action": "remove",
"InstallerActions": {
"setup_the_elder_scrolls_ii_daggerfall_1.07_(28043).exe": {
"Title": "The Elder Scrolls II: Daggerfall",
"Name": "setup_the_elder_scrolls_ii_daggerfall_1.07_(28043).exe",
"Url": "/downloads/the_elder_scrolls_chapter_ii_daggerfall/en1installer0",
"Kind": "installer",
"Action": "remove"
}
},
"ExtraActions": {
"daggerfall_chronicles.zip": {
"Title": "The Daggerfall Chronicles",
"Name": "daggerfall_chronicles.zip",
"Url": "/downloads/the_elder_scrolls_chapter_ii_daggerfall/59183",
"Kind": "extra",
"Action": "remove"
},
"daggerfall_manual.zip": {
"Title": "manual",
"Name": "daggerfall_manual.zip",
"Url": "/downloads/the_elder_scrolls_chapter_ii_daggerfall/59163",
"Kind": "extra",
"Action": "remove"
},
"daggerfall_ref_card.zip": {
"Title": "reference card",
"Name": "daggerfall_ref_card.zip",
"Url": "/downloads/the_elder_scrolls_chapter_ii_daggerfall/59173",
"Kind": "extra",
"Action": "remove"
}
}
},

I was able to re-add them to my collection as free games, but a bit startling that they were silently removed and that I had to manually re-add them.

See thread: https://www.gog.com/forum/elder_scrolls_series/tes_arena_daggerfall_removed_from_my_library

EDIT: I can confirm that beyond having to re-add the games, the game files have stayed the same.

For me those two titles are still probably just title listings without manifest data, as I downloaded both games and got updates probably prior to using gogcli.exe, and as you probably recall my main manifest only contains games I add to it when I download them (maybe after an update) or have just purchased. I clearly keep the Games List and Manifest as separate entities (files). So the games being removed by GOG and me adding them back hasn't really changed anything.

It didn't change anything once I added them back (I can confirm that the game files have remained exactly the same). The above actions where generated prior to my adding the games back to my account.

However, I'm a little unsettled by the fact that I had to add them back at all (they never should have left my account).

Because I use a client to manage my backup, this is something I can easily investigate, but for anyone managing this by hand, good luck figuring out they removed games (if you don't keep a count) and if you do, finding out which games they removed.

Yeah I gathered that, and lucky for me I discovered they were missing before I updated my title list or they would have gone missing and then eventually come back as new games after me re-adding them.

Indeed, and it is disconcerting, and makes you wonder whether GOG knew this would happen, but weren't prepared to notify us or fix it, which is quite troubling.

Yes, your program is a great boon for those of us who use it, and my GUI lets me know what new games are added, and I do have an option to discover orphan games, but only by comparing against INI file entries. Other than that, I do take note of the game count that is displayed on my Games list before updating, and because I know how many new ones are being added I can tell when something is amiss when the final total isn't what it should be. That would of course be much harder if I did not update the list immediately after every purchase.

If GOG are going to do things like this, even the odd demo being removed from our library, I should probably add a feature that always compares old list to new list.

Then we have games that GOG occasionally rename. But at least as far as my full manifest goes, it is organised by Game ID, rather than title.

gogcli also allows you to do a diff between two manifests to generate an actions file. That can be used for audit purposes even if you don't depend on the storage facility of the tool to manage your game files.

You can even go further and just run all the actions listed in the actions file on your game files (this is what the gogcli storage execute-actions command does in the storage) to reconcile the state of your game files with the state of your manifest.

For gogcli storage specifically:

I added an automatic check (which can optionally be disabled) when applying a manifest to the storage that if the action file include the deletion of a game entry, abort.

No worries, thanks.
I've never really investigated the Storage commands, other than a quick look at them ages ago.

My full manifest just gets added to on-the-fly when I buy a new game ... added manually with a button click or when a game is about to be downloaded via that button. A good number of my games are not in the manifest yet, though their title and ID etc are in the Games list files.

I suspect you and I use your program very differently. :)

Does anybody know of a game where the only way to get the latest version is through one of the patch files or can you safely skip over patches entirely (if you never want to update your game installations in-place) in your backups?

I can't think of a single game where that is the case and I'm seriously contemplating adding something to skip over url patterns in gogcli (the patches seem to have a pretty consistent url pattern so I'm reasonably confident that they can be singled out).

I can't say I do, though someone mentioned a Hitman game recently in another thread, and I feel certain some games just get a patch sometimes, and maybe later they update the main files, which was also mentioned recently in that other thread.

I imagine there may be cases, like that Hitman game, where a patch is purely optional for some reason. As many of us know, sometimes bugfixes add other bugs, and it can be a matter of choosing the lesser of two evils. Or maybe it is purely a language patch or targeted for some specific hardware combo issue.

Perhaps if you did your question here in its own thread, it will be seen by more members and things might come to light.

P.S. I know it is likely silly of me to do so, but when I download a game, I always download whatever patches are available too. Of course the intent is to eventually cull what isn't needed, but that means getting around to checking.

Yeah, it is just that GOG api can be a mess. Recently one of the Crypt of the Necrodancer patch had a 404 (and not one of the "intentionally" dangling files 404 that you can safely ignore, one of the "we should contact support" 404 that if your tool ignores it, at some point you could lose game files) and its been days.

Sometimes, a patch has bad metadata and it slows down manifest generation.

Sometimes, the patches are larger than the original installer (I know I've seen it for at least one game).

Just thinking it is a hassle I could do without.

Anyways, I'll still implement it and I'll just ignore the url of that particular patch for now. Maybe I'll also add games that have ridiculously large patches when I see them. Go on an adhoc basis really.

Yeah, it will be an optional addon to the manifest. You'll be able to add regular expressions for the url patterns you want to skip (which will allow you to be as granular as you wish on what to exclude, from all patches, all patches for a specific game or even a single patch file).

In my case, I just added the patch file from Crypt of the Necrodancer that was bad (it is subtle, but they have two kinds of bad downloads depending on where you get the error code, one that is intentional and one that isn't, that one isn't). I can't be bothered contacting support to fix a broken patch file download that I don't really care about, so now I added an exception for it in my manifest and problem solved.

Its in the main branch for now. It will be in the next release (probably when I finally have a functional upload of games metadatas... tuning it not to be too slow or stop because of errors when getting the game photos is quite a hassle).

Part of me wish I did that, but I don't want to invest in that kind of storage for my entire collection (that would probably at least double my storage requirement, maybe even triple or quadruple it). I might miss out a little, but I'll live with it.

I'll trust that the overwelming majority of devs are not miscreants trying to sneak drm on my purchases after the fact.

I do plan on adding a file lock feature to counter obvious non-upgrade removals (ex: in the extras), but that's it.

New release. Editing the initial post with the changes seems to be a pain in the neck so I'll post it here.

v0.21.0:
- Added support in the manifest for url patterns (regular expressions) of files that should not be downloaded. Mostly useful to skip patches.
- Added a file locking mechanism to prevent certain files from being deleted during manifest updates. The protected files will also remain in the manifest.

Just a heads up.

I've been using v0.21.0 of gogcli.exe and it has been working well, except today it kept failing with the update to The Witcher 3 Complete Edition, and I could not obtain the manifest for that game.

On investigation, using my Get Game Info program, it appears the error(s) are due to missing checksums for several of the language version files (German, Polish, Russian, etc). The English version files are fine, and some others. I am talking EXE and BIN files.

I'm running the main branch which is just a couple of fixes ahead of the the 0.21.0 (for the storage repair command, won't affect anything else, I'll probably do a release once I've added resume support for repairs, shouldn't take long, I've just been very busy with work) and I haven't had an issue for quite some time.

I'm setting up some servers atm, but I'll try to generate a mini-manifest with Witcher 3 this weekend and see what happens.