Well, I've compressed installers that have the same name + checksum + size and the list of duplicates has shrunk considerably:

[
{
"Id": 1207664883,
"Installers": [],
"Extras": [
"gwbw_wallpapers.zip"
]
},
{
"Id": 1207665823,
"Installers": [],
"Extras": [
"shadowgate_hint_comic.zip",
"shadowgate_retro_wallpapers.zip",
"shadowgate_ringtones.zip"
]
},
{
"Id": 1207666703,
"Installers": [
"setup_devils_dare_1.0_(21250).exe"
],
"Extras": []
},
{
"Id": 1266051739,
"Installers": [],
"Extras": [
"tyranny_ringtones.zip"
]
},
{
"Id": 1207662743,
"Installers": [],
"Extras": [
"penumbra_collection_artworks.zip",
"penumbra_collection_manuals.zip",
"penumbra_collection_walkthroughs.zip"
]
},
{
"Id": 1420648167,
"Installers": [],
"Extras": [
"lox_avatars.zip"
]
},
{
"Id": 2000734359,
"Installers": [],
"Extras": [
"we_are_alright_lichtspeer_documentary_pl_plus_srt_subs.zip"
]
},
{
"Id": 1346232158,
"Installers": [
"setup_dungeons_3_-_campaign_-_a_multitude_of_maps_v1.7_build11_512e183_(36205).exe"
],
"Extras": []
}
]

I'm strongly tempted to do the same thing for extras that have the same name + size

Just trying out a new feature of my GUI, where it works its way through the full 'Games' list getting entries for the manifest.

It got as far as 'Asterix & Obelix XXL 2' and failed. I tried it on its own and it also failed, but processing the next entry on the list was fine.

So I am guessing there is something wrong with the title, probably the ampersand.

I doubt you need it, but the slug is 'asterix_obelix_xxl_2'.

EDIT
I tried another entry with an ampersand and that failed too - Beyond Good & Evil™
And another, that also failed - Blade & Bones

I don't have the Asterix game, but for the Beyond Good & Evil, I have it.

The title entry is:

"Title": "Beyond Good \u0026 Evil™",

If you Google "\u0026", you'll find its the unicode for "&". Its a valid (though not required) encoding.

I don't know if they use special French characters or any of that stuff in any of the game titles, but its probably a good idea to support unicode there.

You might want to look at what kind of support Autolt provides for that.

Otherwise, all the characters have a similarly encoded pattern (\uXXXX).

Edit: Anyways, I'll ponder on it. I don't really mind having it in the manifest honestly. I think that Golang will automatically process it ok: https://play.golang.org/p/Jq5rmGVFFJ-

Other sources, maybe not. It really depends on the implementation.

But anyways, if it unblocks you for now, look at what kind of support you have for unicode with Autolt and what kind of encoding it expects. This is what it is.

Edit2: Otherwise, I could also encode in utf8 as a more "portable" solution here, though special characters would have to be explicitly decoded, so no free lunch. Anyways, its late and I'm tired. To be resolved later.

I did try some other alternatives (&& & etc), but none worked. And "\u0026" is also not working for me.

It may be something to do with how it is being interpreted by CMD.

LOL - The forum software changed my second example which was '& amp ;' all one word no spaces.

I also tried the others from here - https://www.rapidtables.com/code/text/unicode-characters.html

Okay, I have appeared to have solved it my end. MY APOLOGY for assuming wrong ... an aspect of CMD I did not know.

It required a change in parameters to CMD.

I had to remove the && I was using to run two commands in the same command-line string.

RunWait(@ComSpec & ' /c ECHO ' & $title & ' && gogcli.exe ' & $params, @ScriptDir, $flag)

is now

RunWait(@ComSpec & ' /c gogcli.exe ' & $params, @ScriptDir, $flag)

So alas removal of feedback in the console of the title being processed.

I did not need to replace the & at all in a title. So 'Beyond Good & Evil™' and 'Asterix & Obelix XXL 2' now work fine as is.

P.S. For now I am just running that second instance when an ampersand exists in the title, otherwise I am still echoing the title to the console. If I come across another character that gives issue I may just remove the echo command altogether. It is really just a nicety to the user anyway. And personally unless troubleshooting I always have the console minimized, so never see it.

I am quite surprised that CMD did not ignore an ampersand inside quotes.

Just discovered another issue I need to look into.

When I use -i="Beholder"

I get 4 games and IDs returned in the manifest.json file.

Eye of the Beholder

Eye of the Beholder II: The Legend of Darkmoon

Eye of the Beholder III: Assault on Myth Drannor

Beholder

So the quotes are not making the return unique in this instance.

If this is not something you can fix your end, I'll have to devise some code to split the return and only use what I requested. Might be safer to do a test for it anyway, and then act upon it.

That said, the overall EstimatedSize at the end of the returned entry (and no doubt the VerifiedSize as well) appears to be for all 4 games combined. Not that I am personally using that section for anything.

EDIT
I have implemented a check and a fix, which solves the main issue, but it either strips the last few incorrect size lines or includes them if the desired game entry was last. So not a perfect fix, but certainly suitable for my current GUI usage.

At this point, I'm working around the non-unique filenames still.

I implemented compressing files that are listed several times for various languages in a single entry, implemented a suffix differentiator for genuine duplicate filenames across files that are really different, implemented a manifest migration command to migrate my previous manifest (now that the language entry for installer is an array, I require a migration) and am now implementing a "storage repair" command reconcile the files I have so far with my manifest changes (because I have like 300+ GBs of game files already and I'm not redownloading all that).

After all that is done and I've resumed downloading my files, I'll investigate this, make sure the format is correct (ie, will actually output "&" at display time and not "/u0026") and equally importantly, make sure characters that can only exist in the unicode realm (ex: French accents) are handled in a portable way.

This is behaving as intended. The --title flag is not intended to act as a unique selector. Its a search parameter that will return all games whose title contain the specified term. If you want to uniquely identify game titles that are a subset of other titles (ex: the first Eye of the Beholder entry in your case), I could add a regex search parameter in there (and then, you could type something like "^Eye of the Beholder$"), but I'm not sure that would make it more user-friendly for a non-technical audience.

Maybe if GOG has a million games and you owned a significant fraction of them, that would become a problem, but so far, I haven't found the few games that are returned when I type something like "Master of Orion" to be a great cognitive overhead.

Otherwise, I could feasibly add something like "return only the closest match" which seems to be what you are expecting here.

I'll look at it tonight.

I guess I am only expecting it, because there appears to be no other way of getting a manifest entry for a game. And of course, doing an updated replacement runs into even more issues.

I've got something like 360 game entries in my manifest now, so obtaining them has mostly been fine. Hopefully the solutions I have now implemented will mean no more issues ... fingers crossed.

Thanks.

gogcli manifest update --id=...

Or if you just want to use an update.json file (see gogcli update generate):

gogcli manifest update --update=updates.json

Or if you want to use a combination and use several additional game ids (why not?):

gogcli manifest update --update=updates.json --id=1 --id=2

The above will update a manifest and keep it consistent (apply the manifest filter on the updated manifest to make sure only the desired files are included, recompute size aggregates for the updated games and the manifest to take into account file changes... and once my latest changes are out, handle duplicate file names) and is the recommended way to update a manifest (otherwise, you gotta do all that work yourself).

Anyways, I can still add a "closest match" selector, but in this case, I believe the functionality you are looking for may already have been implemented above.

Sorry, I just reread quickly and the behavior is as expected. The expected/verified size entry at the end of a manifest is for the sum of all the games in the manifest.

Otherwise, each game entry has the same fields, but applying for the game only (so if you want to just get the size of an individual game, you can use that).

GOGcli GUI v1.1 has been released, with important bugfixes etc.

See here - https://www.gog.com/forum/general/gogcli_gui_frontend_downloader_and_validator/post17

Interesting, I took a very different approach. I generate a manifest file which contains all the game files and then do a differential to generate a bunch of actions that will reconcile the storage state with the state of the manifest.

From there, the actions file shrinks as the actions get executed. By the time all the actions have run, the storage is exactly as the manifest indicates.

The only thing in the manifest that changes as files get added are missing checksums which get filled up as checksum values are computed.

The philosophy is heavily inspired by some of the tools I'm using for my work (namely, Kubernetes and Terraform).

Very good.

Clearly we have different specific purposes.

Mine, is just building the manifest as you go, at need on demand. With a main focus on downloading and validating, and not so much about backups or archives, which I do independently anyway, using TeraCopy ... often with one of my GUI helpers for TeraCopy (if copying or moving a lot of game files).

P.S. The next thing I need to focus on, now that I have a few necessary things in place, is a setup for dealing with Updates ... especially with my convoluted setup.

I'll probably also do something similar for Downloading ALL Games, to what I have done with Downloading ALL Manifest entries, only with an additional Floating Progress Bar window I suspect, to give some additional feedback & control.

I see your point. Technically, you could sort of achieve that with gogcli's storage by doing an apply of with a maximum of 1 and then do a bunch of resumes again, with a maximum of n and specify n game ids each time, but overall, you are correct.

The main focus is to get the entire content of your manifest stored (possibly across several iterations, but the ultimate goal is to get all of it) on premise.

After that, in terms of what its for, it can double as both backup and local access. My intent is to keep 2 local copies, exactly for that purpose (1 to access, 1 just as pure backup). In my case, a Minio s3 setup on my local LAN is kind of ideal for local access. The longer term plan to add a web dashboard on top of it is just additional gravy. The things that motivate us...

Indeed.
The big catalyst for me of course, was GOG getting rid of the old GOG Downloader.

And then being inspired to create GUIs by the likes of yourself and Eddie and Kalanyr.

I also like to have some control over my experience and cater for shortfalls, that I see as requirements.