Yeah, depending on the speed of your connection, downloading an installer with broken metadata that is several GBs to compute the checksum in the manifest might take a while.

It is a tradeoff between dependability and speed where I chose dependability (ie, making sure that your backup is solid).

Fingers crossed that the Galaxy api will be smoother and incur less maintenance on my end and also fewer delays for the end user. I'll post more once I've made progress in that direction.

That leads me to two questions:

1) Do you know of any installers that have broken metadata right now? Except for Encased that is since I don't have that game.

2) Is there any full documentation on the Galaxy API that is public? I only know of one public GOG API documentation but IIRC that documents mostly the web API and only has few Galaxy functions. If you don't have a documentation of the Galaxy API how do you intend to work with it? Do you intend to reverse engineer it yourself using Wireshark?

For the part of GOG Galaxy that pertains to accessing offline installers, I thought that nut was already cracked no?
https://gogapidocs.readthedocs.io/en/latest/galaxy.html

In particular:

[url=https://gogapidocs.readthedocs.io/en/latest/galaxy.html#get--products-(int-product_id)-(str-dl_url]https://gogapidocs.readthedocs.io/en/latest/galaxy.html#get--products-(int-product_id)-(str-dl_url[/url])

Unless those specs are grossly misleading, I suspect this is what we want right there.

I haven't dabbled with Galaxy yet, but given GOG's clear bias towards it, I'm going out on a limb here and hoping they are putting more love into maintaining it than their browser apis.

For everything else (ex: multiplayer match-making and other services), while reverse engineering their entire api and setting up a home emulator for it seems like an interesting challenge (especially if I was between jobs), I think it would be hitting them where they are hoping to make a lot of money (I don't believe Galaxy is just to be competitive with Steam, I think they want to ride on it for their own products) so I'd venture to guess that anyone going in that territory would definitely be exposing themselves to lawsuits (which they may win assuming they could spin it as them just wanting to enjoy their game fully on their LAN, but still a major hassle) and permabans (which they could do little against).

I look forward to what you might come up with. :)

The download itself isn't really a problem. I was hoping there is an alternative way to get the md5 checksums for files without going through the .xml file in the Galaxy API.

Hello Magnitus, hope you are well.

I don't know enough about your program, to know if this is possible. But can an option be added to use a different GOG server to download from?

Apparently Lgogdownloader has such an option ... so I've been told anyway.

Not that I know where to get a list of other GOG servers from, but no doubt someone knows, and could advise.

If you haven't noticed, I have been dealing with absolutely lousy download speeds again with GOG ... ten times slower than it should be. All since their server issues of a few weeks ago. And I am not the only one effected.

I've posted here about it.

https://www.gog.com/forum/general/what_is_happening_with_download_speeds_on_the_offline_backup_files_its_a_snails_pace_now/post11

It seems upon investigation, that I am currently downloading from somewhere in North America. Which is a long way from where I am. There must be better servers closer by.

Something I learned recently is that you can use Galaxy to download the offline installer files. So it's a pretty decent way of getting them if you have trouble with 3rd party tools.

I've known that about Galaxy since version 1, and even used it a couple of times, but Galaxy is not for me, doesn't work well on my system, too bloated etc.

I'd still be using the old GOG Downloader, if GOG hadn't disabled it. Though both gogcli.exe and gogrepo.py are superior in many ways ... far more control in regard to destination, records, etc.

For the moment I am forced to use Free Download Manager 5 until GOG restores my usual server. It's a bit more effort but worth it, and certainly beats just using the browser.

In any case, I would be quite surprised if Galaxy wasn't suffering from the same server issue I am having. It's a server issue not a downloader program one.

I was assuming the same. There isn't? Given how frequently this breaks for offline installers, I'd be surprised if there wasn't. Doesn't Galaxy verify checkums?

Short of using a cloud solution to download games as an intermediate to pretend you are somewhere else, I'm not immediately sure how to do that. That will have to be investigated.

Might take some time though. Hashicorp recently changed their license for Terraform (a tool I'm increadibly dependent on for my work) to a business license and depending on how that unfolds (whether Hashicorp relents and to what degree and how fast the rest of the opentf community picks up the slack), it might take some of my bandwidth for a while. We'll see.

So in other words, it seems GOG's software makes the decision for you, and you cannot for instance use something different at the start of the download link.

I wonder if a good VPN might help, though at the very least that would require some experimentation.

No worries, I know you are a busy man. Thanks for taking the time to respond.

I guess if gogcli.exe used more than one thread per download, then that might also help, but that would be a bit of work etc for you.

A shame there is no multi-thread downloader, that I am aware of that could grab all browser links with one click, activating each in turn and then capture the true link for each, plus give you somewhere to add in expected byte sizes and checksums, which it would test with after.

If this download speed issue persists for too much longer, I may have to look into compiling Lgogdownloader for Windows, as my good friend TheDcoder suggested I should do.

EDIT
Talking of him, He just messaged me the following. No idea how true it is, but he seems confident it is the case.

I doubt GOG has anything much to do with it, CDNs are responsible for routing, not the content provider
GOG just gives them the files to distribute, the rest is up to the CDNs.

To which I replied.

If that is true, then I have been laying some of the blame at the wrong door. Seems weird though, that a service like that would have someone at the bottom of AUS, downloading from North America.

There doesn't seem to be. Or at least, if there is then nobody has found and documented it yet.
Yes, it does.

A few days ago I also had a slow day on our CDN server again (around 150 KB/s max). I also thought I should try a VPN at least once to check if I could go around the faulty server and use another one. When using the VPN to go through the Netherlands the downloads were at a steady 30 MB/s. Or in other words 200 times faster.

That's good to know. Thanks for that info.

I'm not sure, but it is possible that Free Download Manager 5 with its multi streams is using more than one server, so gets around the bad server that way too.

My mate TheDcoder, has now changed his mind a bit too, possibly laying the blame back on GOG more than the CDN, saying the following.

Maybe GOG did stuff up and configured something incorrectly which prevented the CDN from caching the game files.

Ok, for Terraform, a consortium of companies really picked up the slack with the fork and will make it happen so I can mostly occupy a minor role there (I really wasn't sure what the status would be for that, but it seems they got the situation well in hand).

The thing about multi-threaded downloads is that assuming the GOG doesn't arbitrarily throttle you, you won't gain any additional throughput if your home bandwidth is already saturated from the one download so the gain there is a dud for the overwelming majority of users. My understanding from previous occurences is that it is a transient state of affairs. Let me know how your situation progresses.

For the second paragraph, its an interesting idea to better support third-party integration, I'll have to ponder how to elegantly support that. I like very much the idea of gogcli taking the first url as input and spitting out the true download url + expected checksum (assuming the metadata endpoint is retrievable, it often isn't) and download size and then some external tool can decide what it wants to do with that.

Last time the issue lasted some months for me, and even downloading with Free Download Manager 5 wasn't great at times, though certainly better than anything else I tried.

And yep, the only benefit of multi thread is when one thread alone is slower than it should be. Currently I can get good throughput with FDM5, that equals what I should be getting, instead of being 10 times slower as it currently is for me with just the browser or gogcli.exe. Painful to tolerate all the extra work, but at least it is working. Be worse if I did not have some added options with my GUI to do zip, file size and MD5 checking in a semi-automatic way.

New release.

v0.22.0:
- Added a couple debug-level logs
- Fixed problems in storage repair command and streamlined it with how other manifest-related commands work
- Added ability to find games that have file containing certain url patterns in manifest search functionality (mostly useful to find games containing patches or language-specific installers though there might be other use-cases I haven't thought about)
- Added a convenience command (manifest trim-patches) to automatically trim the patch files for a given game in the manifest (given that certain games have 32-bits installers listed as patches and others have their main installer not updated with the latest patch, I judged it prudent to support the functionality on a per-game basis rather than blanket-style on everything so that the user can look at it and decide what they want to do for separate games)