It seems that you're using an outdated browser. Some things may not work as they should (or don't work at all).
We suggest you upgrade newer and better browser like: Chrome, Firefox, Internet Explorer or Opera

×
Thanks Sude for lgogdownloader :)

It's taken a while but I've finished downloading my library. I did one to four games at a time in the order that I purchased them, checking with --repair as I went. I have not seen a single error, except for my own boobery where I didn't use --create-xml automatic on the first few. Even Witcher 2, with both Windows and Linux installers + all extras appears to have downloaded without any errors.

I admit I have only tried one small installation so far, Pirates! Gold, but that seems to be working fine. I need another HDD now, grossly underestimated the size of the downloads with both debs and tarballs + all of the extras for each game.
So I just downloaded just the Linux installers and I ran --repair..

I have no idea if there were any issues. Can there be a summary at the end i.e. "10 checked, 0 errors found" or something like that?

EDIT: Same with --status. It's not immediately obvious if there is anything I need to do and running it again is expensive (it fetches game data every time and I can't cache it in a file because of the use of ^M..)
Post edited September 16, 2014 by xyem
avatar
eiii: Oh, so --status does not check the integrity of the files? Somehow

Status codes:
OK - File is OK
ND - File is not downloaded
MD5 - MD5 mismatch, different version

suggested to me it does check the MD5 sums.
Is there any combination of options which (locally) checks the integrity of the downloaded files but does not automatically repair them?
There isn't an option that does chunk based integrity check like --repair

--status uses Downloader::getLocalFileHash to get the file hash. Downloader::getLocalFileHash calls Util::getFileHash if there is not locally stored XML data for a file. If local XML file exists then it takes file hash from the local XML file and compares it against file hash from remote XML.
Because local XML data is used to speed up the process it makes the process unreliable for checking integrity.
For example the file could have corrupted on the disk but local XML data would still show the same file hash as when it was created from a good file.

You can use --status to check integrity of files if you don't have XML data saved in ~/.cache/lgogdownloader/xml/gamename/
It depends on remote XML data. If remote XML data is not available then the downloader assumes that everything is fine. Which means that this won't work for extras.

Perhaps I should change --status to not use local XML data so it can be better used for checking integrity.

avatar
eiii: Also probably since I use the "--xml-directory" and "--create-xml automatic" options the download takes hours even when there is only a single new game to download (and I have a fast internet connection). I use

lgogdownloader --xml-directory dir --create-xml automatic --download

to download the files. The output is somewhat confusing:

100% 89.83/89.83MB @ 0.00kB/s ETA: 0s

To me this rather looks like the downloader is not re-downloading the file. But why does it try to do so? And if it's not re-downloading the file what is it doing that it takes so long?
It tries to resume but then fails with CURLE_RANGE_ERROR.
Much of the download code and logic comes from the previous downloader I created when GOG didn't have API.
It was easier to just assume that every file is incomplete and let libcurl fail with CURLE_RANGE_ERROR than to try getting file size and then doing checks on that.
The API does help in this but for some reason GOG decided to report the file size as megabytes in the API response instead of just using bytes to tell the exact file size.
The exact file size can be taken from XML data but GOG doesn't provide XML data for extras and therefore extras would need some other method of getting it.

My guess is that the version check in Downloader::downloadFile causes it to take a long time. More about that below.
avatar
eiii: The downloader causes a lot of disk I/O. Does it verify the checksums of already downloaded files when I use the --xml* options?
Downloader::downloadFile uses Downloader::getLocalFileHash to get file hash for version check.
If local XML data is available then it's a fast process as it only needs to parse the XML and get the file hash from it. However if local XML data is not available then it will call Util::getFileHash which can take a long time for big files and causes lots of disk I/O.
Local XML data makes the process much faster which is why the remote XML data is automatically saved for all installers and patches.
I'd like to remove the version check code from Downloader::downloadFile but GOG sometimes does silent updates without changing the file name. These silent updates caused lots of problems in the past which was the reason version check had to be implemented.

avatar
eiii: It looks like the --report option overwrites the report file on each run. It would be nice if I could provide a filename with this option.
I'll add an option to change the report file

avatar
eiii: I also would prefer when the downloader would log more information to the report file, something like what is written to the console. So far I pipe the command output to a log file, but the progress display for the downloads makes these log files unnecessary large and very hard to read. Is there any way to suppress the progress information in the command output?
Perhaps progress information should be printed to stderr instead of stdout. I'll see what I can do.
Thanks for your detailed answer, Sude.

avatar
Sude: There isn't an option that does chunk based integrity check like --repair
What I don't like with --repair is the "Redownload will delete the old file" part:

--repair
Repair downloaded files Use --repair --download to redownload
files when filesizes don't match (possibly different version).
Redownload will delete the old file

I don't like to lose old versions of files by an automatic process. Is there a way to make a backup of the old file instead? What does --repair do without the --download option?

avatar
Sude: Perhaps I should change --status to not use local XML data so it can be better used for checking integrity.
By understanding better how it works and that it only compares local with remote XML files --status looks good as it is now. I would not change it. GOG should provide XML files for every file which is download-able, but that's something different and beyond our control.

Instead of changing the behavior of --status I would prefer a separate option to check the integrity of local files. Something like --check or --check-integrity, which checks all local files against locally saved XML files (including extras when there are XML files available), giving an error when a checksum doesn't match and maybe printing out a warning when there's no local XML file available for a downloaded file.

avatar
eiii: Also probably since I use the "--xml-directory" and "--create-xml automatic" options the download takes hours even when there is only a single new game to download (and I have a fast internet connection).
The downloader causes a lot of disk I/O. Does it verify the checksums of already downloaded files when I use the --xml* options?
avatar
Sude: Downloader::downloadFile uses Downloader::getLocalFileHash to get file hash for version check.
If local XML data is available then it's a fast process as it only needs to parse the XML and get the file hash from it. However if local XML data is not available then it will call Util::getFileHash which can take a long time for big files and causes lots of disk I/O.
Local XML data makes the process much faster which is why the remote XML data is automatically saved for all installers and patches.
I'd like to remove the version check code from Downloader::downloadFile but GOG sometimes does silent updates without changing the file name. These silent updates caused lots of problems in the past which was the reason version check had to be implemented.
Looks like it's my fault then. To not lose local XML files, especially those for files which are changed or not available from GOG anymore, I start every download with a separate XML directory. Understanding how the download works this is probably not a good idea. :)
So at least I should start the download with a copy of the old XML directory or maybe put the XML files into a git repository. Time to write a wrapper script for the downloads I guess.
And yes, when GOG is unable to handle version changes properly please do not remove the version check. ;)

What do you do when the version check fails? Do you back up the changed file resulting in an *.old file? And do you also back up the local XML file?
I would prefer to have time stamps for the old files to not overwrite old *.old files accidentally (something like file-`date '+%Y%m%d%H%M%S'`.old). But when I write a wrapper script for the download I can take care of that in the script.

avatar
eiii: It looks like the --report option overwrites the report file on each run. It would be nice if I could provide a filename with this option.
avatar
Sude: I'll add an option to change the report file
Nice, thank you.

avatar
Sude: Perhaps progress information should be printed to stderr instead of stdout. I'll see what I can do.
Assuming that error messages are already written to stderr this would not be very helpful. You would lose error messages in the log when you suppress the progress information. If you separate the output this way it should be rather vice versa, progress information to stdout and the other messages to stderr.
I still would prefer to have more information in the report file so that there would be no need to capture the command line output of the downloader. An option to suppress the progress information would be another alternative, though with the downside that you also would lose the progress information on the terminal.

avatar
Sude: You can use --status to check integrity of files if you don't have XML data saved in ~/.cache/lgogdownloader/xml/gamename/
I guess that's what I already did when starting with an empty XML directory. :) So I still could use that as a hack to verify the file integrity. But if I understand you correctly this won't work for extra files, even when I have created local XML files with "--create-xml automatic", right?
Post edited September 17, 2014 by eiii
7a00b59 Move gameFile and gameDetails classes to their own files

810506b Create filepaths in gameDetails class
- Add makeFilepaths method to gameDetails class
- Add setFilepath and getFilepath methods to gameFile class

5635909 Add support for setting subdirectories
- Allows user to specify subdirectories for games, installers, extras, patches, language packs and dlc
Just FYI, the KS version of Wasteland 2 for Linux doesn't appear to be available from the API, neither lgogdownloader nor the official downloader are able to download it.

Just in case anybody comes in here wanting help with that.
The KS version of Wasteland 2 for Linux does work for me with lgogdownloader 2.17. I didn't check which files I had access to before it was available though so in your case I guess the list of accessible files need to be updated. I ran --status before starting my download and I guess that might help, although I am am not sure as I haven't used this application much.
Hello, I confirm kkvifte answer. I redeemed the game yesterday and downloaded it today with version 2.17 including extras. There was a 404 for the manual.
I'm trying to use lgogdownloader to download Wasteland 2, but it keeps downloading the windows version. How to set it to linux?
avatar
Elwro_RPGCODEX: I'm trying to use lgogdownloader to download Wasteland 2, but it keeps downloading the windows version. How to set it to linux?
Just add --platform 4 to the command. Check the help to see all available options:
$ lgogdownloader -h
avatar
Elwro_RPGCODEX: I'm trying to use lgogdownloader to download Wasteland 2, but it keeps downloading the windows version. How to set it to linux?
add --platform 4 to your command line should do it.
avatar
Elwro_RPGCODEX: I'm trying to use lgogdownloader to download Wasteland 2, but it keeps downloading the windows version. How to set it to linux?
in terminal you can call up the manual with:

man lgogdownloader

(I'd resize the terminal window to be bigger than default before hitting enter)


--platform arg (=5)
Select which installers are downloaded
1 = Windows
2 = Mac
4 = Linux
7 = All

adding the numbers gets multiple platforms, default is 1 + 4 = 5 and that's why you get the windows installer + the linux installer(s)

but see above, the download for Wasteland 2 on Linux is broken on GOG's end :(
avatar
Dihldrek: Hello, I confirm kkvifte answer. I redeemed the game yesterday and downloaded it today with version 2.17 including extras. There was a 404 for the manual.
People keep saying that, but the downloader only finds the extras, no sign of the actual install files.
avatar
Dihldrek: Hello, I confirm kkvifte answer. I redeemed the game yesterday and downloaded it today with version 2.17 including extras. There was a 404 for the manual.
avatar
hedwards: People keep saying that, but the downloader only finds the extras, no sign of the actual install files.
Here is the command that worked for me:
$ lgogdownloader --directory ~/Games/ --platform 4 --no-deb --no-language-packs --download --game wasteland_2_kickstarter
avatar
hedwards: People keep saying that, but the downloader only finds the extras, no sign of the actual install files.
avatar
Dihldrek: Here is the command that worked for me:
$ lgogdownloader --directory ~/Games/ --platform 4 --no-deb --no-language-packs --download --game wasteland_2_kickstarter
Didn't work.

lgogdownloader --directory /media/L_Drive/GOG/ --platform 4 --no-deb --no-language-packs --download --game wasteland_2_kickstarter
Getting game info 1 / 1
Dowloading: wasteland_2_manual.zip
/media/L_Drive/GOG/wasteland_2_kickstarter/extras/wasteland_2_manual.zip
100% ▕██████████████████████████▏ 3.77/3.77MB @ 0.00kB/s ETA: 0s
Dowloading: wasteland_2_reference_card.zip
/media/L_Drive/GOG/wasteland_2_kickstarter/extras/wasteland_2_reference_card.zip
100% ▕██████████████████████████▏ 0.76/0.76MB @ 0.00kB/s ETA: 0s
Dowloading: wasteland_2_map.zip
/media/L_Drive/GOG/wasteland_2_kickstarter/extras/wasteland_2_map.zip
100% ▕██████████████████████████▏ 90.14/90.14MB @ 0.00kB/s ETA: 0s
Dowloading: wasteland_2_soundtrack_mp3.zip
100% ▕██████████████████████████▏ 102.04/102.04MB @ 0.00kB/s ETA: 0s
Dowloading: wasteland_2_soundtrack_flac.zip
100% ▕██████████████████████████▏ 589.65/589.65MB @ 0.00kB/s ETA: 0s
Dowloading: wasteland_2_artbook.zip
100% ▕██████████████████████████▏ 3.55/3.55MB @ 0.00kB/s ETA: 0s
Dowloading: wasteland_2_wallpapers.zip
100% ▕██████████████████████████▏ 7.88/7.88MB @ 0.00kB/s ETA: 0s
I assume that there's something going on on GOG's end, but nobody working for GOG seems interested in addressing or even acknowledging that there's a problem here.


EDIT: I forgot to say thank you for the suggestion.
Post edited September 20, 2014 by hedwards