It seems that you're using an outdated browser. Some things may not work as they should (or don't work at all).
We suggest you upgrade newer and better browser like: Chrome, Firefox, Internet Explorer or Opera

×
Community
General discussiongogcli - Gog Client(295 posts)(295 posts)
(295 posts)
Pages:
1
2
3
4
5
10
15
20
This is my favourite topic
Posted February 23, 2021
avatar
mrkgnao: Could you explain what features you hope to have in gogcli that are currently missing from gogrepo(c)?
Magnitus has already answered from his perspective, but here's mine.

Even though I have mastered the installing and usage of Python, many struggle with it and many of those fail ... going by the feedback I have received.

So anything that is just a single executable that can do essentially what gogrepo.py does, without the extra of needing to install Python and library dependencies is a step forward, certainly for those users.

I also like that we can interact on a game and file by file basis.

Certainly from my perspective, creating a GUI will be much simpler for gogcli, even though I have done such for gogrepo.py already, if not at this point for the advanced elements of the Kalanyr fork DEV version.

Even just being able to get a list of games, and then populate a manifest file on demand is a step forward to my mind, rather than have to spend hours in many cases building your first manifest from scratch, when you already have a lot of games.

I know you can get just a list of games with gogrepo.py, and I have done that with Kalanyr's fork, and with a modified version of the original gogrepo.py. That's what my GOGRepo Simple GUI program does, and it just builds up manifest entries on-the-fly when needed ... essentially just to get the MD5 and file size values. However that program is still dependent on a Python install. I use that after I have used another downloader, Free Download Manager 5, just to validate (verify) downloaded files.
Posted February 23, 2021
avatar
Magnitus: If you find any issues, let me know and I'll look at it tonight.
No issues to report at this point, and I have tested a few things. Nice to see we have two methods of getting the MD5, one a specific info query and the other in a manifest result.

I note that an exact file size return is only happening at this point for a game EXE file.
I presume that you will either add it later for the extras ... or maybe just rely on a zip test?

Are you intending to do some other switches for owned-games?

A total number of games and pages would be nice. I do realize you show the number of total pages at the top of the first page returned, which can of course be parsed to determine how many pages to additionally check for a full game list.

Currently we have to specify each page number to check, so I was also wondering if you were going to add a return ALL game titles switch?

No biggie if you don't do either of those last two things, as I can certainly deal with them as is, but they would be nice. :)
Post edited February 23, 2021 by Timboli
Posted February 23, 2021
avatar
Timboli: No issues to report at this point, and I have tested a few things. Nice to see we have two methods of getting the MD5, one a specific info query and the other in a manifest result.

I note that an exact file size return is only happening at this point for a game EXE file.
I presume that you will either add it later for the extras ... or maybe just rely on a zip test?
I know of two ways to get file sizes, when you download it (its in the content-length header) and when you get the checksum (which only seems to be available for installers).

Therefore, short of initiating a download and immediately cancelling it (which I find is a bit of an arsehole move to do, even though a well programmed backend should be able to handle that gracefully otherwise its a possible DOS attack vector... I haven't tried making a HEAD request on the download url though, that may work), I don't see a clear way to get non-installer file sizes during the manifest generation phase atm (except maybe the HEAD request thing on the download url, I'll give that a try).

Otherwise, the size gets updated during download (by the time the files indicated in the manifest are transferred to storage, the verified size and checksum entry will be filled for all the files in the manifest).

Given that we do get the expected file size from gog when initiating the download, gogcli uses this to validate the size of the downloaded file is correct.

Otherwise yes, there will be a zip test. I looked into that yesterday and to thoroughly verify the integrity of a zip file, you pretty much have to read all the files in the zip archive to compared the crc32 checksums (something the golang archive/zip package does in the background, I double-checked by looking at their implementation).

For transfer from one storage to another, I won't do that as we already have the md5 checksum at that point in the source storage and the zip file was already validated in the source storage. For the initial transfer from the gog api to the storage (when the checksum for zip files is missing), I believe I might be able to run the crc32 zip checksum verification in the background in separate goroutines without any real delay overhead given that downloading the files from gog is the real bottleneck here.

avatar
Timboli: Are you intending to do some other switches for owned-games?

A total number of games and pages would be nice. I do realize you show the number of total pages at the top of the first page returned, which can of course be parsed to determine how many pages to additionally check for a full game list.

Currently we have to specify each page number to check, so I was also wondering if you were going to add a return ALL game titles switch?

No biggie if you don't do either of those last two things, as I can certainly deal with them as is, but they would be nice. :)
Sure, I can add something to return all the owned games.

Btw, I'm currently outputting the api commands in human readable yaml(ish) format.

Would it be helpful to you if I included flags for json and/or xml output?
Post edited February 23, 2021 by Magnitus
Posted February 23, 2021
avatar
Magnitus: I know of two ways to get file sizes, when you download it (its in the content-length header) and when you get the checksum (which only seems to be available for installers).

Therefore, short of initiating a download and immediately cancelling it (which I find is a bit of an arsehole move to do, even though a well programmed backend should be able to handle that gracefully otherwise its a possible DOS attack vector... I haven't tried making a HEAD request on the download url though, that may work), I don't see a clear way to get non-installer file sizes during the manifest generation phase atm (except maybe the HEAD request thing on the download url, I'll give that a try).
Yes, of course HEAD requests work just fine. That's how I'm getting the file data on extras as well.
Posted February 23, 2021
avatar
Geralt_of_Rivia: Yes, of course HEAD requests work just fine. That's how I'm getting the file data on extras as well.
Thanks for confirming. That means gogcli will be able to get an exact size for the entire collection upon manifest generation. That's awesome.
Posted February 24, 2021
avatar
Magnitus: Sure, I can add something to return all the owned games.

Btw, I'm currently outputting the api commands in human readable yaml(ish) format.

Would it be helpful to you if I included flags for json and/or xml output?
Goodo.

Yes please, probably nicer than me piping to text file or reading STDOUT.

Which reminds me of download progress. Are you going to show that? i.e a percentage etc.
Posted February 24, 2021
Faking a gog proxy might be easier than dealing with plugins. More browsers support proxies. I made a mock one following a guide in perl, but something you could easily integrate into your program is telling people to go to their proxy settings and set it to localhost, 9924 or something silly like that. If you needed, i could probably write something in C for you. Unfortunately some browsers will try to force HTTPS on you. I'd hate to suggest using the hosts file or something, too. However, it sure as hell beats trying to deal with captcha and such.
Posted February 24, 2021
avatar
Timboli: Yes please, probably nicer than me piping to text file or reading STDOUT.
I was planning to generate json/xml to output and let you pipe it, but I can add an optional file argument :P.

avatar
Timboli: Which reminds me of download progress. Are you going to show that? i.e a percentage etc.
I have short/medium term plans to improve the logs -> Put all write operations (file uploads/deletions, game creation/deletion) as info logs and put the read operations as debug logs.

So if you operate in normal info mode, it will look something like this:

Created <game title> [id=<game id>]
Deleted installer <file name> from <title> [id=<game id>]
Uploading installer <file name> from <title> [id=<game id>]
Uploaded installer <file name> from <title> [id=<game id>]
Deleted <game title> [id=<game id>]

I'm not ruling out download progress output for the longer term, though it wouldn't be part of the mvp.
Post edited February 24, 2021 by Magnitus
Posted February 24, 2021
avatar
kohlrak: Faking a gog proxy might be easier than dealing with plugins. More browsers support proxies. I made a mock one following a guide in perl, but something you could easily integrate into your program is telling people to go to their proxy settings and set it to localhost, 9924 or something silly like that. If you needed, i could probably write something in C for you. Unfortunately some browsers will try to force HTTPS on you. I'd hate to suggest using the hosts file or something, too. However, it sure as hell beats trying to deal with captcha and such.
That's an interesting workaround. There could indeed be a listening proxy between that the user use to login against gog.

However, it would possibly be a little finicky though as modern web pages make ajax requests and also download other resources after the initial page load and it would only work automatically if only relative urls were used in the client-side, but often, some full urls are hardcoded. So, it would greatly depend on how their login page is setup.

Otherwise, you can go full on: Change your hosts/dns, generate a self-signed certificate authority, put your self-signed certificate authority certificate as trusted in your browser and then issue yourself a nice gog.com certificate, but that would require way more technical expertize than I would expect most users to have (I could probably automate something for Linux, but it would not be cross-platform and if its not properly undone afterwards, it could seriously mess up the user's experience with the gog website :P). Usually, its not something you want to do on your computer unless you really know what you are doing.
Post edited February 24, 2021 by Magnitus
Posted February 24, 2021
avatar
kohlrak: Faking a gog proxy might be easier than dealing with plugins. More browsers support proxies. I made a mock one following a guide in perl, but something you could easily integrate into your program is telling people to go to their proxy settings and set it to localhost, 9924 or something silly like that. If you needed, i could probably write something in C for you. Unfortunately some browsers will try to force HTTPS on you. I'd hate to suggest using the hosts file or something, too. However, it sure as hell beats trying to deal with captcha and such.
avatar
Magnitus: That's an interesting workaround. There could indeed be a listening proxy between that the user use to login against gog.

However, it would possibly be a little finicky though as modern web pages make ajax requests and also download other resources after the initial page load and it would only work automatically if only relative urls were used in the client-side, but often, some full urls are hardcoded. So, it would greatly depend on how their login page is setup.

Otherwise, you can go full on: Change your hosts/dns, generate a self-signed certificate authority, put your self-signed certificate authority certificate as trusted in your browser and then issue yourself a nice gog.com certificate, but that would require way more technical expertize than I would expect most users to have (I could probably automate something for Linux, but it would not be cross-platform and if its not properly undone afterwards, it could seriously mess up the user's experience with the gog website :P). Usually, its not something you want to do on your computer unless you really know what you are doing.
I've actually done it, though. The cookies are global to GOG's domain, not a specific file or page or something. All you have to do is interecept. The hard part is, well, you have to not allow it to connect to actual gog 'cause it forces SSL. So i had 2 solutions: host file to go to my fake gog server, and a fake proxy that never actually takes me to GOG. If you want, we can discuss this in PMs and i can share code.

The issue with the browser is that many these days make it impossible or near impossible to accept the faulty signed certificate. My recommended approach is to take the proxy route, as then they can just disable the proxy afterwards to start using the site again.
Post edited February 24, 2021 by kohlrak
Posted February 24, 2021
avatar
kohlrak: I've actually done it, though. The cookies are global to GOG's domain, not a specific file or page or something. All you have to do is interecept. The hard part is, well, you have to not allow it to connect to actual gog 'cause it forces SSL. So i had 2 solutions: host file to go to my fake gog server, and a fake proxy that never actually takes me to GOG. If you want, we can discuss this in PMs and i can share code.
That would be great. I would be overloaded if I tackled this right now, but if you like we'll chat once I'm ready to tackle the login.

avatar
kohlrak: The issue with the browser is that many these days make it impossible or near impossible to accept the faulty signed certificate. My recommended approach is to take the proxy route, as then they can just disable the proxy afterwards to start using the site again.
A self-signed certificate is not "faulty". Technically, all the root ca certificate authorities are self-signed (trust has to start somewhere :P).

Making your own self-signed ca certificate is just not "official", as in, any software you get from the internet won't know that they can trust it. You'll have to explicitly tell it that it can (this is how many root certificate authorities in closed off local systems work btw... like a newly minted kubernetes cluster for example).

You can pass your own self-signed ca certificate that you created yourself to your browser, no problem and after that, it will take any certificate signed with that ca as hard cash.

I've done it a couple of times to simulate production on my local machine.

Terraform makes it really easy btw (the generating the certificate chain part).

Here is an example if you are curious: https://github.com/Ferlab-Ste-Justine/kubernetes-certificates
Post edited February 24, 2021 by Magnitus
Posted February 25, 2021
avatar
Magnitus: I have short/medium term plans to improve the logs -> Put all write operations (file uploads/deletions, game creation/deletion) as info logs and put the read operations as debug logs.

So if you operate in normal info mode, it will look something like this:

Created <game title> [id=<game id>]
Deleted installer <file name> from <title> [id=<game id>]
Uploading installer <file name> from <title> [id=<game id>]
Uploaded installer <file name> from <title> [id=<game id>]
Deleted <game title> [id=<game id>]
Goodo.

avatar
Magnitus: I'm not ruling out download progress output for the longer term, though it wouldn't be part of the mvp.
Okay. I think I have some work-around code in an old program of mine, which can tell how much has been downloaded, so I guess the important bit will be knowing the exact file size before the downloading starts.

Most people, including myself like to have some kind of feedback during processes like downloading, especially with large files that can take a while.
Posted February 25, 2021
avatar
kohlrak: I've actually done it, though. The cookies are global to GOG's domain, not a specific file or page or something. All you have to do is interecept. The hard part is, well, you have to not allow it to connect to actual gog 'cause it forces SSL. So i had 2 solutions: host file to go to my fake gog server, and a fake proxy that never actually takes me to GOG. If you want, we can discuss this in PMs and i can share code.
avatar
Magnitus: That would be great. I would be overloaded if I tackled this right now, but if you like we'll chat once I'm ready to tackle the login.
I sent you a friend request, so we can use gog's system. It's actually not that hard if you already know sockets.
avatar
kohlrak: The issue with the browser is that many these days make it impossible or near impossible to accept the faulty signed certificate. My recommended approach is to take the proxy route, as then they can just disable the proxy afterwards to start using the site again.
A self-signed certificate is not "faulty". Technically, all the root ca certificate authorities are self-signed (trust has to start somewhere :P).

Making your own self-signed ca certificate is just not "official", as in, any software you get from the internet won't know that they can trust it. You'll have to explicitly tell it that it can (this is how many root certificate authorities in closed off local systems work btw... like a newly minted kubernetes cluster for example).

You can pass your own self-signed ca certificate that you created yourself to your browser, no problem and after that, it will take any certificate signed with that ca as hard cash.

I've done it a couple of times to simulate production on my local machine.

Terraform makes it really easy btw (the generating the certificate chain part).

Here is an example if you are curious: https://github.com/Ferlab-Ste-Justine/kubernetes-certificates
I've done it as well, but i've had issues where peoples' browsers just wouldn't allow the connections. For my own servers, I've started using Let's Encrypt, since it's an accepted CA. Obviously, that wouldn't work in this case.
Posted February 25, 2021
avatar
Timboli: Okay. I think I have some work-around code in an old program of mine, which can tell how much has been downloaded, so I guess the important bit will be knowing the exact file size before the downloading starts.

Most people, including myself like to have some kind of feedback during processes like downloading, especially with large files that can take a while.
Its really cool. Docker does it when downloading images and its awesome to watch.

I'm just envisioning the complexity in terms of entries that don't have fixed place in the output and also where the amount of info outputed is large.

Docker has something that looks like this:
layer 1 [=====> ] 20mg/50mb
layer 2 [===> ] 28mb/100mb
...
layer n [=============> ] 55mb/100mb

Here's the kicker: All the moving entries in a docker pull have a fixed vertical position on the screen and the amount of output that is managed is relatively constrained (usually less than 20 image layers that are getting pulled).

Now, imagine that you go a download happening with 4 goroutines (ie, threads), one of them is downloading a whooper of a 8gb file and the other 3 and downloading small <10mb files.

You got:

...
goroutine 1 [=====> ] 4mb/10mb
goroutine 2 [=> ] 80mb/8000mb
goroutine 3 [==========> ] 7mb/10mb
goroutine 4 [=======> ] 3mb/5mb

A couple of seconds later, you might have:
...
ex goroutine 1 [==============] 10mb/10mb, done
goroutine 2 [==> ] 100mb/8000mb
ex goroutine 3 [==============] 7mb/10mb, done
ex goroutine 4 [==============] 5mb/5mb, done
created directory for game AZ
goroutine 1 [=======> ] 4mb/10mb
goroutine 3 [====> ] 8mb/20mb
goroutine 4 [=======> ] 5mb/10mb

You need to manage changing output whose vertical position changes on the screen while also keeping a bucketload of other output (possibly thousands of line if you are doing a big backup).

Furthermore, you just converted your output from something that is a generic pipe (things only get appended, I can redirect this to a file, no problem) to something that will only work on an editable medium like your screen (I have no idea how it is supposed to behave if I redirect the above to a file).

Could all that complexity be managed? Of course. What would I rather do with that time investment: the above or a much better looking web dashboard that you can look at from your browser on localhost:8080 ?
Post edited February 25, 2021 by Magnitus
Posted February 26, 2021
avatar
Magnitus: Could all that complexity be managed? Of course. What would I rather do with that time investment: the above or a much better looking web dashboard that you can look at from your browser on localhost:8080 ?
Ha ha ha. Don't mind me, do what you feel you have to. I guess I was just stating the obvious anyway.
Also another reason from the coder's perspective to only deal with one downloading thread. ;)

I found my old code from 2016, which I hope will be sufficient to this purpose.
Not had a chance to check it yet, and so with the aim of making that easier, I have tentatively started work on GOGcli GUI. Spent the morning getting the basics done on that, pinching code from various other programs of mine, especially GOGRepo GUI. I decided to use a ListView instead of ListBox this time around for Game Titles, with a hidden Game ID column, just to simplify an aspect or two ... if you can call it simplifying using a ListView rather than a simpler ListBox. :)
Pages:
1
2
3
4
5
10
15
20
This is my favourite topic
Community
General discussiongogcli - Gog Client(295 posts)(295 posts)
(295 posts)