huan: A lot has been said about this already, I just wanted to add something to this specific piece... which highly depend on filesystem in question. Some filesystems are much closer to databases than others. If you are storing your files on FAT then yes, you should worry about silent corruption. Btrfs on RAID1 with regular scrub on the other hand IS immune to this due to checksums, it will loudly complain if it detects checksum failure. It also has it's own RAID implementation without need for another layer to sit in between FS and hardware, and that one should be able to recover from almost anything short of direct meteor impact (though from mailing list i get the feeling that RAID-n for n>1 is still immature and not recommended for general use). Including meteor impact if you regularly "send" differences to offsite copy (diff/rsync analog that works with filesystem metadata and knows what changed since last sync without need to read whole file to hash it's content).
I've been meaning to move my private data storage to something like this, when I find enough time to do it. And I get to access my files over SMB or NFS or SSHFS instead of having to first download them to local disk before I can use them (highly disk-space inefficient, considering some installers on gog are in the range 20-30G spread over many files).
Ideally, I want something that is somewhat robust against corruption operating on cheaper non-specialized commodity hardware. I'll pay premium for a good laptop (I spend so much time on it that it is just a good investment), but otherwise, I can be rather cheap with my infrastructure.
I`m very much sold on the horizontal scalability premise of just throwing more "lower grade" hardware at a problem as needed rather than pay $$$ for better and better vertical scaling.
My working home setup is a cheap set of "servers" (one of them acquired after someone discarded their old computer) and two 8 TB external hard drives bought during a sale (still from a reputable brand mind you)... that I'm currently leveraging to experiment with Kubernetes and Docker Swarm and DB replication/sharding (figured might as well use them for backup too)
RAID 1 requires at the very least a machine with 2 sizable hard drives. If you want to increase capacity, then you need bigger hard drives or a strategy to split your files across machines.
With GridFS, you can have a replica set across 2 separate drives for redundancy and you can shard across additional sets of drive if you need more space.
Yepoleb: Steam definitely has a better public API, but the stuff SteamDB shows isn't publicly accessible either. I recently found example code on Github called
galaxy-demo-app that suggest that registered developers can generate their own OAuth2 IDs, secrets and possible register allowed redirect URLs, so you can use the web login for own applications. That's a quite nice way to do it, as clients never get access to user credentials. Google's APIs work the same way. They'd just need to make it public instead of restricting it to a few selected studios.
I suppose it makes sense that they would give special access to game devs.
However, for a web site whose premise is to "own" your game, I find that their tooling to efficiently keep an offline backup of your installers leaves much to be desired.
At the very least, they could keep a well documented targeted public API that uses API keys to facilitate the creation of third party tooling and libraries to achieve this.
