If I take a screenshot with F12, it's saved as .tmp instead of .png. Changing the file extension does nothing, all I get is a broken and unreadable image.
Any idea what could be causing this?

Try this: https://www.gog.com/forum/general_beta_gog_galaxy_2.0/gog_galaxy_does_not_save_screenshots/post2 credits to Hustlefan

I guess the game from where you want to take screenshots was made with the Unreal Engine 4/5?
If so, I can confirm that there is a bug in the Galaxy overlay.

The solution for now is to use an older overlay where that bug doesn't exist. I'm using the overlay from version 2.0.59 which is working fine. You can download it here.

Press Win+R and copy/paste the following...

%PROGRAMDATA%\GOG.com\Galaxy\redists

Replace the "overlay" folder with the one from the zip archive. Restart Galaxy and try if it works now.