I have also had problems with setup but the method below seems to work for me. Might be worth a try-
Go into your GOG screamer directory and open (and backup) the dosboxScreamer file, find the line starth.exe and replace with setup.exe. Next find the line cycles=max and replace with cycles=20000. Now click the screamer shortcut that GOG makes for you and setup should start.
This might make setup a bit slow but I am sure you can increase the cycle amount and still keep it stable in order to change your settings.
Once you have done just revert the settings in the dosboxScreamer file (or replace it with the backup file). Hope this works for you.