The game is launching, however it uses insane amounts of memory making it unplayable. (running on 8gb of ram, this happens in right after launch)

Uh
perhaps you can try more ram? Possibly run from an SSD. If you get lots of RAM you can then since it's only 5gb use a ramdisk that should make it's overhead almost seemless.
One would think a game that only itself requires 5gb of storage wouldn't need such an extensive solution though; I mean it did come out in 2010 not 2016 when DDR4 become really prevalent in need.
Perhaps there's something else going on; but I can't tell by your brief description.

As far as I know. Anno 1404 is a 32bit game, so it should not be able to use more than 4GB RAM, and that only if the LAA switch is enabled in the .exe (otherwise it can only use 2GB). Found someone with apparently the same problem, though, in the Steam forums (however it's a post in German, so it may not help you if I put a link here). He said he already tried the 4GB patch that enables the LAA switch, and it didn't help. However, the problem simply went away for him at some point, he suspects a windows update having to do with it.

I haven't played the game myself, yet, so I can't really be of any more help, I'm afraid. I just don't have any idea how a game would be able to use more memory than it possibly could as a 32bit application.

edit: Here is a guide that may help you: https://steamcommunity.com/sharedfiles/filedetails/?id=577231025
It is for the Steam version and includes things like running it without Steam or the Steam overlay, which of course is irrelevant for the GOG version, but maybe some of the other solutions work for you. As the Steam version is also DRM free since the latest patch, it is probably very similar, if not identical, to the GOG version.