Editing the registry usually works. I've tested it with 1680x1050 (can't go any higher on my monitor).

Open regedit (press Windows-Key + R and enter regedit)
Go to HKEY_CURRENT_USER\Software\Sierra On-Line\Gabriel Knight 3\Engine
Edit the decimal value of these keys:
Screen Height = yyyy
Screen Width = xxxx

Be careful not to accidentally switch the values for Height and Width, that usually results in your error message.
Also make sure you tick the radio button to enter decimal values NOT hexadecimals.
See my settings in the attached screenshot. The game runs flawlessly in this resolution, although some texts and the menu are a bit too small for my taste.

Yes, I set decimal values. It's impossible to enter a meaningful resolution in hexadecimal anyway.
No, I did not switch height and width. Tried it again to make sure, and got the same error: "Not enough frame buffer memory is available on the card to run at 1920x1080, falling back to 640x480." The error text makes it clear that 1920x1080 is the resolution I entered.
Here's my registry screenshot.

Under Graphic Options, there are three 3D Driver options:
-Software Renderer
-Primary Display Driver
-ASUS R9 280 Series

The highest resolution I can run the game in is 1600 x 900 but only windowboxed on my 1080p screen, and only using the "Software Renderer" setting.

When I try to run the game in 1600 x 900 with the "ASUS R9 280 Series" or "Primary Display Driver" setting, I get the "Not enough frame buffer memory" error.

I can run the game in 1280 x 720 fine on all settings.

It is really weird that I cannot go up to 1080p!

Anyone who can make anything of all this info?

Could be a driver issue. Make sure the card is using the newest ones from AMD and not the Windows standard drivers.
Have you already tried lowering the AA settings in the cards control panel. For testing, you can also try setting everything back to default there.

Yes, I'm using the latest drivers.
I haven't tried lowering the AA (I'm assuming anti-aliasing?) settings. It actually never occurred to me to do that, and I have no idea how it might solve the problem, but I'll try that later today. Thanks.

Nope, it simply does not work.

I could change the resolution without using the registry.
I simply edited the shortcut "Launch Gabriel Knight 3" by adding
-width 1920 -height 1200
Before: "path\Gabriel Knight 3\GK3.exe"
After: "path\Gabriel Knight 3\GK3.exe" -width 1920 -height 1200

However:
At this res, the game is unbelievable slow and de facto unplayable.
And to make things worse:
My specs: i7-3,4GHz / 16GB / GF970-347.25 / Win 8.1-64
Native res is 2560x1440, and even modern games run fine.

Any value greater than 1920x1200 forces GK3 to fall back to 640x480 - due to framebuffer issues.
Even in 640x480 the movement is barely fluid.

I remember playing with an older GF under Win7 with the highest (i.e. native) res and with no problems at all.

Edit: Changing the settings of AA / AF does not help, either.

Hi.

I did get GK3 to run without any modification on win 10 64bit Home or what ever this basic edition is.
The solution I found for resolution is to use the usable resolution before max.
This meaning that I'm running my monitor/tv on 1920 but in the shortcut i entered -width 1280 -height 720 and this seems to work.

For the game running like crap, there doesn't seem to be a solution for that. I my self can't skip dialogues and after Gabriels thoughts on something, the game will get kinda stuck and slowly reagain the smooth camera movement

I was able to make it run nicely at 1080p, check it out:

https://www.gog.com/forum/gabriel_knight_series/gk3_1080p_solution_found_d3d11_only