After trying nearly all combinations of options I could think of I figured out that 16-bit textures with forced single buffering and no w-buffer give me working graphics with no significant artifacts. However, I get about 10 FPS in some areas. How is that even possible on GTX 980 Ti?

.
.
Consider running the game with this HD mod ( https://tljhd.github.io/ ), which has an improved graphics engine and supports HD resolutions such as 2160p and, yes, good AA too. See this screenshot of the modded game: https://i.imgur.com/lm5LDcp.jpg . The original game has an outdated engine that was not designed with today's PCs in mind, so tweaking it won't help.

To make it a little bit more understandable, the HD mod needs ResidualVM. You need to download both separately.
ResidualVM is for better resolution, bugfixes and so on plus it adds the possibility to mod the graphics with the HD mod.