Since the Steam version appears to be incompatible with it, I'd like to ask whether this version of Rune is compatible with Kentie's Direct3D 10 renderer?

I don't know but I can say that the GOG version includes the original version of Rune Gold as an extra so if the compatibility issue is with rune classic since it is a more recent port then it might still work with Rune Gold.

I'm not entirely convinced any of that would do much for the overall visual quality of such an old 3D game, unless someone also reworked the textures, and perhaps even the models. But that said, just out of curiousity, what is this SweetFX thing?

Well, textures are an integral part of the visual appearance, but they are not everything. On that note, SweetFX is a visual tweaking mod that is usable with lots of DirectX 9, 10 and 11 games. It gives the user plenty of customization options for a number of visual aspects. It is flexible and I tend to use it not only for all sorts of newer games, but also for old games, since you can even alter the image to resemble a CRT display and much more.

Here is the link to it:
http://www.guru3d.com/files_details/sweetfx_shader_suite_download.html