This has been discussed on the forum before... generally the best advice is to draw the runes very small. This way it is easier for the game to recognize them. Don't cover the whole screen, just make the rune in about a 1-inch square, and see if that helps.

Also, people's experience with drawing the runes seems to vary a lot... some have no trouble and others have a lot. I think this may have to do with specific mouse issues that some people have. To minimize it you could make sure you have the latest patch from the developers (released recently... if you're not sure if you have it, you can just re-download the game from GOG as they incorporated the patch not long ago) and you could also try using Nuky's unofficial patch which can be found in these forums. I'm not sure if his patch specifically addresses spellcasting, but it has various performance fixes and those may help the game recognize your runes.

Here's an (old) thread on the subject:

http://www.gog.com/en/forum/arx_fatalis/cannot_cast_spells