The spellcasting system has a very small margin of error; if you make one of the sides of the rune too long or not straight enough the rune won't work. There is no real "trick" to it other than drawing them very precisely. With practise you will be able to draw the second rune (Yok) quite easily, but some of the more complex ones found later on can still be a challenge.
While drawing you can hold down Shift just before you finish the last rune to memorise the spell. It then appears as a little icon on the HUD. You can memorise up to three spells; memorised spells can be cast instantly with the number keys 1, 2, and 3. A lot of the time you can reasonably predict what you will be needing and pre-cast the appropriate spells. Most of the time I keep the memorisation slots completely filled, even if it's just with something as basic as Fireball.

Sounds like recognition problems. It happens.
Practise. The rune recognition can be a bit... temperamental. The first rune ("aam") probably works because it's just a straight line, and is pretty hard to screw up. For "yok" you should be trying to draw a perfect square, minus the top line. Take it slow and steady, in a continuous motion, and try to make your lines as straight as possible.
I've found the demonstration you get by clicking on the runes in your spellbook isn't always helpful. It can take some tinkering to figure out the best way to draw a rune, in order to get it recognized reliably.
The rune "taar" is particularly bad, and ends up getting recognized as 3 different runes depending on how you draw it (aam, taar, or movis).

I have pretty high end hardware: 2X280 SLI and I'm running on Vista.
It is extremely hard to get the spells to work. To the point it isn't fun and I have to try 15-30 times just to get the second rune to work.
Any other ideas?

Are you actually drawing the runes with straight lines? If the rune comes out all angular or on a slope it won't work. Because of the way this system works you aren't going to get very far unless you can reliably draw fairly straight lines. Having a steady hand and good aim will help immensely; you could also reduce the mouse sensitivity to reduce accidental deviation from the path needed, but this will reduce the mouse speed for everything, not just drawing runes. Refer to YouTube videos or the in-game guide (shown when clicking on a rune, although it's a bit hard to see) to see how to go about drawing a particular rune in the most efficient manner, and follow that as closely as you can. The actual size at which you draw the rune doesn't really matter, although if you draw it too large you may have more difficulty keeping the lines straight.
If you start drawing a rune but one of its sides is too angled or clearly the wrong length, release the mouse button (but not Ctrl) and start that rune over; if the lines aren't sufficiently straight it won't be recognised or will be recognised as a different rune, and that extra drawing time will all be wasted. As long as you keep Ctrl held you can save up the runes you got right and try repeatedly on the one you can't get; unless you end up drawing a different rune by accident (which results in a different spell combination) the badly-drawn rune should fail every time.
In the case of Yok you should be able to cast it quite easily by drawing a simple cube (albeit without a top); if the three sides are basically the same in length the rune should be accepted correctly. Yok is one of the easiest runes to master, but only a few are really difficult to draw correctly; with time you should be able to learn to draw Yok better, and this practise will carry over to runes in general.

Excellent advice from Arkose. If I might add, I have found that you don't need to draw a rune as big as you might think. You can actually draw them pretty small - this might help you with the 'straight lines' problem. I also found in the case of Yok, the last line up (the vertical one) can be *very* short (I just flick the mouse straight up quickly and it accepts it). Good luck. Try to stick with it because the game *will* reward you. It's great fun...

I love this game. I loved Ultima Underworld and really wanted to replay - but it is one of those cases where it is just a bit too far back to play.
I have tried every conceivable thing. I thought that because the widescreen and high resolution options were added in the latest patch (I think), I switched from 1680x1050 to 1024x768. Nope.
I changed the mouse sensitivity to NONE in the game. Nope.
I changed the mouse sensitivity to very slow in Windows. Nope.
I've tried drawing extremely slow, fast. I've lined up the cursor on a straight line in the game (Like a brick) and made it work.... very, very rarely.
I've googled every forum out there - many so dead that there are only spam posts for the past few years. Nothing.
I have found a bug. If you set a high resolution or widescreen mode, when you restart the game the overlay graphics are all HUGE and when you try to draw the runes you can barely even tell where the cursor is... doubt anyone fixes bugs anymore but it is 100% repeatable. Resetting resolution in the game corrects it.
I really, really want to play this game. There has been an RPG drought IMHO for so long. I love 1st person. I love the simple gameplay mechanics. At this point I wish I could turn on "arcade mode" for spells like the XBOX version has...

Really sorry you are having so much trouble getting runes to work. Did you try drawing them smaller as I suggested? I am using a 20" monitor (old CRT that still works great) and I'm running it at 1600x1200. Of course, this is a non-wide screen capable monitor. Sometimes I make my runes as small as 1.25" or 1.5". That's pretty small for a 20". Are you using a mouse pad (with the cloth top they first came out with)? Reason I ask is that I installed this on my wife's computer also and she never uses a mouse pad. I found that trying to draw a straight line on a slick desktop was nearly impossible. I know it's a long shot, but I just thought I'd ask if this was the case...
One more thing - I am running patch 1.18 with no problems (the original game disc). I didn't buy the game here (with 1.19). I doubt if that would make a difference, Though. Good Luck.
NOTE: If all this fails, have you played Gothic 2 with the expansion? Available here at GOG. It's 3rd person, but that never bothered me. Not with this game. Greatest RPG ever (IMHO). Gothic 1 is also great (the controls for both are a little hard to get used to at first, but when you do they become second nature and to me it makes it feel more like you're really in a battle. Just thought I'd point this out. But, if you're a real RPG fan then you probably have already played these 2.

Hey cbolin, I know that problem. It took like 50 tries for me to successfully draw the "Yok" rune. I improved since then, but it's still a pain - no less than 5 tries each time. I shudder to think of the runes that will be needed next.
I think that's some kind of error that exists due to our newer OS'es or hardware - because back in the day when I played AF on my first system (Athlon XP1800+), I used to cast spells like Vulcan auto cannon .. just bang bang bang. Now it's the whole different thing, it takes enormous effort just to get one rune recognized. First I thought that's the resolution issue (perhaps mouse movements were calculated in some intricate way that was only tested on 1024x768). However when I switched to that res the issue remained. So now I think that's the continuity (fluidity) issue. Check the other user's posts.. you could walk then *pause* then you're fast-forwarding to the end of corridor. I never encountered that however I get severe lag when any font is on the screen - many others reported it too. When I draw the runes the lines sometimes break in mid-air. So I think *something* interrupts the gameflow, it's *realtime*, which is something that may be taken into account by the gestures recognition system (remember, the runes should be drawn in some finite amount of time).
Whoa, I've written a lot of stuff.
Basically - I suspect engine timer issues.

Hey, same problem. I can finally cast the Yok rune relatively regularly by making sure the three lines are distinct. 1 down, 1 right, 1 up. Do NOT round corners, even slightly. It helps to pause a tiny bit before switching direction.

Try downloading the free SpeedSwitch XP, which is actually designed to save battery life on laptops, but works great on PC's by slowing it down. Activating the 'Max Battery' in the taskbar icon after you start the utility, at least on my XP PC, brings my cpu sped down to 1,000 instead of 2,800 on each CPU. It has helped on lots of games, like Outcast and Freespace, etc. and may help with creating runes with Arx Fatalis.

Same problem here. Using of runes is very difficult for me. However, I found that creating of very small runes makes rune drawing much easier.
But it is still much harder than similar spellcasting system used in amazing PS2 game Okami. I think playing Arx with magic character must be really hardcore.

"I think that's some kind of error that exists due to our newer OS'es or hardware - because back in the day when I played AF on my first system (Athlon XP1800+), I used to cast spells like Vulcan auto cannon .. just bang bang bang."
Hey, i kind of wonder... I'm drawing my runes *really* sloppily and it takes them. Like, really huge, sloppy runes, with weird angles to them. Takes them just fine. I'm running this on XP with a 2Ghz Centrino Duo and an Intel 950 GMA for the graphics, so it's a pretty low-end system.
If anything I think running on a dual-core system might be a problem, since it can cause timing issues (albeeeeit I'm not experiencing issues with the runes, as I said). Maybe this is causing a hiccup in how continuous it detects your lines as being. Maybe if you alt-tab out and set the processor affinity to one core in the task manager it'll improve things.
"Check the other user's posts.. you could walk then *pause* then you're fast-forwarding to the end of corridor. "
I do have this. Even if I set the processor affinity to only one core.

Since you have an AMD CPU, try the AMD Dual-Core Optimizer. This reportedly fixes such speed issues in older games.

Centrino's intel, not AMD.