I only have age of wonders shadow magic v1.3. Is there any way that I can play the LOTR mod?

Which LOTR mod? There are several LOTR mods AFAIK, and even more LOTR maps.

You may need the 1.4 unofficial patch.

This one:
http://aow2.heavengames.com/downloads/showfile.php?fileid=386
Are there better LOTR mods around?

Yes, this is a good LOTR mod, and the best of those I've tried. But IMO there are no perfect LOTR maps. IMO the closest is the Middle-Earth in the 2nd Age 5.0 - http://aow2.heavengames.com/downloads/showfile.php?fileid=607

There is also a relatively new LOTR map by Freya, but I haven't tried that one, but I see it has lots of downloads, so it may be worth trying.

I found the map in question, although incredibly detailed, to be too slow paced, having to wait for scripted events to happen before I could set my military plans into motion.

I found Lord of the Rings in the Third Age - Battle of Azanulbizar - http://aow2.heavengames.com/downloads/showfile.php?fileid=399 - by the same author and using the same resources/mod, to be better paced and more enjoyable.

But as I've pointed out in my comments at Age of Wonders II Heaven, these two maps by Black Rose have a couple of problems:
Save games got corrupted. I didn't see others complain about this, though, so it may be a problem at my end.
Too many units. So many in fact that you can't defeat one "army group" in one battle, but often need two or three battles.

For these maps you do not need the 1.4 UOP, but they use their own resources, so make sure you install them correctly, and (very important) set AOW:SM to use their resources using the AOW:SM Setup program.

Ok, I can start the map, but then the screen is messed up (mouse cursor leaves a trace, windows do not close)

I have a six core computer with 8 Gigabyte RAM. It is not the most modern, but it should be better than anything that was out when this game came out. And I think the problem only appears once the very first message pops up (there is like one second, where it still works but then a message comes). Also the bottom list were the menu's are is completly fine, only were I should see the map, everything is messed up.

Edit: Sorry the problem was related to the game as a whole and not the mod (the game worked flawlessl on earlier PC's so I assumed the problem is the mod, but I didn't try to play normal games before playing the mod) I have to play it in windowed mode on this PC.

I can't say for sure, but AOW:SM was one of the games that I had to reset the affinity on my computer to one core, just to run the game right from the get-go. I could get the game to start, but after just one or two turns it would lock up, or just crash. Once I set the affinity to one core, it ran like a charm. I use "Launch & Set", a free utility I found here: http://www.brothersoft.com/launch-n-set-141709.html.
I like it, because once you place the game's exe file into the target line of the utility and set it to just one core for your computer, you just launch the game from the utility (or any other game or program you add to the utility), and it starts it automatically for you, saving the time and bother of having to manually reset the affinity everytime. btw, it's a very small, simple program, and uses none or practically no resources at all to use. If it doesn't solve your problem, no harm done...if it does, like it did for me, good times!