So I've been wandering off the main quest path a bit and found these caves dug in mountains. They seem to contain monsters, traps and loot. Now my question is: are these caves part of some sidequests or something? Or are you simply meant to use them to farm some loot and xp to help you on the main quest? If they have quests related to them naturally I would want to wait before exploring them.

Next question is about the trainers: what is the best trainer for a cookie cutter build? They all seem to boost different stats and I don't want to mess up my build by training at the wrong place. I have already trained 3 times at the starter town's trainer. IIRC he said he improved our hand-to-hand fighting abilities and I'm not sure if I made a mistake by using 3 training points on that.

Trainers boost your stats, and which stats get boosted depends on which trainer you use.

For each training point that is spent, either you get +1 to STR, DEX, or CON, or your Combat or Magic increases by half the difference between it and your DEX/INT (rounded up). (There are some shenanigans you can do by training while under the influence of a certain drug, but I don't recommend it unless you know what you're doing, as the drug does have permanent side effects.)

Note that no training point is ever wasted; if the stat is capped out, the training point won't be spent; however, you still need to have that training point to be able to train.

For the main character, since they are the only one who can use magic, I suggest focusing on INT and Magic. For other characters, it doesn't matter too much.

Note that, if you play the Forge of Virtue expansion, you will be able to max out all your stats automatically, so even if you were to somehow able to find a bad build, you can still fix it by playing the expansion.

Training works the same way in Serpent Isle; however, the only place to train INT or Magic is in the Silver Seed expansion, and even then, your INT and Magic start so high in SI that one training there will max out those stats.

What about the mountain caves?

It depends. I'm not clear if you have played previous games in the series, some of these are just caves and some of them are the dungeons that have been part of the series since Ultima 4 (and used to be considerably larger). They may or may not have quests connected to them

Regarding training, this will be helpful:
http://wiki.ultimacodex.com/wiki/Training_in_Ultima_VII

About the caves: Some are connected to quests, but it never hurts to explore them as you come across them. Often you'll find some sweet weapons and/or armour.

Keep in mind though: Some of them are pretty gruesome at low levels, especially the dungeons.
You'll know it's a dungeon when you approach the entrance and one of your party members remarks about it.
Be sure to train a bit first, get some decent armour, a few magic weapons (for enemies that can't be hit by regular weapons), and get a few useful spells (like healing spells, sleep, and unlock magic).

Without spoilers: The first dungeon you'll see is probably the one a certain quest character sends you to. This one is particularly hard. If you feel you're getting your rear end whooped, remind yourself that you're free to return to this location later. There's no rush; you have all the time in the world.

So basically if I enter a cave and my party remains silent I can risk exploring it?

You can always risk it, dungeon or not, but yeah, most caves should be pretty easy to walk through, even right from the start.
You can make a save every time you delve inside some dark place (saving often is always a good idea), just in case you regret your choice. But you can't die anyway, so nothing to worry about.
And you cannot break any quests. Dungeons where you're not supposed to be yet have their own way of keeping you out.

In short, go explore to your heart's content and don't worry about anything. If monsters are giving you a hard time, clear out, return later. :)

since you can save your game. Of course save your game then explore it. Early on you shouldn't have problems with like headless etc enemies. Dragons? maybe not now.

As for training, For the most part you'll get 1 training per training point spent, so really doesn't matter where you train in general, just don't train magic on your other characters.

BUT based on how you get 2ndary attributes (combat from dex, etc) you never want to train 2ndary stats only because when you train primary stats, you'll get 2ndary stat increases at certain trainers.

For instance, early on you start in Trinsic and there is a trainer there that can train 1 combat for 1 skill point. This is horrible because many trainers can train dexterity, which will usually raise you 1 combat anyway so you'll get more benefit training a primary attribute than the 2ndary one (combat).

For your avatar, since you can do magic, you'll want to try and train that up, while your other companions who are basically pack mules can train up other stuff. Very soon you'll be in Britain just north of Trinsic, and there is a trainer there that will train you dexterity/magic/intelligence for 3 points PLUS you'll get a bonus point in combat from training dexterity. So, 4 points for 3 training instead of your early on trinsic trainer who only gives you 1 for 1. Thus, don't train in Trinsic lol.

There is also a trainer in Britain i think who trains 1 skill point for 1 dex, which also gives 1 combat as a bonus. so 1 point for 2 skills. Nice!

Your companions you might think of finding a trainer that raises strength for more carry since they are packmules/meat shields anyway, but its not a huge deal. Of course get them maxed at the Britain dex trainer asap to get their dex and combat up, they all need it.

Actually, a +1 combat trainer won't raise it by just +1; it actually raises it by half the difference between it and dexterity, rounded up. So if you have 18 dex and 10 combat, training combat +1 will raise it to 14. If you go on drugs to raise dex to 28 and then train while high, your combat will increase to 19, and will remain there even when the drugs wear off and your dex is down to 16. (Be aware that this also costs str and int.) +1 magic works the same way.

So, if a character's dex is significantly higher than their combat, +1 combat training might be beneficial.

This mechanic is also used in a few other places:in the game: The Heal spell restores half of lost HP, and each hour you regain half of lost MP (unless you're playing Exult which got the hourly HP/MP regen wrong as of when I last checked).

Of course, this is going from memory, as it's been a while since I've played Ultima 7, and I have no desire to replay it, as the game just isn't that good.

no i think this rebound method only works if it raises dex and combat at the same time. I test this with Spark at the start of the game.

Trinsic trainer trains 1 skill for 1 combat point. Spark starts with 6 skill points, tried him 6 times in Trinsic, his combat only raises 1 point each time for +6 combat total. 22 starting dexterity, 10 combat. He went up to 16 combat.

Now, i went to Britain and trained him with the dex boxing combat trainer. for 2 skill points he will raise you up 1 dexterity and 1 combat point (actually 2, because your raising dexterity).

With this trainer, because he's raising dex AND combat at the same time, the rebound works. Spark goes from 22 dex/10 combat to 25 dex/24 combat, so 6 skill points gets his +3 dexterity and + 6(3, +3) combat, PLUS a bonus 8 combat points due to the rebound effect.

So i think you need to train the main stat AND the 2ndary stat for the difference rebound training to work. Otherwise he would have gotten more than +6 combat from 6 skill points from the trinsic trainer.

Same with the magic trainer in Britain for the Avatar. 3 skill points for +1 dex/intelligence/magic. So +2 dex, +2 intelligence, and + 2 magic using up his 6 skill points.

Avatar has 18/18/18 strength/dex/int and 14 combat/10 magic. He goes to 18/20/20 (+2 dex, +2 int), and 16 combat/18 magic (+2 combat due to the +2 dex raise), and +8 magic (+2 trained, +6 bonus rebound)

So of course this is just with the first couple town trainers,but it seems you don't get any bonus rebound training 2ndary abilities (combat, magic) UNLESS you also train up the primary skill in the same session. Thus just training combat in Trinsic with the Avatar/Spark didn't give any bonus rebound, only +6 combat for 6 skill points, 1 for 1. No dex was trained, thus no bonus rebound. Yet when training in Britian where primary stats were trained, you did get the bonus for the primary skills, plus rebound effect. If there were trainers that only trained magic or only trained health, those probably wouldn't get any rebound effect because intelligence or strength weren't trained at the same time.