Does anyone know for certain how armor rating actually works in the game? Is is a simple formula like damage - armor = vitality loss? Or is it much more complicated. And how does the type of damage come into play?

It's the simple way, damage - armor.
You can easily check it activating the combat log.

Example: "Geralt hit for 50 - 10 <armor icon> = 40"

And if a skill says you get x% damage reducton you actually get x% armor bonus

As a follow up on this: I assume that 'x%' is based on Geralt's current armor rating? So if his armor rating is 20, then 5% damage reduction means his armor rating is effectively 21? Or if his armor rating is 30, then 10% damage reduction means an effective armor rating of 33?

One reason I'm asking about this is on another site they claim damage reduction is straight damage reduced and not applied to armor. For example if you have 10% damage reduction, 20 armor and 30 points of damage the two possibilities are:

1) Armor is 20 * 10% = 22 armor. 30 damage - 22 armor = 8 damaged received.
2) 30 damage - 10% = 27 damage - 20 armor = 7 damage received.

Plus how would posion resistence work then? I think in the logs I've seen 10 damage plus 2 poison damage. So not sure how it all fits together.

This is correct, yes.
I checked values just before and after taking a +x% armor skill This is just my guess:

Elemental damage (like poison or incineration) isn't mitigated in any way. But being "critical effects with a chance to apply", your poison/incineration/etc resistance works as "lower the chance to trigger the critical effect on you.

Let me add why I'm asking. Let's say you have a choice of 10 vitality or 2 armor. If damage received is just the basic damage - armor formula then vitality is a real possiblity based on number of hits..

For example I start with 100 vitality
10 damage - 2 armor = 8 less vitality = 92 final amount.
10 damage received with no armor = 10 less vitality. 100 + 10 additional - 10 = 100 ending.

Using these numbers it would take five hits for the armor to be better than the additional vitality.

high vitality is important, especially on insane. high armor is not that good, depends on playstyle, if you have quen (shield) up most of the time then armor doesn't mean much, if something goes wrong higher hp will probably be better.
also: even if your armor is higher than the enemy maximal damage they will still do some so called "minimal" damage which is not negligible.
I actually made an armor build, had 179 armor and 720 hp in the end, Geralt was strong, but a simple quen is superior in many ways to high armor (quen damages and staggers the enemy while even with big armor you still get damaged and staggered by "minimal" damage)

How do you get such high armor values? The best armor has +30 with 3 sockets for +6 more. Add another +6 to 8 for trophy and +7 for boots, gloves, pants gives you around 50 armor.

Strength mutagens and Madness mutagen also give a bit of armor bonus but it is +2 armor at most and even if you use only strength in mutagens and go with alchemy tree in one of your trees you'll most likely get 10 slots to use so you'll get 20 armor, add the bonus of impregnation II to that, you'll get 26,5 additional armor. I guess I had about 70 armor in my last playthrough(Vran armor with +2 enhancements, elder blood set and draug trophy) and about 4-6 of that was with mutagens. If i had gone alchemy tree + impregnation II + all armor for mutagens i'd have about 90-100 armor which is no where near 179.

I had 9 mutagens with +4 armor each, then the +% bonuses from skills and virga potion. could had been even higher, close to 200 I believe (there is an other potion which gives armor bonus, I don't remember its name, haven't used the best armor trophy because hp was important too and I could had get more skills to give me armor or armor bonus but other skills were more important

The description in-game for Madness Mutagen says it gives +4 armor. I've not slotted it to test, but that's what it's supposed to be from the description, not +2. Just as a point of interest. That's still not going to get you anywhere near 179 armor, even with 10 Madness Mutagens.

Impregnation actually doubles the strength mutagen's bonus so it will be +4

Here's a related question:

Given there's a minimum damage dealt no matter how high your armor, does anyone know what the optimal armor value would be?

adding it up I had 83 base armor, the virga potion gave +32% (with catalyst skill) and I had most of the skills which give damage reduction %