If you both have 4 cards effected by weather, his 8 is more fun than your 4 (for him)

Exactly, not only more cards than you, but better cards.
It's easy to give him 5 points with Shillard, & get back less the 5 points with the two cards you pick up.
Then he will use a decoy on Shillard & get two great cards.

You can buy one at White Orchid, but he has two.

The biggest problem with the AI, is it doesn't know who won or lost the previous round.
So if you win the first round, it will often pass in the 2nd to save cards for a 3rd round it won't get to play.

Also if Baron takes off with his wife to the healer, you will get a red x for not playing Gwent with him.

I was not checking journal often and now i m starting all over again because i had quit a few red x in quests for not

playing Gwent

I wasn't suggesting you leave it that long, just play a few easier merchants first to get a bit of practice & a few good cards to make it easier, before you start his quest.

On my last playthrough, Thaler got killed before I had got the Geralt of Rivia card.