If I understand correctly, you are using the feline set. If so, as with any Witcher gear, it's usually best to use the complete set and not substitute other pieces of armor. Unless you are just enjoying with determining how high you can get your critical damage, you may discover that the protection you get from a good set aids Geralt by mitigating many damage types and allowing him to get in more hits against opponents. I especially like damage mitigation from monsters. So, to answer your question simply, a full set of witcher armor is best, in my opinion and as learned with experience.

I would seek the Oferi merchant and fund him to where he can make special runes and glyphs and add sockets and ultimately enchant your swords. It is worth the coin, yet you will have to do some massive grinding to come up with the coin. I prefer to increase the range of Whirl through the Whirl Skill in the Swordsmanship Tree.

These are just my opinions, yet the game is more fun because there are so many different ways to build or work your character that you can make the game your own. There are some great build suggestions online, yet ultimately you will enjoy doing things your own way. Have fun, whatever you decide to do.

From a statistical point of view, if your chance of inflicting a critical when wearing the Assassin Gloves is greater than 30% you will inflict more damage if you change to the Nilfgardian Gloves.
e.g. Assume while wearing the Assassin Gloves your Chance of a Critical is 50% and your critical damage is 100. 50% x 100 = 50 damage per strike on average.
With the Nifgardian gloves your Chance of a Critical is 40% but your critical damage is 150. 40% x 150 = 60 damage per strike on average.