I would advise you to spend the skill points depending on your style of gaming. For example, if you usually roll and try to eliminate as many enemies as you can with your sword and you are doing it well that way, I would advise you to specialize in Swordfighting. Otherwise, if you prefer to use Signs, you should specialize in this -the most important signs to improve would be, for me, Quen and Igni, because the first allows you to reflect the damage you receive towards your enemies, and the second one because if you improve it you can make enemies burn and lose a lot of health points-. I don't recommend you to specialize in Alchemy, although if you have some skill points left after specializing in Swordfighting/Magic you should invest a couple of points in skills allowing you to improve the effects of potions. Provided you use potions very often, that is.
Another hint: if you play on Hard or Insane make sure you improve your equipment as much as possible -steel and silver swords, and armor-. If you don't do it, the last fights will be reeeally tough -personally, I found the last monster boss really hard because I fought against him with a silver sword from Chapter 2, far worst that the ones I should have obtained in Chapter 3. On the other hand, I got a really good Steel sword, a Elven sword of 40-42 points of damage and I could deal with human enemies in the last chapter as usual-.