Well, for some reason my NPC Hirelings can't use the weapons they're supposed to be proficient with, or any weapons at all for that matter.

And now that I've finally managed to two-man my way through the cave of skeletons and found the portal, touching it crashes the game.

Help?

What Zaxares said. Although there is a dagger of throwing/axe of hurling in the game that I consider pretty good. I forgot which.
You can cheese certain enemies with it, enemies that could drain your level when they touch you.

Regarding the crash, did you use any save game editors or anything like that? Does the crash give an error message?

I didn't even realize it's a front/back row dichotomy. Any way to swap character positions to re-organize the party? Also I'll mark this because I noted the issue regarding the crash was trying to exit the caves through the portal while the NPCs that left my party had spells running. I rested and unequipped them and it fixed.

You definitely can swap them around, yeah. Again, I'm fuzzy on how exactly it was done, but maybe try clicking on the portrait of the first character and then the portrait of the second character?

Gotcha has a good point too about the Dagger of Throwing/Axe of Hurling. (Those weapons pop up in the SSI D&D games, but I couldn't remember if they showed up in Strahd's Possession too, or in their later games.) Those are actually decent ranged weapons to use if you want to stick a Mage/Thief in the back row and still have them contribute to combat.