It depends on two things; relative weapon speed differences (i.e., their weapons may be faster), and how relatively new you are to the combat system. There is a significant learning curve to melee combat system in M&B, and if you're just starting then you have a double whammy in effect ... your character is weak and undeveloped, and so are you (the player). Both are remedied with experience.

That said, I've logged a couple hundred hours into Warband, and I freely admit that I suck at foot-soldier melee combat. I do alright against a single foe, but against multiples I tend to lose (as realistically should be the case). I much prefer archery and mounted combat (where I can run away like brave, brave Sir Robin) ... I lean heavily toward mounted archery ... but that isn't always an option when caught flat-footed in an in-town ambush. However, having your character gain levels, increase strength, and increase weapon skills does help ease the pain.

Good luck to ya.