I've sunk in many hours into the original but I wonder if it's inferior to it's successors like Warband? There's so much stuff that irritates me but it's doable.

Should I skip it entirely?

You cannot skip it entirely if you have already sunk hundreds of hours into it.
I prefer it to Warband but that is me. Some like the first game better due to several large mods for it. Did you try the mods? Did you try the Warband demo yet? That does not show you late game play though. The scope of the original game is quite narrow, something not immediately visible when you start. For me it is a game for a tired brain where I only have to seriously think a few times per game which way to take my army. I did not like a couple of the combat changes made in Warband but it does have more options available to you, what I could see.
Off to my sixth victory.

To answer your question: Get more informed and make your own decision. If vanilla, I think more people like Warband.

Warband is definately better, it fixes some issues (better camera for instance especially if you use widescreen).
Since you're already accustomed with the base game you might want to try the Bannerpage mod which enhances Warband somewhat.