I'm curious about what this series has to offer. Where should I start?

I would recommend starting with Blood Omen: Legacy of Kain, but it's not available on GOG (or any other digital distribution site, I'm afraid), due to rights conflict between several companies.

If you can't get it, you can watch a story summary somewhere on youtube and start playing from Legacy of Kain: Soul Reaver.

as Tuthrick said, Blood Omen: Legacy of Kain is the first game you should play. these series are story driven, lore and story is the most important/enjoyable thing here. Starting from the very beginning is advised because of story aspect and because first game is very good and solid.

game is easy to find on abandonware sites. also you will need some patches like bink support for videos to play (important, as most of the story is told by these). this page has overview of patches available

pcgamingwiki.com/wiki/Blood_Omen:_Legacy_of_Kain

Just adding my vote to Blood Omen 1, especially since Verok's wrapper just came out of nowhere to make the installation and the experience even better than with Cuban Raul's patch and the HQ video packs (included in Verok's installer package). You get everything you can dream of with this tool: easy installation, no more technical issues, some options from the PS1 version, filters... I've already played BO1 with previous fixes and loved it, so it can ony be better now!


About SR1, some technical info which may come in handy:

The only episode that's still a bit of a pain to get working is Soul Reaver 1. Well, with some tweaks it's easy to turn the PC version into something really enjoyable, but the audio part is lacking compared to the console versions (some crackling in places, and mostly the dynamic music system is greatly reduced (basically atmospheric tracks for each main area rather than dynamic versions of them depending on the action - I think it's at times a bit over-the-top on consoles but the PC version goes too far the other way), and some minor SFX are missing (faint rain and thunder) - that's because of the specific system used on consoles but not translated on PC. You do get a very consistent experience graphic-wise, which is very easily boosted with tweaks, wrappers and shaders (game set to 800x600 and upscaled with dgVoodoo + shaders = cool graphics with all effects (like rain drops) as intended). Console versions have the upper hand there, especially on Dreamcast which also has improved characters textures (I prefer the original ones (PS1/PC) for the main protagonists though).
There's a fan-made patch for the Steam version which does a great job at fixing those issues (and others), but it's still not 'vanilla' (the music transitions are made much more robust, and there's an 'echo effect' added for the Spectral Realm etc.), definitely worth a try though (you can get it to work with the GOG version but it still involves Steam running, so in this specific case you're better off buying SR1 on Steam for now).

SR1's state is a bit frustrating then, and you may as well start with BO1 for reasons stated above, but also because SR1 might still get some more improvements in the meantime. But still, if you don't like BO1, this one is quite different and in all cases, it's a great experience as it is.

I skipped SR1 entirely after 1 hour of bumping into walls and such. it's a horror to play with keyboard and mouse, camera is so bad. steam version worked out of the box for me (on Linux with Proton) and it looks like they already integrated hack to play bink videos (you had to do that manually with first game). After I finished Blood Omen I can't force myself to play SR1, finished it on youtube for the story alone.

Start with BO1 and BO2. Or skip BO2 - it do not have any plot. Bo1 in any way - first. Then - obvious - play SR1-SR2-Defiance.