So, looks like one more time I won't be able to finish Kingmaker. This time I'm stuck at Against all odds.

I actually cleaned Castle of Knives before Against all odds triggered. I just happened to be around, the location popped up, I checked it out and did it. In the end
[spoiler] I let the guy take care of the soul in the crown[/spoiler] which seemed to please everyone (except maybe the soul).

No, when against all odds triggered, I cleaned out Bald Hilltop one more time, then went off to Thousand voices and saw the exclamation mark over Castle of Knives.
And now I'm in there and exactly nothing happens, no portal, no enemies, no NPC, nothing.

I'm I done with PFKM again or is there some way to fix this?

Did you already have the scene in your Throne Room with all your companions? That has to happen before you go to the Castle of Knives for the portal to unlock. Sometimes the scene can take some time to appear, so you might want to advance time a bit if you get to your Capital and nothing happens.

Yeah, I had that. I came off the kingdom management screen and all the companions were there and each got to say their piece. Very movie like. And then I set off to Bald Hill and then Castle of Knives.

I also had this issue. Saving, closing the game, restarting the game, and reloading while in the Castle of Knives area fixed the issue and caused the NPC to appear.