I found the correct house with all the Blue Stripes, but none initiate the quest. I get some "floating conversation", but nothing else.

What am I missing?

P.S.: Might it be possible that you need at least one point in Dagger Throwing?

You don't need any points in dagger throwing. You need to talk to the first guy sitting at the table on the right when you enter the second room. His name begins with an "F".

You can't start the quest at just any time. Sometimes when you go there they'll have nothing to say.

If you just keep checking back periodically, you'll eventually walk in on something different than usual. Then you go into the second room and talk to someone (guy at the table, but Ves works, too) and you'll start the quest.

I don't think you can party with the Blue Stripes until after you've gone to see Loredo. It might just have to be nighttime though (I'm in Flotsam on a playthrough and after meeting Loredo I walked over to the Blue Stripes and did the Party quest).

So you can either just meditate until night and try it to see if it will start, or meet with Loredo (which must be at night) then do the Party quest afterwards.

Edit: BTW, you don't need to spec in dagger throwing to do the quest; in my first playthrough I didn't take dagger throwing and was able to do the quest.

I did it before seeing loredo, but after doing the Kayran. So that might be the trigger for the quest.