SPOILERS

How do I convince him to leave his exile? I've got 18 int, 16 Charisma, 3 persuasion (and expert), I've talked Randver about the stone and shape and still, after conversing with loghaire for a while about it, he tells me to leave him be and refuses to speak again! What more do I need?

You need to use the correct dialogue options.

You are right! It was my fault: I didn't pick a line that seemed a bit too rash but was the right choice!

Visit him after a while, he can join your team. Also, there's cool dialogue when Zan Alurin is in your team.