I keep looking at walkthroughs but I can't find one that tells me how to get Virgil back when he leaves to go take care of whatever he needs to take care of.

When he leaves after T'sen Ang?

You have to go to the Sobbing Onion (which is a pub in Caladon) and then go to that establishments basement.

If you want more spoilers beyond that let me know.

Do you know where to get pilgrim robes to get into the bottom of the temple in Caladon?

There are some in Shrouded Hills (in the big building that has rats in it - the deserted Panarii temple), but if you got those and then sold them ages ago then I'm not completely sure. I think you can steal some from the temple in Caladon itself.

Other than that, there are a couple of ways to get into the crypts below the temple of Caladon. Ask the various people in the temple.