So, I'm having a problem with the enchanted gem puzzle. At most, I can only place two gems in the empty grid (and according to the screen shot in the hintbook, the two colors I place are reversed).

Also, there is a spirit running around outside that runs to "the southeast" that I can't catch.

How do I get into the nameless library?