Externica: Well, the circumstances differ a lot. Ivy seems to be downright insane. Her first line? "It's raining blood."
Even with the patience of a saint, you're going to hit your limit if you have to endure such rambling all day and night. I was annoyed by Ivy after a few seconds and I consider myself to be patient. At least Joe's first line is about finding a doctor for his wife.

Alan on the other hand.. well, let's just say I don't want to start an argument about a different game here.
Curious reader can look up the Alan Wake-forum, there's a topic about someone asking whether he should buy this game or not. It's either there where I list my problems or linked to another topic about why I dislike Alan Wake (the character).

In any case, my take on Joe might change as the game progress. So, nothing is set in stone yet. He might be as bad, worse or not so awful in the end.