If you've got the food to support them, more population means more workers means more resources at your disposal. Food is relatively easy to produce (especially if you're using bread, which produces much higher yields than the alternatives), and it's not hard to build additional taverns and chapels to keep up with your population.

The biggest problem with bigger populations is that it makes your castle bigger, requiring you to build more walls and have more defenders to cover the area. You really don't want to have any significant infrastructure outside of your walls (generally just woodcutters, whose workplaces are very inexpensive), because even a relatively small probing attack will trash everything.

In terms of resources, wood is top priority because all other workplaces require wood. Then food to feed your people, then stone and weapon production.