I really love the city and economy building aspect of the Settlers games, but I hate the military aspect. Is it possible to start a game without an opponent so you can just focus on the building part?

Yes it looks like it's possible.

If you go to Single Player, Freeplay Game, after choosing the map you'll see the AI that will play against you.
You can choose three things; starting location (HQ), AI/Human (MP), Team.

The second button labelled AI, can be selected and set to CLOSED.
This should disable that AI player.

Thanks for verifying this for me, Vince.