It can be a challenge to grow the population in the vanilla Banished. The problem, IMHO, is the aging is much faster than it should be . . best guess is maybe 4 - 5 yrs of aging for every year that passes. That causes the population to age quickly but also allows students to graduate much sooner giving you a skilled work force much sooner.
If you don't build new houses often the younger folks will not marry and move out on their own. So I ended up building constantly trying to keep up with the fast aging. If you click on your houses you might find only one old person living there which locks that house down to one person. One trick I learned was to click the update to stone button for the house which causes the older person to move in with someone. I then hit the reclaim button to stop the update and find myself with an empty house that younger folks could move into and start a family. Works most of the time.
I eventually got tired of the exaggerated aging and started using the One year is one year mod which slows the aging down to a realistic rate. The only downside, besides the ages being misrepresented on some dialogs, was it takes much longer for students to graduate. Once I reach a point where I have a couple of schools with max students it seems to work well. Once I have several max student schools I don't notice the longer time it takes to get educated workers. It can be frustrating to use the mod early in a game as your students are your new workers. You might give the mod a try if you want a more realistic aging.
One year = one year mod I have not noticed that reassigning workers affects growth. I do that regularly especially in the first ten to fifteen years. I did notice that rotating crops does not seem to make any difference in output or harvest. Eventually, with a market and town hall, you will get nomads. A few at first than 80 to 100 at a time. At that point I stop reassigning my workers. Nomads bring their own problems, disease, housing shortages and overwhelming of the food supplies.
At some point I usually build two boarding houses to accommodate their arrival but once their numbers are 100 or more I usually deny entry as it can cause chaos with that many new mouths to feed. =)