I have never seen my settlers man the Towers in case of a raid. They seem to be pretty useless at this stage. Or is there anything I overlooked?

They will only man guard towers when the alarm is raised, otherwise they will do other jobs or attend to their own personal needs. In my experience the number of guard towers necessary to actually protect a decent-sized village is so large that it'd take more than half your population to staff all of them, so you really wouldn't want to do that anyways.

Guard towers are very fragile and will actually collapse if enemies so much as throw a couple spears at them, so you really need to build 2-4 at every gate to have a reasonable defense.