I installed the Circle of Eight mod (without the New Content).

I assumed that the setting in the front end for max NPC hit points referred to NPCs that could join your party--not every goblin, spider, and lizardman in the game.

Once I left town, I found out I was in error.

How can I fix this? I went into the front end and unchecked the boxes. When I loaded up my save game there was no difference. Do I really have to start the game over? If so, can I use the same characters, or do I need to create new characters?

Unchecking the boxes might not be enough, because the state of already created monsters could be saved in your savegames. Thus, the change only affects new spawned monsters.

If you're using Win Vista or newer, you shouldn't install ToEE into the Program Files folder. Windows might prevent correct reading or writing due to security concerns. This can cause problems, like not applying your options, as they're simply not saved. To be save (re-)install the game to a custom folder.

If your unchecked options don't take effect at all, you might have to re-install the game.
Mod-savegames can't be used with the vanilla version.
You can try to use your savegames with an identically patched version of Co8 Mod (even with a different configuration).
There's a chance to "import" some former settings/effects with your old savegames, though (e.g. already created strong NPCs).
Characters shouldn't be used in older versions, but you can try. If it doesn't work, you'll see right in the beginning.

On the Co8 website are additional tips to keep in mind:
http://www.co8.org/forum/showthread.php?t=7737
Scroll to The Co8 Guide to Successful Gaming with ToEE