When I cast a spell from either alchemist, mage or priest... if the spell is intended to hit a large group, i.e whipping rocks, blades, etc... it seems that if the monsters that were visible on the screen that represent the group, if they have been destroyed, the spell cast that should affect the rest of the group pops up as being cast yet has NO effect on any of the remaining monsters.

Is this suppose to be happening?

I've noticed it even goes so far as.. for example, if my alchemist cast a spell of stink bomb on say a group of 7 gorn spearmen, if the front 3 that are being displayed have been destroyed by melee or other means, the ones that show up in the back of the formation are not hit by the spell and take NO damage at all.

I am getting tired of investing spell points into trying to win a fight when, based solely on some apparent graphical glitch in the game, monsters that should be affected by a spell yet having that spell have no effect. The spell name pops up that it was cast yet, no monsters are displayed as being hit by the spell. It just disappears.

Frustrated.

This is normal, and part of the challenge.
Think of your characters as mindless drone that are incapable of adjusting in the middle of combat.