Possible reasons:
1) Known bug? Quote from
http://www.gamefaqs.com/pc/564960-master-of-magic/faqs/2059 :
Invisibility items don't make you invisible in combat.
(I'm not 100% sure about this as I have never enchanted invisibility items before...)
2) Known bug #2. Enemies always know where you are at the start of the combat. You need to move at least 1 square away before they lose track of your location. I've encountered this numerous times when using Nightblades & Shinbo (naturally invisible units). The enemy will charge towards me & engage in melee if I do not move my invisible units.
If that's the case, then your Malleus must have been continuously webbed by the spiders until the gorgons(unknown flying monsters) got close enough to melee him. I'll suggest sending in a few "junk" units in with Malleus to get the spiders to waste their web spells.
3) Highly unlikely, but maybe the unknown flying thingies you described is able to sense invisible? Either true sight (Angels) or immunity to illusions (Demons)? Off hand, I don't recall ever seeing a lair that contained monsters from 2 different spellbooks or a flying nature monster that is able to sense invis, but I could be wrong....
Hope it helps. :-)