OK, I encountered a creature not documented anywhere with nothing to do but exit the game.

While on a boat, I entered the harbor in the Isle of the Avatar ready for the pirates, dealt with most, but then with no boats next to me I was attacked by Phantoms.

Except these were unlike anything I've seen or read about in Ultima IV, the phantoms were ships, with icons of normal ships flickering between different facings. Now here's the kicker: they did not attack nor move. Simple enough except for one problem. Being on a boat, I cannot attack all of them. The ship combat view leaves angle you cannot reach without diagonal attacks, and one of them is in such a dead space. I also don't have the Undead spell mixed, and no mixing in combat.

So now I'm stuck in a combat with a monster, unknown in any online docs I can find, that won't move or flee, which I cannot attack, nor even retreat from.

Did you try Tremor? That spell can hit anywhere on the battlefield, and should have a chance of either killing the enemy or making it flee (especially since it is next to the edge of the map), unless the enemy has too many HP for that to work.

If that's not an option, or if the enemy has too many HP, then I guess you've found a softlock.

(By the way, this would not be a softlock in console versions, as both NES and SMS allow you to attack at arbitrary angles, and NES also lets you run from sea encounters. With that said, in SMS, if this glitch can happen there, it would still be a softlock if you don't have any weapon that can reach that enemy (slings have limited range in SMS), you would have a problem. One the other hand, the NES version is different enough that it probably doesn't have the same bugs; it doesn't even have the hoard of pirates in that spot, nor does it even have the wheel, hull durability, or out-of-battle cannons that make this part of the game play the way it does!)

Edit: I have had this happen in a different game, Final Fantasy 6; the enemy turned invisible, I only had Gau and Umaro, both set up to use only physical attacks and healing, left, and the enemy's attacks were healing my characters; also the battle was marked non-runnable for whatever reason.