cogadh: GOG seems to reject games for arbitrary reasons. I think more often than not, they reject for aesthetic reasons, as in, they feel a game is not going to be appreciated by their customer base, rather than technical reasons. They are unlikely to comment to us on the exact reason this particular game was rejected, but they might tell the dev if he asked. There is also the possibility that the game has not actually been rejected, it just hasn't been released by GOG yet. They do hold back releases to maintain a steady weekly flow of new games, so it is possible that Nitemare is still on the wait list.
EDIT - Looking at the website for the game, the one thing that sticks out to me is the lack of 64 bit OS compatibility on the Windows version. That's a huge chunk of the customer base effectively excluded from the game. There is the DOS version that can be run in DOSBox, but as has been seen before, a DOS version can sometimes be seen as the "inferior" version of a game and without 64 bit compatibility, that's really the only version GOG could sell.
Having tried it, the Windows version is just the DOS version with a Windows 16-bit shell. So the DOS version would work (GOG is selling the DOS version of Dungeon Keeper since the Windows one doesn't work so well. Maybe it's a disagreement with the developer (who self-publish).