Gundato: Take a look at UT3 (and, to a lesser extent, 2k4). Those flopped because the core game, while interesting, was largely expected to be overshadows by mods (and why not? look at UT and Quake 3).
1) What on Earth gives you the idea that UT2004 flopped in any way?
2) While UT3 was certainly not as successful as was expected, the reasons are varied, and expectations of mods over core game is far down on the list, if it is there at all. Major contributors to the game's failure are:
a) At the time of release, the game was an obvious console port. This is a bad thing to be for the next installment in a historically PC centric multiplayer FPS franchise.
b) At the time of release (and for several months afterward) there was no Linux server for the game. The core audience they were hoping to entice into buying the game (the large UT2004 fanbase) and to run servers for it, were all running Linux servers. From a marketing standpoint, the low priority on making a Linux server available to the very people who needed it to provide Epic with a market for their game (by hosting servers for it), must be said to be catastrophically stupid.
c) The singleplayer part of the game veered wildly away from the established formula of the franchise. The game is called Unreal
Tournament 3, and they removed the tournament part from the game. They kept the gameplay, but shoehorned it into a far fetched, illogical plotline, revolving around a war between humanity and an alien race. This led to some ridiculous explanations of how the gameplay fit into it all, e.g. the Field Lattice Generator, or FLaG. This may have been an attempt at humor, but was not well received.