Yep. I can remember only a few incidents where modders were actually facing threats of legal action against them. And most of them were based on modders using assets or IP that they didn't have a license for (like creating a "Lord of the Rings" mod for any game, or taking models and textures from a different game and using them in the mod) - which is a totally different beast. In these cases, the legal threats came of course from the party whose assets and IP was used, while the publishers of the modded game typically remained silent on the whole issue.

As you and others already said - usually, nobody cares. Except when a mod is causing a ruckus, like the Hot Coffee mod, or like the nude mod that contributed to the ESRB changing Oblivion's rating from Teen to Mature. But even in these cases, the modders weren't actually persecuted. Bethesda and Rockstar just used their EULAs to maintain a better public image, by decrying the modders as "hackers" who had done something they weren't allowed to.