IMO, D&D is basically the most poorly designed and badly balanced combat system to be widely used. Literally everything else I've ever encountered across every other RPG,- be it turn-based, real-time-with-pause, MMO, FPSRPG, ARPG - heck, even JRPG, is better.
Here are my biggest gripes with D&D combat:
Martial classes are basically autoattack machines. There are a few passive feats that generally do boring stuff like +1/2 to hit or damage, and then highly conditional mechanics such as Cleave that are situational at best and largely deadweight at worst. There's generally an utter lack of "special attacks", and classes that have those tend to use a charge system that recovers after resting like a caster. Outside of these, you have modes like expertise, power attack, and rapid shot that quickly get boring.
Extremely bad caster balance. There's an old D&D saying regarding "quadratic casters". I would actually argue that casters in D&D scale cubically. As a D&D caster, you get more spell slots (or charges for the spontaneous caster types) in each spell rank as you level up, you get access to higher level spell ranks as you level up (face it, anything really worth using in combat comes at rank 3+), and the effectiveness of above-mentioned spells grows inherently with level and also with the difficulty class that prevents enemies from saving.
I also don't like randomly losing spells to passive RNG-based interrupts (concentration checks)
I personally find the memorization/casting level charge system to be clunky and a poor experience compared to energy/mana systems. In most cases, it only serves to prevent people from adapting to the situation at hand.
Extremely poor support for dual-classing and hybrid characters. If you don't progress your classes evenly, you're punished with an experience penalty. Casters inherently make bad hybrids for the most part due to cubic scaling. If you dual or multi-class, you lose out on important higher-level class features. General rule of thumb is that if you want a hybrid character, you're almost always better off picking a true hybrid class such as a ranger or paladin than mixing 2 classes to get what you want.
Extreme reliance and dependence on consumables. Failing to have enough consumables can lose you the game no matter how good you are; overloading on consumables can allow bad players to skate through content they are obviously unqualified for.
Extreme reliance and dependence on resting produces an un-fun attrition-based meta. A good combat system allows characters to dynamically recover in the middle of combat instead of forcing them out through depletion of resources that will never regenerate in combat.
RNG and dice everywhere. Probably a symptom of being an outdated tabletop game. While this makes for interesting roleplaying, it makes for an extremely poor combat experience that's imbalanced at best and downright nonsensically frustrating at worst.
Too much passive event hooks from feats instead of an active-skill system like a modern MMORPG. For those of you who aren't programmers, an event hook is a function that's called whenever a certain type of event happens, and event hooks can trigger other event hooks. In a tabletop game, a poorly designed ruleset littered with a minefield of event hooks can quickly have both the DM and players tearing their hair out about the order in which such events trigger and how to handle those that conflict.
There are too many ways to build an inviable or downright bad character. Some people say that's "old-school". I say that's ridiculous, unfriendly to newbies, and a lame excuse for bad design.
Frustrating "comedy of missed attacks" at low levels. The problem is that the base chance to hit is 50% (no bonuses vs. no armor), which means that combat at low levels quickly turns into a frustrating stalemate of missed attacks where neither side can do much of anything. I much prefer the modern system in ARPGs, some JRPGs, and modern MMOs where characters hit and do damage by default except if there is some special condition that might cause someone to miss their attacks (for examples: cover, attacker being blinded, invisible target, evasion buffs, active defense cooldowns, or an exceptionally high evasion/agility value).
Too many broken, un-fun "save or die" mechanics that take your character out of the game for failing a dice roll. These shouldn't exist.
===
As for what the "best" system is, I'll give the award to the best MMORPG of all time - the original Guild Wars (the modern Guild Wars II uses a dumbed-down system and doesn't count).
Mechanically speaking, Guild Wars is basically the anti-D&D.
First and foremost, my favorite thing about Guild Wars is how it effectively removes consumables. I've often said that consumables promote bad combat balance and bad gameplay. On the low end, a consumable-based meta encourages bad players to win by throwing a backpack full of wands, scrolls, and potions at enemies. On the high end, some D&D-style RPGs might include content deliberately designed to be too difficult so players are forced to rely on consumables, which means that no matter how good you are at the game, you lose simply for having an empty backpack.
In the original Guild Wars, there are generally no consumables that affect combat. You have a constantly-regenerating energy bar, a health bar that conditionally regenerates on its own, 8 active skills, and 8-10 attributes (proficiencies) that back your skill.
Regular skills have an energy cost, a casting time, and a cooldown. Warrior-type classes have special skills that charge-up when attacking or doing damage. Generally, energy costs and charge-up are used to set the pace of skill usage, with cooldown mainly serving to limit the spamming of skills that would become unfair, un-fun, or mechanically abusive if they didn't have a cooldown.
Unlike D&D and old-style MMOs that draw heavily on D&D, there are no inherent penalties or "casting pushback" for taking damage or getting attacked while casting. However, there are "interrupts" that can be used specifically to interrupt enemy skills. An interrupted skill immediately goes on cooldown and the energy cost is forfeit. Additionally, many skills impose some form of additional punishment on an interrupted enemy, such as damage or further extending the cooldown time.
Energy itself regenerates over time, dependent on the base armor value of your primary class. Health starts to regenerate if you stay out of combat for a certain amount of time (this means you don't take damage or receive hostile effects, and you don't take any hostile actions).
Importantly, maximum energy and energy regeneration in Guild Wars don't scale with level. This eliminates the balance problem of "quadratic casters" in D&D. To increase your energy regeneration, you need to put special skills on your skillbar known as energy management that help you cycle energy in more quickly or otherwise regenerate it. Some skills can even steal energy from enemies.
All classes have a wide array of choices for skills to put on their bar. Martial (fighter) characters have special attacks in lieu of spells, and the scaling for both casters and fighters is kept balanced.
Characters in Guild Wars always hit with attacks and spells by default. If you miss, it's due to some special condition. For example, blinded characters have a 90% chance to miss with attacks. Alternately, there are active defense buffs that cause you to block attacks. There are some debuff spells that can additionally cause missed or failed attacks.
Dual-classing is a fundamental mechanic built into the game. The energy system is purpose-designed to support dual-classing and keep energy usage fairly balanced for most class combinations.
The effects of RNG are kept to a minimum. Most effects are thoroughly nonrandom, and those that are can be mitigated or countered.
===
Closing thoughts: modern mobile trends
I generally don't like the modern trend of removing energy/mana and using only cooldowns because it takes out a dimension of skill from the game. This trend comes from the mobile dumbing-down of everything and generally flattens the skill curve - which decrease the opportunity for good players to make a great build that maximizes energy/mana economy while bad players go OOM. The other problem with a strictly cooldown-based system is how it generally pigeonholes players into using only "regular attack" while waiting for cooldowns or the best opportunity to use a cooldown.
