Where RL genuinely shines
RL is not a universal upgrade over hand-designed control, but for certain tasks it genuinely shines. The clearest cases are tasks that are extremely difficult to hand-design a control law for, because the number of interacting variables makes a hand-derived formula impractical. Dynamic legged locomotion is a classic example, keeping a robot balanced and walking over uneven terrain requires continuously coordinating many joints in response to constantly shifting contact forces, a relationship far too tangled for a PID loop to capture directly. Dexterous in-hand manipulation, like rotating an object within a robotic hand using only the fingers, is another, the contact dynamics between fingers and object are complex enough that nobody has successfully hand-derived a general control law for it.
In both cases, the appeal of RL isn't that it's inherently 'smarter,' it's that trial-and-error search through a simulator can find a working policy in a space where human-derived formulas simply don't exist or don't work well.