To me it seems like the complexity is just irreducible. There's so many formats, so many bits and pieces that can go in a video stream, they're not very visualizable, and they have surprising edge case interactions. Not to mention there's a lot of "normally the program figures this out for you, but there's an option to override it if broken" knobs and dials.
Now, all of that aside, I do like Handbrake and I do think it offers a ton of functionality with so little friction that it's one of my very favorite and most-used apps. No login, no project setup, no x, no y, no z. Just a thin wrapper around a badass command line utility, with tons of options for users to override, and sensible defaults. There's a lot to love about Handbrake!
But "my grandma can use it", or "a plumber can use it", or "a person who doesn't understand the technicals and just needs to do one stupid thing that the app can definitely do, can use it" are signs of good UX. You wouldn't say any of those things about Handbrake.