[I used to previously work at Google on Android TV, all opinions my own]

There's a constant push for more and more features because consumers and cable operators shop based on that. There's also a constant push for ridiculous BOM reduction to the tune of individual cents being saved. Android TV is non-trivial to monetize so it's hard to dedicate a whole team for performance improvements. There's an imaginary bar for "good enough" (read not too shitty) and you get this result. I hate the result but I can't see what market dynamics might change it.

Unsubsidized "HDMI output" type TVs would cost so much that they would sell poorly, resulting in even higher prices, ad nauseam. I think we're stuck with this.