For control… it's really a question of balance. "all the things" sure would be nice, but with a mid 4-digit price I think putting a small fully-open Linux embedded board inside would be awesome. Some existing SBC or SoM is preferable over rolling your own, the latter would just splinter off a separate community for no good reason. Also make sure the embedded system has full control over all functions (there should be enough GPIO & I²C…) and has its own power control. Having it able to actually drive the TV display would be nice (and easy these days) but isn't even the point :).
If you don't stick that in, how about a slot/connector for a Raspberry Pi compute module — or some other reasonably sourceable similar module?
Barring that, RS-232 for control is a bit dated, I'd really expect both an USB device port (control in) and an USB power output, ideally USB-C PD capable, to run some small system off. If you have the pins & functions, please spend the extra $5 and wire up leftover "random" interfaces, e.g. CAN or RS-485. Ethernet without adding a full SBC is kinda "meh", putting together some embedded OS with networking is significant effort for very little return.
Honestly to compete with mass market TVs, it's going to be difficult to avoid have more- not less "bloat". Built-in Android would be a great selling point (saves you $200 on a NVidia Shield as long as it doesn't track you or lock down features). Built-in AI upscaling is a good selling point.
I would also think if you have a OLED or edge/direct LED backlight LCD then you also need to run much more complicated pixel/zone brightness algorithms....
There's probable more memory allocated per frame on a modern 4K TV than an 8-bit computer had in total.
No, the reason that UIs are slow are because they are often a web browser, running a Javascript bloatware UI with it's shadow DOM or whatever, and there's a heaps of bullshit going on before anything happens on screen at all.
And this is fine! For many use-cases, comparatively beefy PCs, web pages etc.
But for what is effectively an embedded product which will get updated seldom if ever, THIS is the time to take a step and rethink what you are doing.
For a non-smart TV you definitely don't need Linux, (but you could!) and you don't need to care if it's realtime or not. There should be only one or two processes running anyway, so there shouldn't be any competition between different threads. It's should all be coded like a classic, tight game-loop.
AI upscaling and Android... dear lord. Just give me something which isn't fucking slow and frustrating to use. I thought I was in the thread which discussed non-smart TVs, did I miss something?