TV remotes are famously bad, like you said.
I think a good solution would be to use a Wii-like remote, that just has a handful of buttons on it. Put the common options (volume, on/off, last channel, menu) on the remote, and everything else as an on-screen menu, which you would point to with the Wii-mote.