I think it's partially about the hardware. iPhone OS is a lightweight, stable embedded OS that runs well on cheap, low-power hardware. The old Apple TV is expensive in part because it's a low-spec Mac desktop with a custom UI (read: expensive Intel + Nvidia hardware). An iPad logic board without a screen or battery is probably pretty darn cheap, and more than powerful enough for video and music.

Touch is just part of the UI layer. A D-pad for navigation and two buttons (Go + Cancel) are enough for a set-top box. I don't buy any of the "touch remote" theories, and I don't think games are going to be a part of the picture. In the casual games world, iTV doesn't have the same advantages over Wii that iPhone has over the Nintendo DS.