I think that's the direction, too. But they need to be "docked" wirelessly. Doing it with wires, or by actually docking them in another device is way too cumbersome and people won't do it. If they do it wirelessly (maybe with Wi-Fi Display technology?), and make it connect with everything, it will probably get a lot more used this way.
This is why I've been hoping that Google would just make Android have 3 UI add-ons, that get activated based on what type of device they are in. If it's the phone, then it shows the phone UI, if it's a larger screen, like a PC screen, the tablet version gets activated for it, and if it's a TV, the "Google TV" version gets activated there. Of course, they should try to make them as similar as possible, but you shouldn't underestimate that they are ultimately different form factors and they need their own optimized UI.
Also once that UI is activated, it should probably be activated on the phone as well. For example, I think it would be best if you could control the TV UI directly from your phone by touching it and interacting directly with it rather than "tabbing" through the TV UI with a remote, which is slower, and reminds me of how a dumbphone's UI was controlled before touchscreens. Pressing the UI element directly is a much faster approach.