Yup. OnLive tried it with similar results. There are many markets where 10ms - 20ms latency doesn't matter that much, but realtime gaming isn't one of them. It's extremely disorienting to play a game with even the slightest framebuffer lag.
My regular gaming setup is a 144 Hz monitor with a PC. Everytime I try gaming on a console with a 60 Hz TV I feel like I'm completely drunk because of the noticeable input lag. And that's not even cloud gaming. That's just the Console + TV input lag...
Plus your suggestion can be implemented with cloud compute (a backend behind the UI) - it doesn't need to be cloud streaming.