Google can't take back the code it's already released other than the non-open source apps like Google Mail. But it can stop releasing new code. HTC et al are making too much money from Android, I doubt they'd stop investing in the platform, resulting in slow platform divergence a la Unix in the 80s and 90s.
3.2 hasn't been fully released to the open source community, but HTC and all the important players have it already, and I doubt that their license allows Google to yank it unilaterally.
Google does have to keep releasing code it based on GPL'd work, primarily the kernel, but that's a very small portion of the Android IP.
The first litmus test will be the release of Honeycomb.
Best case scenario? To me, that would involve Google moving towards a more open development model similar to what virtually all other successful open source projects employ.
They will be forced to make moves to placate HTC and all of their other partners, but whether those moves benefit us is the question.