The third alternative is Valve using Linux, or something, which isn't enormously plausible.
1: http://bbot.org/blog/archives/2011/05/08/valve_isnt_going_to...
1) There have been several leaks indicating that Steam for Linux exists, or at least has been prototyped. I imagine that pretty much the only reason why it hasn't been released is because it's a small market with large support requirements.
2) Porting games to a Linux with decent video drivers is no harder than porting to the Mac. Once you've done one, you're 95% of the way to doing the other. The largest task is porting to OpenGL which you have to do to port to PS3 or to Mac anyways.
3) The largest problem game makers have with Linux is supporting a large variety of hardware along with a large variety of distributions. If you can say "We only support SteamLinux on SteamBox", that problem completely disappears.
Short answer: DirectX.
Scream all you want about OpenGL, but MS has honestly created a superior product. Between the hand-wringing over specs and the massive fragmentation via vendor-specific extensions, OpenGL doesn't hold a candle to DirectX. Even old OpenGL stalwarts like Carmack himself have bailed from that wagon.
There's a reason why Mac ports are still relatively rare in the industry, and more importantly, where they do exist, they are universally horrendous in quality and performance.
Carmack has acknowledged that DirectX is better, but Rage, for instance, still supports it for systems that don't support DirectX.
Which is to say, porting effort to PS3 does not also give you a PC/Mac OpenGL port.
This is the big problem I see - UE and idTech both support OpenGL, though in reality only UE has any significant licensee base to speak of. There is, however, an awful lot of home-brewing even at the AAA-levels: EA's gone and thrown a lot of weight behind the Frostbite engine, Ubisoft seems quite fond of their Anvil engine (Assassin's Creed + more), and the big massive CoD franchise runs a proprietary engine too.
An OpenGL-only platform might automagically include all UE and idTech licensed games, but that's hardly an impressive snapshot of the gaming industry, particularly the core gaming demographic (as compared to say, the mobile gaming demographic, where OpenGL already reigns supreme). This hypothetical platform won't just leave the vast majority of Steam's existing titles in the cold, but will also have the unenviable task of forcing many devs to provide cross-platform support where none currently exists.
Valve is a juggernaut in the industry - but even that's a very, very tall order. There's certainly a renaissance of interest in OpenGL thanks to the mobile gaming side of things, but I'm extremely skeptical of claims that any player, even someone the stature of Valve, can get a majority of existing Steam devs to sign up to support OpenGL.
Now, you are actually in fact thinking of LibGCM, which has very few similarities to OpenGL, is much lower level, simpler, and is in fact, the library that most PS3 developers do end up using.
Do however keep in mind, that any major game that runs on Mobile, OS X, or Linux, has pretty much API compatible library usage with OpenGL.
Then while the PS3 doesn't give you OpenGL, any cross platform game that runs on both PS3 and another system generally has a graphics system abstracted enough to make porting to an OpenGL system while not trivial, a rather well contained and limited amount of work. Surely enough to make it warranted to port to whatever graphics API Valve chooses to use, should it be a successful console.
And frankly the Microsoft monopoly on rendering API's. These days modern OpenGL is close enough, to not really matter. Having nearly feature parity, and these days all the fancy stuff happens in shaders anyways, the API just doesn't matter as much as the platform lock-in these days.
Yes, the steam console doesn't automatically get access to the entire library, but keep in mind, Steam on the mac has been hugely successful, and it had a very limited selection of games to begin with. You say that valve can't get a majority of existing devs to sign-up to support OpenGL, frankly they don't have to, rather they have to get just enough Steam devs to do such, and that's something history has shown that they do in fact have influence over.
It might require Valve to get people to recompile their games and tweak some stuff, but like you say, Valve are a juggernaut.
And frankly Microsoft could be licensing them some sort of Windows variant OS. Afaik, they maintain a positive relationship with Microsoft, and other than the conflict of interest with consoles, Microsoft knows that Valve is a very powerful player in the market. Also, do recall that Microsoft has licensed out their OS before, it as a different world at the time, but the Dreamcast was using a Microsoft derived kernel.
As for CPU usage, a properly tuned Linux kernel with unneeded services turned off shouldn't be significantly different from a hypervisor once it's booted.
Valve may try and set things up so that X11 is optional but even that should get mostly out of the way if you use the right libraries. (DRI etc.)
I'm not trying to suggest that they will use a hypervisor, but rather suggest that they have a ton more options then seem to be listed here.