Hacker News Reader

Comment by wmf | Hacker News Reader

There was a rumor about 8,000 servers with only 1,600 simultaneous users; in other words an out-of-control burn rate.

That rumor was reportedly a rhetorical question asked by Perlman during the last company meeting. Anybody know why they didn't use a flexible virtual server solution like Amazon EC2?

"There's no way to exactly estimate how many servers we'd need. So we literally bought thousands of them, and all the equipment and networks to go with it," Perlman told employees. Those servers, he said, came with lengthy contracts -- contracts that tied OnLive's capital up in maintaining servers that few (if any) users were actually using. "If you've got 8,000 servers and 1,600 users, how could we ever get to cash flow positive, right?"

-- http://www.joystiq.com/2012/08/18/documenting-the-death-of-o...

There are EC2 instances with beefy consumer graphics cards?

(How do the GPU clusters even work? And, at $2 it seems way more expensive.)

Kinda... there are GPU instances, but they are targetted towards GPU cluster computing for HPC applications. I suspect that GPU clusters work largely the same way that generic/CPU clusters work. You have a cluster manager that dishes jobs out to the worker nodes. The common HPC job scheduler that we use (Torque/PBS) includes support for scheduling jobs that require GPU access. (I assume SGE does as well).

    Cluster GPU Quadruple Extra Large Instance

    22 GB of memory
    33.5 EC2 Compute Units (2 x Intel Xeon X5570, quad-core “Nehalem” architecture)
    2 x NVIDIA Tesla “Fermi” M2050 GPUs
    1690 GB of instance storage
    64-bit platform
    I/O Performance: Very High (10 Gigabit Ethernet)
    EBS-Optimized Available: No*
    API name: cg1.4xlarge

I suspect EC2 is not good for games, since the only GPU they offer is pretty high-end. Also, I think OnLive has more data centers than EC2 to reduce latency.

As the other guys here have mentioned, they basically need CDN-level latency virtually everywhere for OnLive to properly work and while Amazon does have a CDN (Cloud Front), their normal datacenters only have a few regions within the United States (which aren't close to any major metropolitan hubs (except maybe DC)).

The other big thing is that GPUs have very limited scheduling (i.e. no pre-emptive multitasking), so it's virtually impossible to virtualize into multiple VMs efficiently. So when you get a GPU machine you end up with isolated hardware which doesn't share the same economics as other Amazon solutions. As GPGPU become more popular, this should, however, improve.

You don't need preemptive multitasking for a known set of games, though; context switching can occur at frame boundaries (or even finer-grained than that, the GPU just has to go idle in some sense). It's necessary for GPGPU because you may have one kernel that runs for 10 minutes, but that's not the case with graphics apps (especially not a known set).

I also question the need for one VM per client. There's a very small set of inputs (input devices from the client) and a lot of static output (rendered frame). Why not share the same VM for a set of games that share the same GPU?

I have no inside knowledge into OnLive, but you're correct, you could certainly context switch for games. I would assume that the limiting factor would likely be texture bandwidth if you were switching between too many different scenes (on the same or multiple VMs). It might be that some of OnLive's IP is smartly managing resources in n->1 VM->GPU configurations (and maybe why games IIRC require some adaptation).

1. Seems a bit strange asking that question AFTER you spend the money. Surely you should ask it before you spend the money.

2. Worst case, at the very least you sell your excess capacity as cloud computing resources so your servers aren't sitting there doing absolutely nothing.

Reply on news.ycombinator.com