Hacker News Reader

Comment by ChuckMcM | Hacker News Reader

This is an interesting result, customer pays slightly less than a million/month (1279 x 24 x 30 = 920880) to run such a cluster. Using a 'Westmere' class processor (2 procs/mobo 6 cores per proc) that is 2500 machines, at 400W per its a MW of electricity (call it 111K$/month with a .15/kWH cost(includes cooling)). It would be interesting to price out the other costs for the machines to understand what sort of revenue that would be.

I wondered about spreading it out around the country through, seems like you would incur a lot of latency which might become a bottleneck.

View on HN

It's pretty well understood in the industry that EC2 is a cash cow for Amazon. I wouldn't be surprised if their operating costs were 10-30% of the hourly rates they charge (based on back-of-the-napkin calculations like you did, but also based on their competitors' rates which are much lower).

However, the cash flow generated by EC2 is likely negative as I believe that (1) all the profits are re-invested in expanding, and that (2) Amazon injects money into it from other sources.

The advantage of an AWS cluster is that you can shut it down when you don't need it, whereas your own cluster needs to be running all the time to justify its comparative cost.

Aye, I agree with that whole heartedly. Its the whole timeshare market I wonder about. Basically if there is a market for this 'size' cluster for about $1,300 an hour could you build one of these and rent it out like Amazon does but more efficiently. (Sort of can you cut costs by specializing into a particularly lucrative segment of the market).

So if 100% 'occupancy' on your cluster is worth a million a month, and your cost of 'owning' a cluster of this scale is a quarter million a month, then you need to do better than 25% occupancy to break even, and anything better than that and you make money. It is an interesting financial exercise if nothing else.

The company I work for, R Systems (http://www.rsystemsinc.com/), basically does this. We own and admin several medium-sized compute clusters (200-500 node clusters) which are rented out to customers who need a lot of compute on a temporary basis.

A lot of what we do is heavily custom, and we provide a lot of support for our customers, but we still typically beat Amazon on both price and performance. EC2 might work nicely for embarrassingly parallel workloads, but they don't have Infiniband available if you're latency-bound... :)

Definitely, and agreeing with ajdecon.

There are a lot more to supercomputing than just having lots of machines in a cluster. Special network connection, network topology, routing techniques, specialized CPU/GPU, specialized storage, all these can make a difference.

I'm sure you can build a special "Amazon for Supercomputing" infrastructure, rent it out to clients, and beat AWS on price/performance. Just add fast network, mix of CPU/GPU, SSD, large RAM and distributed RAM disk. Have some standard network topologies for easy configuration. Have some standard cluster layouts for different computing needs. May be having the software in place for the typical supercomputing needs. The clients just need to provide the data to a cluster and the answer will be spitted out.

The total costs of operation would be much greater than HW Investment + HW Operation (Power, Space, Cooling). Especially initial setup costs and maintenance costs are a big dent; other humongous cost drivers are high availability, etc.

The cluster, announced publicly this week, was created for an unnamed “Top 5 Pharma” customer, and ran for about seven hours at the end of July at a peak cost of $1,279 per hour, including the fees to Amazon and Cycle Computing

Given that the article states that the entire system only ran for about 7 hours, I assume that it was one of those ideal use cases for cloud computing. So the benefit of having a disposable system adds value that is also missing in the napkin calculation. Sure, if you ran the 30k-core cluster for eternity, you might as well build your own data center. But for this case, the comparative cost analysis seems a bit pointless.

One more small nitpick: the $1279 you extrapolated from in your calculation was the peak cost, not the average cost.

Reply on news.ycombinator.com