Cray CS-Storm Delivers High-Performance Computing In Million-Dollar Package | Hacker News Reader

Cray CS-Storm Delivers High-Performance Computing In Million-Dollar Package

techcrunch.com

The Cray brochure for this system has more information, including detailed specs: http://www.cray.com/Assets/PDF/products/cs/CrayCS-Storm.pdf

A few fun points from the brochure:

- The internal network ("interconnect") is Infiniband in either a torus or fat tree topology

- Up to 512 GB RAM per node

- Power usage is up to 63 kW per rack

- You have the option of including liquid-cooled rear door heat exchangers, if your datacenter air cooling isn't up to the task ;-)

- Compute nodes can have disks installed, but the system supports diskless booting

- Each compute node weighs 88 lbs

One thing to remember about these machines is that the internal high-bandwidth, low-latency network is a big part of why they get such good performance. Most applications that run on these machines are long-running computations consisting of many processes spread across the entire machine, with lots of inter-process communication throughout the lifetime of the job. Think weather prediction, simulations of fluid dynamics, or complex mechanical systems.

(I work on similar systems to these every day. Huge amounts of fun, but quite different needs than your typical web service, or even most Hadoop-ish data analysis problems.)

In what cases would this be better than AWS/other IaaS? Seems like a steep investment. Do labs use stuff like this?

Real scientific computing needs high-bandwidth low-latency networks.

Microsoft Azure actually offers InfiniBand for connectivity, which is interesting. I'm not sure how many people are using their HPC-oriented offering.

Go on then, let me know what "real scientific computing" is.

I work with bioinformaticians, which is an area with lots of research going on. I don't see much that is complex there. Split up your jobs and send them to a cluster. I take it that's not "real scientific computing" though?

From the article: planet-wide weather forecasting, seismic measuring for oil exploration, financial analysis or crash simulations. You can't split up a physical simulation.

You actually can split up a physical simulation. Imagine you're trying to model heat diffusion through a piece of metal. You split the chunk of metal up into many small pieces and model temperature in each one. Each node in your cluster is simulating the behavior of some portion of the model; at each timestep, nodes simulating chunks that border each other must swap the state of their "edges", sharing the outer edges of their own state with their neighbors.

Edit: unless you meant that you can't split up a physical simulation on loosely-coupled nodes and still get good performance. Then, well, yeah.

Well I guess the simulation would have to be broken up to run on multiple GPUs at once anyway. But even if you had a good interconnect, the GPUs in this box (K40) have almost 2x as many processors and 3x as much RAM. (Edit: compared to AWS GPU instances.) If you're doing a 3D sim, each node needs to communicate with 26 others, so just having 1/3 as many nodes is going to help a lot.

Split up your jobs how? Is the job embarrassingly parallel? Because if it isn't, and most things aren't, it's going to require significant programmer overhead to simply "split up your jobs". And then once you've split up the job, the different portions are going to need to exchange their in-progress results, because most real scientific computing programs involve iterating until either a certain number of timesteps have passed or a certain residual is reached.

I mean, I only write HPC programming frameworks for scientific computing as my job, what do I know. But if you're doing anything more complex than map-reduce, a high-speed interconnect is essential, and even in mapreduce it's pretty handy.

Approximating numbers from https://arstechnica.com/information-technology/2013/11/18-ho... a petaflop on AWS would cost $20k per day. So considering hardware alone, this would be cheaper in 50 days. Once you add in power and cooling for this device, and the "reserved instance" discount from Amazon, it would take longer to break even. But probably not more than a few months.

Edit: it answers this in the article.

On the opposite, I've never heard of a lab that does use cloud computing for HPC. Only for highly variable workloads with low networking and privacy requirements something like AWS would make sense.

Most labs could not afford this. But I could see universities buying lots of these for shared, general purpose calculations. I myself would not mind having access to a rack of them!

But does it come in a beautiful form factor with bench seats?

I'm a bit confused, is the described 1PFlop system really only one million dollar, including cooling and power? That would be amazingly good value. The single digit PFlop systems I know of have cost two to four orders of magnitude more than this, even when powered by GPUs.

The computers cost 10 billion dollars? Who buys them? That would be above the noise floor of even the NSA's budget.

That might be true, but to establish it, we would have to know the NSA'a budget (which is classified).

Sorry, I meant one to three orders of magnitude. The most expensive computer I know of is the Japanese K computer at ~ 1.25 billion and 10 something Petaflops. This was mostly politically motivated (Japanese chips) and too costly for what it does IMO. A typical new supercomputer in the Top 500 today is in the order of 100 million.

from the article "250 Teraflops per rack" and "a single NVIDIA Tesla K40 chip costs around $4,000. ... If you have four fully loaded racks, that’s over 700 chips. We’re talking some serious power for serious cash." so that's about 3m for 1PFlop

4000 is only for the GPU though, not the whole node.

I was reading this, thinking the numbers don't mean much (to me) without context. What does your average cluster cost and how many petaflops would you get for you buck there?

So going to get one as my new laptop. Might have to invest in a wheelbarrow though.

Reply on news.ycombinator.com