Cray XK6 System

You can no longer make a supercomputer the old Cray way, by having Seymour Cray design the fastest, most optimized CPU he possibly can and sticking it in a box. Now you take thousands of processors (with an increasing trend toward low-power low-heat processors rather than just the most powerful ones) and glue them together--and the interconnect becomes one of the most important parts!

Huge companies like IBM can afford to make their own custom chips for their supercomputers (see Blue Gene L, P, and Q), but Cray isn't in that category. They do create a lot of software for using their systems, and the hardware is very well integrated, which is probably why they still do pretty good selling to national labs.

We're realizing that now while you can buy a bunch of computers and a 10 GbE switch, it can be worth it to get a nice fully-integrated supercomputer shipped to your door. It's ready to go, the software is installed, the cables are the right length (and neatly arranged!), and when a node burns out you just mail it back to Cray for a replacement.

That "interconnect" is the secret sauce of any modern supercomputer. Cray's proprietary link connects CPUs directly, almost like they were all on the same motherboard. It's easy enough to buy tons of cores. The hard part is reducing latency between them, and maintaining reliability if a link fails. Cray's networking package forms a "hypetorus" to reduce hops between all nodes, and will automatically route around damage. It's incredibly intelligent for such a fast system.

And I guess you didn't see the software environment either? It's really impressive. Check out the "Brochure" PDF for a lot more information.

Darn. You beat me to it.