Accelerated Computing Powering World’s Fastest Supercomputer | Hacker News Reader

Accelerated Computing Powering World’s Fastest Supercomputer

blogs.nvidia.com

63 pointsPoalopat42 comments

"If every computation were represented by a single grain of sand, you could fill up the Houston Astrodome with sand 350 times in a single second."

There's a new one. Can anyone comment on the utilization of these megacomputers? Do they have somewhere near 100% usage, with a queue extending weeks? Also, is all this computational power really... necessary? I've seen some intensely inefficient simulation code in my time.

Yes and yes. I used Titan for my research in grad school. There’s dozens of researchers waiting for as much compute power as is available. Tasks like quantum Monte Carlo simulation make extremely accurate chemical predictions but require a massive amount of compute power to do so. You may think, well just design a better algorithm, but we’re starting to run up against the limits of computational complexity. Further fundamental improvement for simulating fermion systems accurately would require quantum computing.

Simulation of turbulence at very high Reynolds numbers is something that requires these mega computers. You can have a look at DOE's INCITE program awards to get an idea about these simulations:

  [0] http://www.doeleadershipcomputing.org/incite-awards/

Several years ago I had the opportunity to attend a non-classified project coordination conference / catchup meeting with some DoE scientists, among them attendees from LLNL and Los Alamos. I overheard discussion and some bragging about the number of cycles they had their simulation kernels at. At least some of them are dedicated to extracting every last bit of performance out of their code.

I wonder what portion of time on those supercomputers is used in optimizing the code itself; and how much could be done.

With those massive clusters you could afford testing in parallel trillions of mutations to your simulation kernels and prove the correctness of the fastest ones -- pruning should be extremely fast by finding counterexamples. Or even higher level architectural optimizations. Surely they could afford at least a few % of the total time on this pre-optimization (although the tools to achieve this automatically would need to be quite sophisticated!).

What do you do when these mutations end up crashing constantly?

http://fftw.org is optimized by doing massive parameter sweeps on each architecture (it only considers correct implementations).

There are also a few “software synthesis” and “sketching” approaches that use a constraint solver to find all correct implementations of a high level spec, subject to some implementation pattern. Then they either try them all with brute force or pick the one that optimizes some objective function.

fftw isn't exactly developing an optimal kernel from scratch. It's testing a range of different methods and simply choosing the best for one's parameters.

Facebook's Tensor Comprehensions framework, which generates CUDA kernels through a genetic algorithm, is closer to the sort of approach which would take greatest advantage of the hardware it's on.

Yeah, I was in a very HPC focused group run by ex-national lab people, and there were several PhDs worth of work going on at any time trying to speed things up. Compute, I/O...all of it.

Yes, the utilization for these supercomputers sits upwards of 99%. There's a lot of research into supercomputer queueing that helps maintain this high throughput. Larger jobs usually take days to weeks to run.

There are separate debug queues for smaller development and test jobs that prioritize latency over throughput, and this is a main source of inefficiency (but, as someone who develops on supercomputers, trust me, it's necessary to make them usable).

As for the efficiency of the codes--there are a lot of barnacle-encrusted Fortran codes in astrophysics, chemistry, and elsewhere. Some of them probably contain plenty of inefficiency, but the cost of re-writing them is high. Oftentimes either the people writing the codes aren't interested in maintaining good software quality (because they're focused on the science) or the incentives aren't aligned to re-write code that's already established, but from the 80s (re-writing old code is generally not how one gets a Ph.D.). But they do very important things, like simulate nuclear fusion or the interaction of chemicals in the atmosphere that create weather patterns and storms, so it's very important to run them.

99% is a bit higher than most systems expect to run at - any mix of job sizes in the queue will tend to leave small gaps when nodes need to stay idle until jobs "fit" perfectly again. 95% is a pretty common target for these larger systems.

There isn't necessarily a lot of research into packing these better - the basic algorithms have been unchanged for quite some time, and a lot more effort goes into deciding how to prioritize different groups that are sharing access into the same system.

Most machines will allow for back filling if your jobs are short enough and small enough. But that will still leave some of the machine under utilized.

I don't have any customer data on hand at the moment, but I'd roughly describe it as basic scheduling on strict priority order getting the system to 80+% usage, and then backfill boosting that up to 90%+. Careful backfill tuning, and care in defining the priority structure gets you to 95+%.

97% is the highest specific value I can recall for any of the larger sites with a heavily mixed workload, absent having a nearly infinite supply of short+small jobs at hand to use to fill those gaps. And a lot of users aren't trying to chase that - for these large scale "capability" systems the goal is to scale out as large as you can anyways, the smaller stuff is usually relegated to "capacity" systems elsewhere with a less expensive architecture.

One thing that at least some schedulers can manage is the idea of a min+max runtime for a job, combined with a min+max node/cpu count. If you have users willing to 'scavenge' otherwise wasted time by running under such a regime that can put you closer to full usage.

I use Edison (a DoE supercomputer), and getting through the queue there is kind of a nightmare.

I was on the team that made the Aries interconnect. I hope all is well once your job is placed. :)

If you're the one that came up with the idea of the "quiesce", I feel like you owe quite a few of us in the systems software community a beer or three...

The next revision of the TOP500 will come out this month, let’s wait to see how it stands against the competition. The previous supercomputer mentioned in the article, “Titan”, currently is the 5th fastest in the list https://www.top500.org/lists/2017/11/

Forgive me for being naive, but what OS runs on this beast? What framework do people code to to make use of the environment?

Probably something that looks like Red Hat. All these codes will use some form of message passing with MPI being the most common. Finally to get on node parallelism some codes will hand write cuda for their problem and others can get away with calling Nvidia libraries or using something like new versions of OpenMP.

I want to see the output of “top”

Unfortunately you would be disappointed. top is only going to display what's running on your current node, not the whole machine. There is probably some sort of global top, but just logging in and running top in your shell will only run top on the head node. That head node might still be beefy, but only as beefy as a large shared memory machine can be.

Why not just fork a child top for each node?

RHEL 7.4.

https://www.olcf.ornl.gov/for-users/system-user-guides/summi...

If you look at the statistics over time here for accelerators:

https://www.top500.org/statistics/overtime/

it appears the number of top500 systems with accelerators is not really increasing much - it has been sitting at around 20% of the top 500 machines for the past ~4 years.

Can anyone working with these systems comment on why that might be? Are accelerators still tough to apply to a lot of the problems these machines are used for?

Depends on what the computer is designed for. Imagine you want to parallelize up to about 16 nodes or 1000-2000 cores of a new xeon. And your code is not optimized for accelerators. Well you might build a large machine with hundreds of nodes, but with the intent to run many 16 node jobs at a time. The largest machines though mostly are aiming for jobs that use a significant fraction of the total machine and are chasing peak performance so accelerators it is. Also I didn't look too closely at the link, but some machines that aren't technically accelerators like K computer or the old IBM Bluegene computers are closer to general purpose accelerators than they are to fat xeon nodes.

As a point of interest, there has been some cool work done recently regarding accelerating high precision dense system solvers using tensor cores + iterative refinement. See http://on-demand.gputechconf.com/gtc/2018/presentation/s8478...

The only up-to-date information I've found so far regarding the storage system for this beast is:

* 250 PB GPFS

* 2.5 TB/s

I'd like to find some information on the burst buffer that is sitting between compute and the capacity storage system.

This supercomputer is surprisingly underwhelming, since it's equivalent to just two of Google's TPU v3 pods.

The numbers I've seen for a TPU v3 pod are ~100 PFLOPs, whereas this article claims over 3 EFLOPs, so that's at least 30 TPU v3 pods. Arguments about how useful FLOPs as a measure actually are aside, that's still quite a lot of computing power.

No, it's 200 PFLOPs:

> Built for the U.S. Department of Energy, this is a machine designed to tackle the grand challenges of our time. It will accelerate the work of the world’s best scientists in high-energy physics, materials discovery, healthcare and more, with the ability to crank out 200 petaflops of computing power to high-precision scientific simulations.

EDIT: OK, it's 200 PFLOPs of high precision math, 3 EFLOPs of lower precision math. I take my comment back.

You are off by a factor of 1k.

(off by a factor 15)

TPUv3 pods have a stated performance of 100 peta "flops," which is half precision operations on hardware designed specifically for machine learning.

The 200 petaflops number of Summit is referring to general purpose double precision computation. This is totally incomparable to anything the TPU can do at any performance.

I will certainly gets dozens of downvotes for obvious fanboyism, but I'm prepared to take the heat.

For the doubters and the disbelievers that have been wondering what is the relevance of IBM in this day an age: this. This is what IBM is all about.

And it's not just about PFLOPS; each node has 1/2 terabyte of memory, globally addressable across the entire cluster using RDMA over Mellannox 200Gb/s EDR.

It's also P9: 44 cores per node; but most importantly each node drives a couple of V100 through NVlinks, which allows the GPU to share the system's main memory.

NVlink allows gpus to share system's memory?

Think of it as a much higher bandwidth PCIE. You still have to think about data locality, the same way you do with cpu cache levels. It just makes the problem of not having data on the gpu less painful than before.

Summit node CPUs can access GPU memory coherently and unified memory allows for a single pointer across all processors. On most systems that involves page faults migrating pages, but summit has something called ATS that allows GPU to directly access all system memory.

https://vimeo.com/262870773/recommended

IBM integrated NVlink into their Power CPUs, well I'll be damned.

FYI - It is generally frowned upon to make a comment on HN along the lines of “I will get downvoted, but...”. It is completely unnecessary and you could just make your point without it.

Also, you can defend yourself later when people respond. Prematurely saying “I am ready to take the heat” is not how we communicate here.

But can it run Crysis?

Please don't post if you don't have anything to add, or a valid question to ask. If you feel like posting memes Reddit is a better medium for that.

NVIDIA is my dream company. Does anyone have an in?

Reply on news.ycombinator.com