Hacker News Reader

Comment by ACCount37 | Hacker News Reader

Workloads did change over time. Back when we were first approaching practical exascale, the dominant workload for a supercomputer was thought to be physics simulations - and they often benefit from high numerical precision.

Now, the dominant compute-hungry workload is AI, where precision takes second place to the independent parameter count. To the point that the capacity of BF16, which were originally designed as a radical optimization for AI workloads, is sometimes considered wasteful now.

AI workloads have some truly peculiar and counterintuitive properties - the kind of things you might expect to see in biology instead of conventional computing. Intrinsic error tolerance, for one. It did necessitate some rethinking and reprioritization, and I'm not quite sure if we converged to the general shape of an "optimal" AI accelerator as of yet.

View on HN

> the dominant workload for a supercomputer was thought to be physics simulations - and they often benefit from high numerical precision.

It still is. Just like how "mainframe" used to be a very general word, and over time gained a very unintuitive definition referring to a very specific type of computer with a specific purpose, "supercomputer" almost invariably means it's a highly bespoke cluster dealing with FP64 workloads. I don't see anyone referring to these AI clusters as supercomputers, for the same reason they aren't referring to the racks as mainframes.

I wonder if there's a term for this kind of semantic narrowing?

I think the term in this case is marketing.

NVIDIA used to use the term supercomputer more often five or six years ago. It's now fully committed to the term AI factory. I think this is supposed to make you think of their expensive kit as a productive asset building your business rather than an expensive tool for your boffins. Your average business executive is instinctively going to question whether they need a supercomputer but don't yet have the same preconceptions about AI factories.

I don't think that Nvidia single handedly caused the move away from the term supercomputer. But I think they had a hand helping push firmly in that direction.

It's definitely older than that. Originally, way back in the mechanical days when super-computing was hyphenated, it was about how many millions of instructions per second could be handled. That changed in the 1960s with a series of CDC mainframes that put all of their focus on floating point pipelines, moving the category to refer to floating-point crunchers used for scientific simulation (which is what the frontier of computing was used for). Eventually parallelism would be added in at some point, and the term has fossilized into a "monolithic cluster with massive FLOPS intended specifically for scientific simulation". Because for the longest time, that's who needed all the compute.

The PIM/m of the FGCS defied this a little, sometimes being called a "symbolic supercomputer", but never without the "symbolic" qualifier. It was the first time in history where the frontier of computational throughput wasn't about scientific simulations. It wasn't even about arithmetic!

Right now is also interesting, because the world's most powerful FLOPS monster monoliths aren't being used for scientific simulation. They fail at being supercomputers for that reason, but a computer scientist from the 60s would probably raise his eyebrow at that.

FP64 is not that precise; proper simulations usually need much higher precision. Even ancient Intel could do 80-bit FP.

This very much depends on the simulation. CFD historically used 64-bit, with extended precision used internal to an extremely small number of poorly conditioned operations, or with very stiff equations; today it's not at all unusual to use single precision or (strategically) mixed precision, depending on the problem. I think the strongest statement you could make about simulation workloads, rather than them "needing" a specific precision, is that they are usually properly analyzed to determine the needed precision, and then run at that precision.

One could argue that the interesting parts of simulation require higher precision. When you are in stable conditions, that's where your intuition might be sufficient already; once you hit those ugly parts then you start requiring as high precision as you can get to have any correlation with reality. Even a simple solar system simulation through gravity interactions has significant drifts in FP64.

Datatype requirements vary in time and space for a given application. We're just beginning to optimize for this, so there is yet much to discover.

ancient intel did 80 bit fp until your intermediate result was written from registers to memory. Since you typically have no control over when that happens, even in very low level languages like C, the 80 bit precision mostly had the effect of making your code produce different results on different compilers. If you needed 80 bit precision and deterministic results you had to write the algorithm in assembly.

Reply on news.ycombinator.com