Wafer Scale Compute: Setting Records in Computational Fluid Dynamics

The corresponding paper is very interesting: https://arxiv.org/pdf/2010.03660.pdf

Particularly the SpMV CS-1 code listing on page 5, and this comment (in section VI.B) about accuracy:

> The measured normwise relative residuals with mixed and 32-bit are shown in Figure 9. Up to iteration 7 the mixed precision implementation tracks the 32-bit, but then fails to reduce the residual further. With this precision, machine precision is about 10^−3. We have observed this accuracy for very well conditioned systems. Here, the growth of rounding errors during the iterative solve explains the loss of an additional factor of 10, leading to a plateau at a relative residual of 10^−2. We have not yet tested whether this accuracy is acceptable in the MFIX code.

> We expect that for some realistic situations, mixed precision solvers are usable as is; in others they may need to be coupled with a correction scheme such as an iterative refinement or an outer iteration that solves a nonlinear system; and in other situations one may need to use higher precision arithmetic

[MFIX is the NETL CFD described in the article]

It feels a bit odd for the authors to declare the 200x speedup over fp64 runs on Joule, followed by "we don't know if these results at a lower precision are acceptable."

very cool stuff regardless. 18 GB of SRAM on the same (very large) chip is insane.

"The wafer scale CS-1 is the first ever system to demonstrate sufficient performance to simulate over a million fluid cells faster than real time. This means that when the CS-1 is used to simulate a power plant based on data about its present operating conditions, it can tell you what is going to happen in the future faster than the laws of physics produce that same result."

This is mind-blowing

> The vector multiply-add operations are the most memory bandwidth intensive of any part of BiCGSTAB. Each such operation reads over a billion values from memory and writes a half billion values to memory. This requires three memory accesses per scalar multiply-add operation. The CS-1 has full bandwidth to all of its memory: it can load two values and store one value for every multiply-add performed on every data-path lane, on every core, on every machine cycle.

wow

There's a lot of stuff to be suspicious of with this... A 7 point stencil is 1st order which is not likely to be very useful in the real world. If it was 1st order FVM it might be more useful because at least that is conservative.

I also don't buy that 50% of time is spent setting up the linear equations. They say that that is a conservative estimate; I would say that it's unrealistic - in the CFD code base I work on, it's much much less than that - about 30%, and that's when the linear solver is well parallelised and the equation set up is not.

This is incredible. The death of Moore's law is going to yield wild stuff in ASIC land.

Why did they choose not to support double precision floats? They are pretty needed for fluids/hpc work.

Cerebras processor has 1.2 trillion transistors and uses 20 kW of energy. Apples new M1 has 16 billion transistors (so is a factor of 75 smaller) and uses only 20 W (so a factor of 1000 smaller). So Apples M1 is 1000/75 = 13.3 times more energy efficient than Cerebras processor. So if your aim is processing power per Watt you should better build a supercomputer from a cluster of Apple M1 CPUs.

18 GB on-chip SRAM (at L1 latency), they say on the homepage.

Looks like RAM with local compute. Another startup is sort of doing this (Arm cores etched in the usual DRAM silicon process [1]). I wonder if this is a long term trend in computing. It does brings a whole new meaning to "getting the code to the data".

Also, the way to program it seems like it is very rough and constraining. I wonder how our current cushy languages and runtimes would (could?) adapt to that architecture. Would OpenMP even fit the job?

[1] https://www.upmem.com/

Potentially naive question: how close is this to being able to simulate a mouse brain?

Of course, assume we actually know how to simulate neurons in a meaningful way.

How can they not include a picture of the wafer.

I wonder how they're cooling this. That's 10 electric kettles worth of heat output :)

> Cerebras has built the world’s largest chip. It is 72 square inches (462 cm2) and the largest square that can be cut from a 300 mm wafer.

Normally, chips are measured in mm^2.

So that would be 462000000 mm^2 die area, if I did my math right.

(Light grey sans-serif body font means you hate your readers' eyes.)

So wafer scale integration it's back

Looks like an SGI onyx.

Imagine a Beowulf cluster of these