Cerebras Wafer Scale Engine Gen2 7nm 2.6T Transistors | Hacker News Reader

Cerebras Wafer Scale Engine Gen2 7nm 2.6T Transistors

servethehome.com

66 pointsramshanker54 comments

How do you connect such an enormous chip to a circuit board? What packaging and wire bonding technologies are available? I don't think it uses a conventional package like a (Flip-Chip) BGA - the process of soldering a 65535-pin BGA, and instantly losing millions of dollars if there are defective joints is simply unimaginable. The huge chip also have a heat dissipation problem that conventional packaging is unable to solve.

A lot of system-related questions were answered at HotChips last year: https://www.hotchips.org/hc31/HC31_1.13_Cerebras.SeanLie.v02...

And SC19: https://secureservercdn.net/198.12.145.239/a7b.fcb.myftpuplo...

Thanks for the links!

So everything is custom-developed, including wire bonding, packaging, connectors, heat spreaders, an impressive achievement. Unfortunately it's only a three-page overview, I'd love to read more on technical details.

It's mind-boggling to consider what it takes to power and cool a 15 kilowatt 'chip'. That is a huge amount of electricity.

For comparison's sake, 15kw is enough power to heat a house in a moderate winter climate. Getting a furnance worth of power into and out of a chip without melting it is, indeed, very impressive.

15 kW is 7x more than anything that can be (legally) powered from a standard European 230 V, 10 A outlet. With a 2.3 kW wall outlet, you can already power a Bitcoin miner unit or a small IBM mainframe computer. It's indeed impressive.

Which brings us to the next topic: What does the power supply look like, and how does power delivery work? Given the huge amount of power, the PSU for the chip alone is an interesting piece of electronics. Assuming its power supply is similar to a PC or server... First, the mains power is converted to a system DC voltage such as 12 V. For a typical modern cheap and boring switched-mode power supply, the standard efficiency requirement is 80%. But in this case, it means the PSU itself will waste 3 kW of energy, this is unacceptable both from an environmental and thermal perspective. A really good supply can achieve an efficiency of 95%-98%, and this is when things get interesting, at least you'll see some expensive controller chips and high-power, high-frequency transistors in such a PSU, terms like SiC, GaN or IGBT come to mind, switching 1,250 amps on and off. And it doesn't end here, the next step is converting the 12 V system voltage to the chip's core voltage, say 1.0 V, by a Voltage Regulator Module near or on the motherboard. Now, don't even mention the technical challenge of designing a VRM. I can't even imagine how they run the huge current on the motherboard, it's 15,000 amps!

Unfortunately, there's no information about power supply and power delivery.

There are probably several power domains in the chip and therefore several separate power supplies, which avoids the need to manage 15kW in a single spot.

Edit: In the slides shared above [1] (page 16), it says "12x 4kW hot-swappable universal PSUs".

[1] : https://secureservercdn.net/198.12.145.239/a7b.fcb.myftpuplo...

Not sure why this was downvoted - a relatively small natural gas forced air heater as is common nearly everywhere I've lived in the US is about 50,000 BTU or very nearly 15kW. Its a lot of heat.

Definitely a challenge, heat dissipation is "easier" if you can glue it to a plumbed 1/2" thick copper heat exchanger.

I had an opportunity see a laser sintered copper heat exchanger that was very very efficient. Pumping in water and pulling out steam efficient. It was pretty cool.

Did anyone put their first gen to use? Saw that huge chip last year (or maybe the year before at this point... time doesn't exist anymore...), but I never heard of actual users or buyers.

Based on press, there are already some systems being deployed (Argonne National Labs, Pittsburgh Supercomputing Center).

I'm sure companies that do a lot with AI - the FAANGs, universities - would have their hands on these to see if they're viable for their businesses, either for their own use or to rent out to customers through their cloud offerings. But they don't really advertise with it, probably corporate secrecy.

I think it has the same customer base as the quantum processors of today.

Is there some recent cloud offering that might be using cerebras systems ?

The most interesting part of this to me is the staggering amount of SRAM on the processor.

I wouldnt have thought this amount of sram to be practical from a size and cost standpoint compared to DRAM.

You cannot make DRAM on the same process with competitive, high-end logic, and they want to embed a lot of buffers on chip to reduce communication power use, so SRAM is the only game in town.

Intel had eDRAM (Crystal Well) for a while as the cache for their integrated Iris graphics.

From wikichip: The eDRAM silicon die is separate from the main Haswell die but is packaged together with it.

Huh, I should have done more reading.

Crystal Well was only eDRAM in the marketing sense.

There have been other, actual eDRAM technologies that can be manufactured on some logic processes, however none of them are compatible with the state of the art TSMC 7nm logic process.

Yeah, but it's not enough. The whole chip doesn't have enough SRAM to hold a tenth of GPT-3, let alone GPT-4. These are the kinds of models you would hope a chip like this would help with, but as soon as the model gets too big to fit in the SRAM then the advantage goes way down. And I think the models of the future will continue to get much, much bigger.

There are a few companies doing analog compute-in-memory for ML. Usually on flash processes.

Maybe cerebras could fit well for that.

As for the models of the future getting much bigger - is the GPT family representative ? Are we seeing the same growth is in other domains ? And isn't there a big niche for smaller than GPT models ?

I can't help imagining INTEL / AMD / NVIDIA doing the same with their own chips.

Top500 Supercomputer in a single Rack. Just imagine.

And if we take liberty to dream extreme, "Available on Amazon, On Demand". :D

Someone is going to figure out that if they cut the wafers on a diagonal they can make even bigger wafers.

And someone will be spending a lot of time trying to figure out how to support a wafer cut lengthwise without shattering it during processing.

That wouldn't work. Modern chips need to be built on epitaxial surfaces, and the only plane they can reliably produce an epitaxial surface on is the horizontal one.

Watched the talk at Hot Chips today[0]- 400,000(!) cores on a single wafer is absolutely remarkable! There was a slide not posted in which there were virtual segments of the wafer designated for each step, and the example given by the presenter showed a perfectly in-order workflow execution.

[0] https://www.anandtech.com/show/14758/hot-chips-31-live-blogs...

Their 2020 announcement is 850K cores, which is even more impressive!

So, just wondering from a practical perspective, how do you train/serve models from these? I doubt they play nice with e.g. PyTorch or TF.

Maybe they're meant to be used with a DL compiler like TVM [0]?

[0] https://tvm.apache.org/

You can use standard ML frameworks :) The code then goes through the Cerebras Graph Compiler to produce an executable that runs on the wafer.

More of the software stack was described at HotChips today, covered by AnandTech: https://www.anandtech.com/show/16006/hot-chips-2020-live-blo...

Still wondering how they’ve handled interconnecting beyond each reticle limit/exposure. Are they staggering the exposures for each layer? And how do they handle packaging/IO/power delivery?

I guess these are the secret sauces and we are unlikely to get in depth information.

In the PDF[0] that aveni linked above, they mention they "add wires across scribe line in partnership with TSMC, Extend 2D mesh across die". So the wafer isn't exactly one big chip.

[0] https://www.hotchips.org/hc31/HC31_1.13_Cerebras.SeanLie.v02...

I think reticle stitching has been done previously in image sensors and silicon interposers so it may be an evolution of that.

Ironic this article was posted the same day as : Why Don't They Make BIGGER CPUs? - (Techquickie) https://www.youtube.com/watch?v=8JAWz9Da5og

Now I know its just a LinusTechTips level video, but I guess they havent heard of "Wafer Scale Engine", and I hadn't either, but now this proves the video is already obsolete…

Not obsolete yet, the technology is not mature enough, and the previous arguments against WSI are still valid.

People have been trying Wafer-Scale Integration [0] since the 1970s, there was quite some hype of building a "super chip" back at that time [1], but all efforts failed miserably. Cerebras' success is only the beginning, even if this approach is workable (which remains a question), there's still at least a decade to go from a HPC-specific chip to a general-purpose chip. Another possibility is that WSI will forever be a technology used in massively-parallel computers.

[0] https://en.wikipedia.org/wiki/Wafer-scale_integration

[1] Giant microcircuits for superfast computers, Popular Science, 1984. https://books.google.com/books?id=eAAAAAAAMBAJ&pg=PA66

Another possibility is that the company will just fail, because it can not offer an advantage over commodity chips.

I already mentioned it,

> even if this approach is workable (which remains a question)

No need to repeat.

I think it is important to separate the several technical failures in wafer scale from the commercial one: Sinclair's Anamartic worked exactly as planned but the prices for conventional hard disks began to drop and that caused investors to pull out.

http://www.computinghistory.org.uk/det/3043/Anamartic-Wafer-...

The problem is that the real project was a massively parallel computer just like the Cerebras (scaled to 1989 technological limits) and the disk replacement was just a way to develop the needed techniques and finance further developments. If the investors had had a little more patience then computing in the 1990s might have been a bit more interesting.

Not at all obsolete, defect rates rise exponentially as you increase die sizes, so wafer scale chips are stupid and wrong, but in financial sense only.

If you’re paying up for an entire batch and you only need one working example out of it, it probably matter less.

They disable bad parts of the die, like bad sectors on a disk.

This only works for some things, if you lose a clock or some part of the chip is just spewing garbage into whatever bus it’s using, you’re hosed.

They fuse it off

At that point, is there really a benefit of having it all on one wafer, as opposed to just using chiplets?

Presumably there is, but wonder the same thing.

With wafer scale the amount of connections that can cross a border between two tiles is limited by the spacing rules for the top metal layers and by the number of layers you can dedicate to this (1 or 2, I would guess).

For TSMC N7 Design Rules[0] the pitch is 720nm for the top two metals (1388 wires per mm) and 76nm for the middle metal layers (13157 wires per mm).

In the case of chiplets on an interposer, if we suppose that the interposer can use a process similar to the top layers of the wafer then the number of wires between chiplets is the same. But you need pads and solder balls from the chipets to the interposer (and back on the other side). These pads can be in a grid pattern (like in a BGA package) so the area is the limit instead of perimeter. Let's be optimistic and suppose a pitch of 30µm for the copper pillar micro-bumps.

With a conservative 1000 wires per mm of perimeter the pads are not the limit with a chiplet larger than 3.6mm per side. With a more aggressive 10000 wires per mm the chiplets would have to be larger than 36mm per side, which is just beyond the reticle size limit of many chip fabs.

[0] https://en.wikichip.org/wiki/7_nm_lithography_process

Might it not be more optimal to attach the individual GPU dielets onto a silicon interposer to avoid the yield issues that are inherent when making a large monolithic chip, and save some of the overheads of dealing with dead dielets? I think I read about some researchers/companies going that route but haven't kept up.

Part of the genius of the Cerebras on-chip routing is that it can route around malfunctioning dielets.

Even then, wouldn't it be better to just use commodity chips on an interposer?

I don't actually know, but that seems to be the route the industry is going and economics of scale will likely make it the winning approach.

It must have some redundancy in order to survive process errors. Does it have to be defect-free to yield?

Answered in the PDF [0] that HN user "aveni" linked below,

https://www.hotchips.org/hc31/HC31_1.13_Cerebras.SeanLie.v02...

> Redundancy is Your Friend

> Uniform small core architecture enables redundancy to address yield at very low cost

> Design includes redundant cores and redundant fabric links

> Redundant cores replace defective cores

> Extra links reconnect fabric to restore logical 2D mesh

So does this beat 8 V100s in a DGX? And if yes what TPU v3-n is it comparable to?

by transistor count, by energy consumption and i'd expect by the price it is an equivalent of 3 x 16 V100 DGX/Lambdasystems. Can it beat such a minicluster? Vertical vs. horizontal scaling seems to be still an open question for DL. It looks like the horizontal - ie. V100 approach in this case - would naturally have advantage over the vertical when it comes to sheer scale, i.e. like the maximum possible size of model.

Interested in how Google TPU new generation compares to this one. Is this comparison even reasonable?

How much does it cost or rather how much does that entire rack with all the supporting systems cost?

What does this mean for Moore's law being dead or not? Anything?

Nah, were near the scaling limit of silicon. And huge chips are super expensive and impractical.

One day we'll find an easier way to print chips. Won't end moore's law but may shift algorithm implementation back to hardware after decades of software eating everything.

Medium term, they'll get a new lease on moore's law by switching off silicon to smaller atoms. And/or switching to something that handles heat better so we can go 3D. This is already happening in power electronics with GaN. My bet is on this tech slowly spreading into digital logic

Reply on news.ycombinator.com