Not sure what the downvotes are for, but ExaFLOP != ExaFLOP for a long time.

An A100 is an ExaFLOP GPU, if you count INT8 as FLOP.

Then you have the 10 different flavors of FP8, FP16, etc. and even the multiple flavors of FP32.

What people usually count as "ExaFLOP" is FP64, but this hardware can't even do FP32, much less FP64.

So yeah, an ExaFLOP != an ExaFLOP anymore, because these types of announcements never say "An ExaFLOP of what".

Everybody's brain has an infinite througput of FP0 (0 bit FP format). That doesn't mean that your brain can compute faster than Dojo, even though infinite ExaFLOPs >>>> 1 ExaFLOP. The reason is that we are counting different things.

So what's the throughput of this hardware in actual FP64 ExaFLOPs ? Zero, it doesn't support them. Not a very impressive marketing statement though. But what this means is that this hardware would be extremely bad at, e.g., solving linear systems of equations.

Locality has everything to do with performance. It's why your L1 cache is faster than your L2 cache.

Based on their presentation and design choices (and a tiny bit of experience) it looks to me like the limiting factor is actually bandwidth and latency not flops or electricity or heating or space or even the cost of the transistors themselves. A ton of flops doesn't help if you can't feed them data to process.

The reason they gave for doing things like "knocking out the walls of cabinets" was bandwidth, not space.

It turns out that the highest bandwidth solutions are also quite compact. One giant piece of silicon instead of many smaller pieces of silicon is great because the different parts of the giant piece of silicon can talk to eachother very quickly with very high bandwidth. Short wires are better than long wires, because you have less noise so you can fit in more signal (also less latency).

It's also just not clear to me how you imagine making things bigger would make them cheaper.

I'm not a hardware person at all, but my understanding is that distributed computing systems are hard to scale due to latency, and a huge factor in latency at this scale is physical distance.

This was also mentioned in the AI Day presentation, and listed as a major factor in the design of Dojo.