An inside look at the custom CPUs in Tesla's Dojo Supercomputer

You have never built a super computer, have you?

You have to connect thousands of cables, have hundreds of nodes, with thousands of components, everything interconnected, and if you connect one wrong, the computer outputs incorrect results. You have to routinely update the software, and if a software upgrade introduces a 20% perf regression (which happens), then your 10 MWh cluster starts burning 2MWh for nothing. Or maybe your cooling system sucks, and after a minute of running at full capacity, you need to throttle your cluster to 0.1% of the peak to keep it cool enough that it runs "something".

That's why all systems in the top500 i've been involved with (15 or so) run these benchmarks as an integration tests on every single cluster maintenance (node updates, servicing, OS updates, etc.).

Submitting these results to the Top500 costs you nothing... if your cluster actually works. When you submit to the Top500, they ask for access so that they can re-run them themselves, which happens typically during / after the next maintenance to avoid impacting any users.

If they haven't submitted, 100% sure their cluster does not deliver what they say it should deliver on paper. Maybe it delivers 1% of it, or 0.01% of it (seen both cases in real life). If they haven't fixed it, then maybe it can't be fixed.

HPL, MLPerf, Spec, Stream, OSU.... these are not "benchmarks for bragging rights", these are tests that show that your system works.

Running benchmarks is not only useful but necessary. It's a point where planning meets the real world. Private clusters are also routinely benchmarked, often as part of validation.

In the end if someone just tells you their supercomputer "would" be in the TOP500, they just tell you that it is definitely not in the TOP500. There might be good reasons for that, but still, it's like claiming that you would win Olympic gold if only you'd bother to participate.

I don't know, given Elon's showmanship, is it really that different?

> Not clear what you're disputing here?

The article, that assumes "FLOPS on paper == FLOPS in practice".

I only work on a lowly private cluster but running standard benchmarks is utterly routine here (it is in fact automated). As others with HPC experience pointed out, running benchmarks is pretty much mandatory when bringing a new system up, not just to ensure it actually performs as promised, but also to weed out bad and marginal components. You do one or two weeks of intense benchmarking and testing and you're assured numerous nodes will fail and need parts replaced. When you apply patches, you benchmark again. Why? Because the suppliers fix for "code XYZ crashes nodes" is probably "let's uh just reduce power limits by a few %". Or because of unintentional issues limiting performance. When a node crashes, you benchmark it again. Why? Because linpack and friends are good at making marginal hardware fail.

So it is literally unbelievable that Tesla not just stands up a cluster, but created their own hardware to do so, and didn't run any quotable benchmark and only has the theoretic FLOPS numbers for marketing.

This reminds of Magic Leap. 3 billion dollars invested in a 3D render.

Sure, your heuristic may be suspicious of 3D renders, but has it considered men in robot costumes?

That was great, thanks for linking it.

In their presentation yesterday they made it clear that they have a full software stack for programming it. I don't know what is marketing hype and what is real, but they suggested it is very easy for them to program for it.

yes im aware. that thread is a day old and probably quite dead. i post here because i am interested to read peoples responses. and its certainly related since tesla bot will use fsd and be trained on the hardware discussed in the article.

to be honest FSD seems to be coming along albeit behind schedule. ive watched videos of the fsd beta and it is the closest thing ive ever seen to a car driving itself in an uncontrolled environment. i wonder what you make of dirty teslas latest driving video on youtube?

i think this comment illuminates the issue. seems like an industry veteran (unconfirmed) thinks having the thing walking around is plausible but is confused about interaction.

https://old.reddit.com/r/robotics/comments/p7t14o/tesla_reve...

> Either onboard compute is minimal, or it has a battery life measured in dozens of minutes.

For industrial applications you could probably have some sort of novel power system, like a tether from the ceiling or special floor that delivers power through the feet.

I don't know about cesspits & idiots, but a little smell can develop that does tend to bring the intellectual level toward the below-average rather than above.

For those that want to push downward, the present system highly leverages their ability to do so simply through the process or habit of frequent downvoting.

If there is any perception that there was a time when there was very little detectable smell by comparison, it would be good to assess the average downvote-per-user rate then, and compare it to today's figure.

Then measure each user along this scale, perhaps including a time component, or relative to activity in some way.

Allowing for a reasonable standard deviation, it might be better if frequent downvoters past a certain range had the weight of each downvote normalized and see what happens.

This could possibly also be tuned to achieve a target level of discourse relative to a previously-considered-desirable data point in time.

Alternatively, users alone appear theoretically able to overcome the issue if there was a widespread concerted or random effort to frequently upvote the comments or postings seen descending, whether fully deserved or not, keeping them at least neutral without having a negative effect on the commenter's rating.

Mathematically a small uptick in "compensatory upvoting" habits among average users could bring the target way up as long as the overly-frequent downvoters are in the vast minority.

Then when there is true downward consensus it will still always drop through, but those who participate mainly to downvote will have less negative impact.

The only thing worse than the "nattering nabobs of negativity" are the non-nattering nabobs of even worse negativity.

Haha, don't know where the downvotes are coming from, this is the absolute truth right here.

He knows the world idolizes him as a slightly eccentric genius engineer with a heart of gold, and I think there's probably some truth in that, even. Nevertheless, if you are smart enough to be a good engineer, it doesn't take much to realize that you can greatly expand your aura by occasionally trolling people with something stupid just to see if you can get away with it. The fans will adore him more, the critics will pounce, but the net result is more exposure, and more exposure gets investment money. Rinse, repeat.

Personally, I think one of the smarter touches is to release/leak information that looks bad shortly before releasing information saying that the problem was overcome. This constantly reinforces the impression that anything Musk-related is always overcoming impossible odds. "Andrej didn't think we could do it, what do you think now, Andrej?" "The Starlink terminal costs $2400, but just a few months later, it costs $1000!"

I'm sure FSD will be here by Christmas.

Even with reversible computing?

> Transistor density isn’t doubling.

It pretty much continues to double: https://en.m.wikipedia.org/wiki/Transistor_count#/media/File...

Both TSMC and Intel have transistor density increases on their process roadmaps, although they definitely aren't doubling.

Moore’s law was about count, not density.

This isn’t far from the truth. But embrace it rather than feel upset by it. The world of programmers can be unlocked by a good API. And the underlying components can be C{,++}, as V8 is.

Yeah but it’s happening in a computer! Definitely feels like a The Matrix reference. Plus the whole thing is doing tons of matrix-based math.

Also, the reason most laptops run hot is that most modern high performance processors are thermally limited. It means the cooling is never "sufficient" because if you make the cooling better then you get a faster processor instead of a cooler laptop.