Tesla turns on 10k-node Nvidia H100 Cluster
techradar.com
techradar.com
It links to a Tom's Hardware article (https://www.tomshardware.com/news/teslas-dollar300-million-a...) from August 28 that says "Tesla is about to flip the switch on its new AI cluster, featuring 10,000 Nvidia H100 compute GPUs") and says "Tesla is set to launch its highly-anticipated supercomputer on Monday..." (presumably the September 1 event).
So, like, does Tesla actually have 10k H100s? Or do they have an order for 10k H100s? Or an intention to buy 10k H100s?
Is the sole source for these articles this (https://twitter.com/SawyerMerritt/status/1696011140508045660) random Twitter post by some guy who runs an online clothing company?
I don't mean to snipe, but this article doesn't seem to rise to the extremely high editorial standards of such tech-press luminaries as "TechRadar" and "Hacker News".
If you would’ve just scrolled just a little bit on that Twitter post that you linked. You would’ve seen these:
https://x.com/sawyermerritt/status/1696012091964915744
https://x.com/tim_zaman/status/1695488119729238147
Also, just FYI. Sawyer posts most of the Tesla and SpaceX breaking news on Twitter before major outlets even write their articles.
For example, here’s one just 12mins ago as confirmed by Elon: https://x.com/sawyermerritt/status/1728092021628313777
A “random Twitter post by some guy who runs an online clothing company” is definitely a wrong assumption.
I don't see those when I scroll. I see
"Buckle up everyone, the acceleration of progress is about to get nutty!"
and this is the end of the post?
Maybe I'm misusing this thing?
> https://x.com/tim_zaman/status/1695488119729238147
So another guy who claims to be a Tesla employee says (again, strangely future tense) that this is true? I mean, I am willing to believe--'cause he paid $20 for a blue check--that he probably is a Tesla employee.
But the use of future tense is a bit weird, right? And the lack of any followup?
> A “random Twitter post by some guy who runs an online clothing company” is definitely a wrong assumption.
I guess I'm old. Back in my day, "evidence" wasn't some random dude's online posts. But I know things have changed. ;)
==
More seriously:
https://www.hpcwire.com/2023/08/17/nvidia-h100-are-550000-gp... says Nvidia is producing 550k H100s in 2023. And there's obviously a significant lead-time requirement.
So, yes, I can sorta imagine Tesla pre-ordered 2% of global supply of H100s early in 2023 and was bragging about it at the end of August just 'cause.
But I can also imagine this is smoke and mirrors, and they have, like, a handful with the rest on backorder, and we haven't heard more about it 'cause Tesla doesn't have marketing people, it just has wahoos who post things on Twitter.
Either way, I guess?
That seems to be the case here. ;)
> So another guy who claims to be a Tesla employee says (again, strangely future tense) that this is true? I mean, I am willing to believe--'cause he paid $20 for a blue check--that he probably is a Tesla employee.
Another case of misuse? Here’s a tip for you. When you see a company logo/icon on someone's Twitter/X profile. That means they are verified to be affiliated with that org.
“Accounts affiliated with the organization will receive an affiliate badge on their profile with the organization’s logo, and will be featured on the organization’s Twitter profile, indicating their affiliation. “
https://twitter.com/verified/status/1641596848921276417
Instead of inferring that Tim Zaman is a random Twitter user who paid $20 for a blue check. Why not just Google his name? ;)
https://letmegooglethat.com/?q=Tim+Zaman
> I guess I'm old. Back in my day, "evidence" wasn't some random dude's online posts. But I know things have changed. ;)
I linked a video where CNBC was interviewing Sawyer but it seems that you didn’t even bother to check it.
This seems to be the problem today. People refuse to do the bare minimum (which is not even much) required for critical thinking. Instead of verifying information, people tend to uncritically repeat inaccurate assumptions, even when provided with additional information in good faith.
Like, whats the actual news story here?
Totally agree there’s a lack of critical thinking at play.
Also, I think the X.com links only work if you have a login or something, fyi.
I've never worked inside one of the leading edge AI companies like OpenAI, Google, Microsoft or Meta.
Is this comparable to what they would work with?
My first guess is that it seems much smaller. And if you are running many parallel training jobs then you are getting about 1,000 chips at most to work with.
Or is this about what the leading competitors are working with?
Azure, for one, seems to have orders of magnitude more chips at their disposal.
That said, these GPUs aren't just the GPUs. They are whole chassis. They are huge onboard storage arrays, TB's of RAM, 800G networking (and associated cables), racks, cooling, power distribution, backup power, etc...
None of it is easy.
I did a lot of unique optimizations to autotune each individual GPU for performance by tweaking the software knobs on them. They are all snowflakes. Same model, different batches (heck, even same batch!), can produce wildly different performance results.
Over the years, I did try to find some alternative workloads for it, but nothing could even pay for the power costs. The GPUs were very old models (rx470-rx580) and the rest of the hardware wasn't that advanced, like it is in AI, so none of it was transferred.
I'm in the process of building my own AI supercomputer now. Really looking forward to seeing how it turns out.
Nobody else on the planet was able to automate the tuning like I did, which had a direct influence on ROI. I know this because it required a very specific change to the AMD drivers to enable that functionality to happen.
That said, latest 4th gen nvlink is 900G...
https://www.nvidia.com/en-us/data-center/nvlink/
But unless you're sleeping with Jensen, you're not going to see it for 52 weeks+ after you order it.
Our lead time hasn't been horrible actually, but I work for a pretty big corp
I don’t know much about AV processing, that’s highly customized to only a few customers but I’d expect it to also have very large computational requirements to do video processing and reinforcement learning.
Probably some scientific modelling that can be done on these, so I bet some universities and private labs will be buying them. NASA, SpaceX, RocketLab, Helion, etc.
There's also probably a lot of AAA game studios and art studios for movies etc. who are each buying dozens of these graphics processing units for… graphics :P
I can imagine they either underestimated the software effort needed to squeeze as much performance as possible out of those things, or they underestimated the pace at which Nvidia scales FLOPS/$, or both.
What would you say grants you the standing to opine here?
Until then, all of you sound vicious, bitter, and hypocritical.
I love Carmack, but having an idea for something is infinitely easier than actually successfully executing on it, especially as incredibly successfully as SpaceX.
Carmack did execute, they had a working rocket, and with time they could've solved the software issues, but they didn't have the funding to blow up dozens of prototypes.
Carmack drove that project, he wrote code, he built rocket engines himself, he ran missions.
Elon, notably, doesn't do any of that. He just had more money, or was willing to commit more money, to seeing it through. For Carmack it was more of a fun diversion (the X Prize) than a business he wanted to build.
And given how much faster SpaceX has been than anyone else, I can only believe this "could've, would've" in form of the slightly longer hypothetical "if only Carmack hired all the rocket scientists (and raised all the money to give them freedom to go fast) before Musk got there".
Sucks to be if you paid the early bird fee for it.
Also if it didn’t exclude everything that has been delivered. That long list would be interesting to see as well.
Previus article: https://www.tomshardware.com/news/teslas-dollar300-million-a...
This is second-hand blogspam.
Yes.
I was curious why this statement lead with fp64 flops (instead of fp32, perhaps), but I looked up the H100 specs, and NV’s marketing page does the same thing. They’re obviously talking about the H100 SXM here, which has the same peak theoretical fp64 throughput as fp32. The cluster perf is estimated by multiplying the GPU perf by 10k.
Also, obviously, int8 tensor ops aren’t ‘FLOPS’. I think Nvidia calls them “TOPS” (tensor ops). There is a separate metric for ‘tensor flops’ or TF32.
Also, back in the day, integer ops were just called 'ops', grumble grumble. But yeah FLOPS specifically refers to floating point. Calling them TOPS doesn't make sense to me, since tensor cores were meant for matrix operation speedup, and these matrices are rarely integer.
> knowing you can drop to high precision when necessary without penalty is nice.
I guess I maybe don’t know why you’d ever have 1:1 fp32 and fp64 perf. Aren’t the fp64 multipliers (for example) basically 4x fp32 multipliers? I am under the possibly naive impression that if you have all the transistors for 1 fp64 core, that you’d end up with all the transistors you need for 2 or 4 fp32 cores. Maybe that’s not true today, but there does have to be at least 2x the transistors overall for 64-bit vs 32-bit, and lots of those should be shared or reusable, no? It doesn’t seem quite right to frame naturally higher 32-bit op throughput as a “penalty” on 64-bit ops. You’re asking the hardware to do more with 64, and it makes complete sense that given the exact same budget for bandwidth, energy, memory, compute, etc. that 32-bit ops would go faster, no? If the op throughput of fp64 and fp32 is the same, doesn’t that possibly imply that the fp32 ops are potentially being wasted / penalized, just for the sake of having matching numbers?
I see mention of using this supercomputer for training models. Is that the only purpose? What other types of things do orgs usually do with these supercomputers?
Are there any good boots-on-the-ground technical blogs that provide interesting detail on day-to-day experiences with these things?
In other words, they're used when you want to share some kind of state across all of the computers, without the potential overhead of communicating to some other system like a database.
Physics simulations and like, molecular modeling come to mind as common examples.
In the case of ML training, model parameters and broadcasting the deltas that get calculated during training are that shared state.
The cost of finding the next prime is likely into the millions now.
Weird to think that his next company's compute platform is this.
Unforced rewrites are usually always a bad idea.
The key thing here however is that Elon didn’t want whatever he asked for in a vacuum, despite what the book says. Surely this was his engineer’s preference.
That seems like a bold claim. Google, Microsoft and Meta make so much more money than Telsa that if making AI chips was so easy, then they could clearly out design and build Tesla without thinking too hard about it.
What makes you think that Telsa, a company with far less AI workers and knowledge, and far less money than the above companies can out design and out build them?
Presumably because Elon himself will be involved in the design, and Elon, as we all know, is one of the world's great thinkers. ;)
From what I heard about SpaceX it seems to be a place grads go to burn out while being paid below market rate simply because they're excited about the idea. Maybe that impression is wrong, so I'd like to hear other perspectives.
I recently spoke with someone who quit SpaceX because (among other reasons) they felt Elon was a meddling micro-manager. That's just one anecdote, of course, but the Internet is full of them, replete with summary firings (https://www.businessinsider.com/tesla-elon-musk-ruthlessly-f...), worker safety issues (https://www.washingtonpost.com/technology/2021/03/12/hundred...), and just general bullshit (https://www.reddit.com/r/EnoughMuskSpam/comments/9e360m/elon...).
I don't deny that his public image, for years, was an overall positive one. I really enjoyed Jill Lepore's digging into it here: https://www.pushkin.fm/podcasts/elon-musk-the-evening-rocket.
But it seems like people who worked with him knew, for a long time, that he was full of shit. And increasingly, the public seems to as well.
It's part of the reason why AMD has had quite a bit of success here but is in single digit market share for "AI" otherwise.
Most people - even large orgs with thousands of GPUs - are so trapped in CUDA the theoretical on paper performance and cost benefits evaporate immediately when you spend all of your time trying to port everything over to the point you get equivalent performance and functionality.
https://twitter.com/SawyerMerritt/status/1696011140508045660
Wow, that's some really early hardware access. /s
https://en.wikipedia.org/wiki/Tesla_Dojo
From the History section (although Technical Architecure is also worthy of consuming in its entirety):
> In August 2023, Tesla powered on Dojo for production use as well as a new training cluster configured with 10,000 Nvidia H100 GPUs.
I’ll take the L wrt being flippant if we’re using words very specifically in this context, that’s fair. It’s great to see Tesla expand its training resources is my sentiment, regardless of how their aggregate ML compute is segmented.