Nvidia DGX GH200: 100 Terabyte GPU Memory System
developer.nvidia.com
developer.nvidia.com
How the rest of the industry can respond is such a mystery. And will it be lone competitors, or will a new PC era be able to start, with an ecosystem of capabilities?
edit: actually now i can't find the claim, maybe i misremember what the papers said.
Software for TPU is still in its early stages. CUDA is well established. You can test on a gaming GPU that you can find (locally!) in many markets. XLA is meant to solve this, but first impressions matter and my first impression was that it has not yet "solved" this issue.
TPU is only available via Google Cloud - as far as I know they don't have NVIDIA's widespread distribution to various HPC/supercomputer systems. This also has implications on scaling up more than a few pods, as they will need to be colocated with speedy interconnect (which is provided by the various existing HPC systems that use NVIDIA's chips).
Finally, I think many people are discovering that the supposed benefits of TPU are marginal at best in the face of the types of natural scaling issues that both GPU's and TPU's suffer from when scaling out to e.g. hundreds of pods.
I'm certain that someone with more experience than I could give a better answer though - and again, all speculation. I refuse to use TPU because Google Cloud's system for getting access to said TPU's was horrible for me when I tried it. I believe John Carmack has a nice tweet thread specifying the same issues I ran into.
In general, Google has a habit of developing tech for other Googlers first, and as such winds up ignoring a lot of real-world scenarios faced by researchers/practitioners. NVIDIA on the other hand has been working directly with a ton of institutions and businesses ever since the inception of CUDA.
That their TPU's have seen any adoption at all is mostly due to their research program which granted very cheap access to TPU's to tons of people.
(this is not meant to be reassuring).
Coupled with a magazine or a show presenting new product categories for those interested, customers will eventually visit a physical or online shop and check out the goods. And then word of mouth will do the rest.
Aggressive advertising will mostly just help you get ahead of your competitors and perhaps speed up the adoption rate at the cost of increased volatility of the market and to the detriment of people's mental health.
We would be better off regulating aggressive ads away.
That's a hell of a load-bearing "just" you managed to insert there. Getting ahead of your competitors in market share can be the difference between having a company succeed or fail.
Or: all advertisers of all brands with a same or similar product must collaborate. Only voluntary input counts as collaboration; if a brand simply doesn't care about presentation of itself in the advertisement, they have trivially collaborated. Easiest way to implement this is giving every owner of all relevant brands a right to veto every entire final advertisement product (this right could also be surrendered, for all or some possible vetoed advertisements, in exchange for something in a contract).
Ignoring flaws of this proposition itself, what could be society's reasons for rejecting it? Does society perhaps want havers of more money to gain further advantage over havers of less money?
Maybe 50 years ago that would have worked. Today, not so much. Go to Amazon and look, well, just about anything. What is BEHENO, what is DINGEE, what is Etoolia, what is Romedia, what are the over 300 different 6/7 letter companies that show up when I search up some random product.
Unfortunately your consideration causes its own parasite effect of countless companies forming up to feed of the big advertisers budget.
And, you're also crating a regulatory nightmare. Say I put up an add for XXYZXX company, and it includes ZZXYZZ and YYXZYY information (I mean totally random picks), and I just happen to have a stake in those companies too. Now you're going to have to track hundreds of thousands of these entities to ensure no fraud is occurring, and in most cases the fines for this kind of behavior are well under the cost of doing business.
Everything you've said so far just creates bigger messes and solves nothing.
As a result, spending ad money on Google is ridiculously expensive, but companies accept this because there is no alternative hoping to "build long lasting relations" with the people who make them pay upwards of 1 dollar per click
Also, if your established it probably a good idea not to let new competitors get a foot hold in the market with an easy google win.
It's also pretty effective for local businesses because not a lot of local businesses are tech savvy enough use it effectively.
FWIW, the cheapest (quality) clicks I've seen, at least in the B2B space, is closer to $3/click, and it can quickly balloon to upwards of $10/click especially on company brand names where competitors are bidding on another company's brand name.
Knowing this, I cringe every time I'm screensharing with someone and they search "[B2B Company] login" to login to a tool they use every day. Each login = $2-$10
It's not uncommon for companies to spend $100k+/year JUST bidding on their own company name.
I just don't get where all the marketing money is coming from. Bootstrapping is clearly not an option these days
That bag of chips did not cost $4 to make, not even a little close.
Conversely, in Communist systems they could never get this right. Factories were just told to produce 5 or 10% more than last year, didn’t matter if the product quality was worse or if people didn’t want it.
Marketing is already difficult to tell apart from other company communications, product documentation, etc. What about a company blog showing how to use their products? Is that marketing or product documentation?
Tax ad companies, and spend that money on education. The better ad companies are doing, the more we spend on education, the fewer gullible marks we produce, the worse ad companies will do.
Informative ads make the market more efficient. Persuasive ads actively make the market less efficient.
Most ads in the US in 2023 seem to be persuasive.
Perhaps the ad industry would become more useful (and smaller) if we managed to effectively regulate it to significantly reduce the persuasive bits.
I think that most people would support this if you explained it right - from the free-market perspective, this would give you a better market.
So we can have internet anywhere. To click on ads.
Think about all the stuff ChatGPT and GPT-4 can do with even minimal prompting. Even when they hallucinate, the text is still ostensibly coherent and natural sounding. Now imagine a similarly powerful model, but its input is a ton of metadata about your behavior and its output is ads.
Now consider that adtech has had substantially more funding for substantially longer than research into LLMs, so ad serving models are probably way more powerful and optimized than even GPT-4.
It's freaky to think about indeed.
More competition would be fantastic. Better pricing at scale would be fantastic. But there is absolutely no doubt that nvidia is far ahead of Google right now. Tesla made some believably pushing claims about their own efforts with their own hardware, so who knows maybe they're the real challenger.
I don’t think they will exist in 10 years.
gmail is a freebie! the core product is how they index your messages to create an anonymous profile that they will then offer on reverse bid to advertisers when you do a search or visits an AdWords site.
Businesses fail. google will likely still exist, but alphabet, I don’t see a future for - just a gradual withering followed by a collapse and disintegration into myriad properties in a fire sale. They are brittle, overburdened by unity of disparity, culturally adrift, and they aren’t taking risks any more. Inertia will keep it all going for a while, but not forever.
Sure, I may be wrong, but I do put my money where my mouth is, and I am right more often than not.
I see a lot of hate for alphabet on HN. It seems very emotional. I think people feel personally betrayed by thier bad behaviours because they were 'supposed to be better'.
The thing is, there are a lot of companies you can hate. Exon, mcdonalds, blackrock, even Microsoft, there are people who are very mad at these companies.
That's not an argument that the company is doomed. If you are really putting your money where your mouth is (what shorting google?) Then I hope you have a better reasoning as to why they will fail not just eventually but this year.
None of the companies you list are likely to collapse soon, as they remain focussed on their various missions, and have a unity of purpose. Out of all of them, I think Microsoft is the most likely to fail, as they are likely to be blindsided when the user-focussed desktop OS era ends. Their diversification efforts have been a mixed bag, and without windows, they are far, far less significant.
What I do look at is sentiment analysis - what other people feel and think about businesses, as that drives the market.
No, I don’t short, as just buying equities which are beginning significant growth is just as effective and doesn’t drive demise - I held goog for nearly 20 years, and sold off late ‘21, as I think they’ve peaked, and anything from here on is speculative froth.
You’ll note I keep saying “I think”, rather than making statements of fact - because this is purely what I think - I am not a Sybil.
You seem to have missed this:
>> They are brittle, overburdened by unity of disparity, culturally adrift, and they aren’t taking risks any more.
My perception is that Google split into a number of focussed business units when they became Alphabet, with the Google component being execution focussed and the more speculative stuff spun out into other group companies like deepmind, waymo, etc. That's why the Google unit stopped doing nice incubator projects that we were all excited about.
From what I've seen, this cash cow execution business unit has been fairly effective - in particular they've done a good job of entering the cloud market space producing a differentiated product that is penetrating their target customers. They have not been able to compete with Microsofts excelent and deeply embedded IT sales capability, so they've done well to go after people with big problems that other vendors more civillian offerings are not so great for. They are currently the first choice platform for AI training for instance.
I'd contrast this to Facebook who seem to be trying to become a deep tech VR hardware vendor in the same business unit as their cash cow entertainemet and advertising business which has confused investors and probably distracted their focus.
We can see that Google has innovated. For instance, a lot of Tela's stock price is based on the idea that they are going to run autonomous taxis, and instead of owning cars we will just hail a Tesla when we need one. Telsa does not run autonomous taxis, but you can ride a Google Waymo taxi today in Pheonix, and they are running autonmous trucks which is a big industry Tesla aren't even attempting yet. They are doing a lot in medicine and medical devices. This seems a lot more diversified and innovative than other companies - it's just not as visible to the HN community as an RSS reader or some other internet thing we care about.
We can also say that... on the AI thing, I think it's very early days. Microsoft have a shakey looking deal with the first mover, but Alphabet and Facebook have the advantage of actually using AI extensively in their real buisnesses and may be able to deliver product market fit better. Time will tell.
On the stocks front, I agree with your overall thesis - I think it's harder for these conglomerates to grow than a new company just because they are already giants in their niche and even adding a new niche generates less growth in percentage terms than for a smaller company starting from a lower number. I just wouldn't actually bet against google as much as I would some of the others.
That might have been a reasonable assessment back in Balmer’s era. But what you saying has already happened years ago…
They have mostly reinvented themselves since then. Enterprise/office isn’t going anywhere. Xbox if fine too. And there is a lot of growth in their cloud/etc. business.
IMHO out of Google, Amazon & Facebook, Microsoft seems to be the least dysfunctional and and general best positioned one to be successful in the future.
We can both give the money to a mutually trusted third party now.
Their search products have actually gotten worse with AI. Google Images running just off basic image recognition (as in is this the same image) and the context of where they found it was far superior at identifying what an image is than ML Google Image.
The OG version could identify a frame from a movie and provide higher res versions. The ML version goes “errr looks like a woman on a street, here are random photos of unrelated women on unrelated streets with maybe a similar color scheme” close to useless why would anyone want that. Yandex Image search blows it out of the water simply by being Google Image Search from a decade ago
The overall theme is that product is no longer the focus, but rather navel-gazing - that’s to say, their internal world no longer aligns with the external world, and that is a fundamentally dangerous place for a business.
Do you just have more email now?
E.g. what’s possible with embedding where the query terms are matched with similar messages that mean the same thing.
Also worth noting - TPUv4 uses a 6-way 3D torus interconnect vs the 3-way "multi ToR" NVLINK topology; the total bisection bandwidth of the TPUv4 pod is over 1PB/s!
Can't wait to see what TPUv5 looks like. As you say, it's probably already chugging away with v6 on track to tape out in a year.
That said, I think NVidia has nailed bringing the ecosystem along, and I think making the whole setup look more like "one huge GPU" could simplify a lot of ML programming.
I am actually disappointed I haven't seen more of that style in CPU programming. Where's my 20,000 core 100TB RAM VM instance?
You could simulate this with a bunch of regular machines and a networked hypervisor.
You could do some kind of smart caching so that processes rarely need to wait to access RAM stored on a remote machine.
Combined that with a big lock eliding/speculation scheme (ie. When a process reads memory that might have been written by a remote CPU, you continue as if it hadn't, and if you later find out that data was written then you rollback). These rollbacks 'undo' all work done in however many microseconds it takes for data to travel from one side of the machine cluster to another.
Reads of RAM that aren't cached yet on the local node can also be speculated - you just assume that RAM contained null bytes and continue execution, rolling back and replaying when the actual data arrives.
So if you can make sure that processes are contending for locks and writing conflicting data less often than once per system-roundtrip-latency, then you should get a high performance system.
But upon further thought, a lot of things such a system would need are actually rather inefficient to implement in software (ie. rollbackable RAM), yet quite cheap in hardware (for example rollbackable RAM can be implemented with regular RAM plus either a buffer of 'overwritten data' or a write queue)
The biggest issue it seems is bandwidth and humans' patience for a response...
So 1/100th of the CPU and 40% of the RAM. (I suspect the RAM comparison is reasonable - I'm not sure about how to compare the CPU's).
The nvidia device uses a fabric with 900 GBps switched fabric between any of the 256 nodes in the system. The TPUv4 3d torus network is basically a ring network of 56 GBps connections creating separate rings. From a raw perspective, the nvidia solution is the overwhelming winner. There is absolutely no contest.
Machines with terabytes of RAM do exist and get used - working well on such setups is a goal of modern JVM GCs for instance - but making a single machine that large which acts like a single machine isn't easy, nor especially desirable. One machine is a unified failure domains outside of mainframe-land, so if you had a 20k core machine with 100TB of RAM you could never reboot it to apply OS updates and it'd die all the time from failed parts.
Even once you get beyond that most software stacks use locking and stop scaling beyond a few hundred cores at best and that's assuming very heavily optimized stacks. AI workloads are easier because they're designed from scratch to be inherently parallel without lots of little locks and custom data structures all over the place like a regular computer has.
Disk storage is one of the places where you can parallelize and scale out relatively easily and you do see datacenter sized disks there.
In hindsight the 40GB of SRAM feels kind of quaint, but nevertheless their very fat nodes let them get away with more than Nvidia could with A100s, as you can see in the slides.
CS2 is a little old now. I bet an update is just around the corner.
If the competitors (mainly AMD, Intel and to some extent ARM) will keep seeing growing volumes and insane margins they will be attracted to bring to invest and take part of that market.
Till now gaming GPU market did not bring to AMD the necessary margins to really push them to bring a better competition to Nvidia. Even 10/15 years ago when ATI was way ahead of Nvidia technologically for 2/3 years (the HD 4000 and HD 5000 generations vs the Nvidia flops of the 9000, 200 and 400 series) Nvidia was posting billions of profits and ATI posted a whole...19 millions of profits across 3 years.
But today's GPU market thanks to it's non-gaming sales is much bigger to ignore (which is why Intel entered it as well) and those players will likely react.
You don't need to have the best premier product, you need to have your products good and priced well enough that they will be chosen over the competitor's.
So I guess my question is - what use case is there for a huge truck that goes 200mph and take 4 trips, when you could just buy 16 regular trucks, and move your apartment in the same amount of time at half the cost.
The other approach, which you do when models themselves are massive, is model parallelism. You split it into multiple parts that run on different nodes.
In both cases, you need to distribute weight updates through the network although the traffic patterns can be different.
To maximize the performance in both scenarios, systems designers optimize for all-reduce and bisection bandwidth.
There are also other tricks, for example the TPUv4 ICI network is optically switched, and it is configured when a workload starts to maximize bandwidth for the requested topology ("twisting the torus" in the published paper).
'The street finds it's own uses for things' is the well known Gibson adage, I and typically it's a comment aimed low. But our entire era of amazing computing began with the Gang of Nine enabling lowness in a degree such that it quickly became the highest tech, the best. Sure you can still buy a mainframe & they have amazing feats but it's not where the value is, but and the value is where it is because possibility was unchained, I unleashed from corporate dominion, and spread wide. I think we can find amazing new futures with CXL & mad bandwidth connectivity.
And that's because it is very high skill work, otherwise, it would have been 100 engineers paid tens of thousands.
However, I’m not even sure enough text data exists in the world to saturate 100T parameters. Maybe if you generated massive quantities of text with GPT-4 and used that dataset as your pre-training data. Training on the entirety of the internet then becomes just another fine tuning step. The bulk of the training could be on some 400TB dataset of generated text.
LLaMA can apparently run quantized to 4 bits per param (not sure if worth it though), which would allow you to run a 200TB model on one of these cards if I'm understanding right.
In any case, it's almost certainly not bigger than 1T, even if it's not a dense transformer (PaLM-2 is and makes do with 340B, but it isn't exactly on par).
From the GPTQ paper https://arxiv.org/abs/2210.17323:
"... with negligible accuracy degradation relative to the uncompressed baseline"
A machine like this would top out below 2 trillion parameters using the training algorithms that I'm familiar with.
For LLMs you need to load single row of context size, that's vector of ie. 8k numbers, which is 32kB for single precision floats.
Even for fp64 it adds only 16 bytes.
RMSPRop, Adagrad have half of this overhead.
SGD has no optimizer overhead of course.
But all of that (trainable parameters, activations, optimizer state) is like 12 bytes per trainable parameter, not 80.
For backpropagation you take the diff between actual and expected output and you go backwards to calculate derivate and apply it with optimiser - that's 8 extra bytes for single precision floats per trainable parameter.
Why do you need 80?
The ML engineers in my group chat say "1 DGX to train 7b at full speed, 2 for 13b, 4 for 30b, 8 for 65b"
I hope these models move significantly beyond text at some point. For backend programmers it's ok, but for the rest of the technical world (circuits, mechanical engineering, front end, sound, etc), it's fairly limited.
No, those rumors seemed ridiculous even then. Many AI influencers were posting some of the most absurd material, often makes basic mistakes (like confusing training tokens with parameters), but anyone in the field could have easily told you that 100T parameters sounded ridiculous.
On that note, "100 Terabytes of GPU memory is exactly what you need to train that class of model." is also likely false. That's how much you'd need to fit such a model into memory at 1 byte per param. Not train it.
GPT-4 was trained on image data. Besides gaining understanding of image content it also showed improved language abilities over a GPT-4 trained with only text. Facebook is working on a smaller model with text, image, video, audio, lidar depth, infrared heat, and 6-axis motion data. If a GPT-4 was trained with data like that, what capabilities would it have? Rumor says we will know in a few months.
~6DP * precision
Where D is number of tokens*mini batch size and P is number of parameters.
So if you want to fit fully into memory with a mini batch of 1, context window 32k, and 16 bit precision, that’s 144e12/6/32e3/2 = 375M param.
If you apply one token at a time then
144e12/6/2 = 12 T param
Ofc, in reality you have model parallelism as well…
Hello spurious regression
You can't train a 100T model with "only" 100TB of VRAM, you need for each parameters 4 bytes + 4 bytes (gradient) + 8 bytes (AdamW optimizer) + forward activations that depends on the batch size, sequence length etc, maybe more if you use mixed precision and also you need to distribute the weights.
But in this case what is really ridiculous is the compute requirement. The required compute for optimal model growth roughly quadratically (both your model and your data grow linearly). So for 100T model you need 1e30 FLOPs. This machine gives you 1e18 FLOPs per second. It will take 30k years to train this model on one of these (or 30k of these to train it in a year, but then utilization will start kicking in).
I.E. ; "answer this question where the outcome is the most beneficial to quality of life"
A strange game. The only winning move is not to play. unplugs self
Please ELI5 where I can have a glossary of AI/ML terms - where do I get fluency in speaking about Tokens, Models, Training, Parameters, etc...
Please dont be Snarky - This is info that everyone younger than I am needs as well.
Is there a Canon? Where is it?
https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-...
It's pretty lengthy but doesn't require a PhD to understand. If you can get to the end of it you'll have a much better understanding of what's going on.
That rule of thumb is wrong. The chinchilla paper has it anywhere between 1 and 100 tokens per parameter.
Also, I think training in simulated realities will be big, especially for learning how to interact with complex systems, for developing strategic planning heuristics.
EDIT: so the whitepaper is surprisingly good for that (somehow all the articles are very weird...): https://resources.nvidia.com/en-us-grace-cpu/nvidia-grace-ho... - essentially they connect the GPUs with NVlink instead of PCIe (so, vertical integrators heaven) and then NVLink forms a separate interconnect for GPUs. So this is cool and essentially what Fujitsu, Google, ... have done for some time. A fun thing is, that they like to add up their nvlink-duplex bandwith and don't do for PCIe... (which then suddenly would have the same bandwith as the GPU-side).
Still very cool to see the mainframe come back alive ...
(it's a bit sad they bought Mellanox - monopolies are sad...)
What is also beyond me is how someone at Nvidia thinks that the label sequence "1.00E+2; 1.00E+3; 1.00E+4; 1.00E+5; 1.00E+6" for the vertical axis in "Figure 1" is more readable than "100; 1,000; 10,000; 100,000; 1,000,000" would have been. The latter is 5 chars less (total), even. Or, if exponential notation is important for the Big Serious Computing People, then perhaps they could have dropped the ".00" part from each value? Or, if I'm allowed to dream, gone with actual exponential notation?
Same. It's just more efficient and readable that counting the 0 while considering the culture bound digit group separator norms like thousands/millions 3,3 vs laks/crore 2,2,3 cf https://en.wikipedia.org/wiki/Indian_numbering_system?useski...
Personally, I think it'd have been better to say the the .00 adding nothing: 1E2 1E3 etc would be far better.
In this case, it is pointless though, since the precision is actually known.
And at 1/8th the power per GB, you have 700 Watts / 96GB / 8 * 480GB come to around 450 Watts per. And 115kw for the 256.
However while the hardware isn’t cheap, it’s still likely not a blocker. Costs do inhibitor more experimental research.
Selling shovels is good business, but doesn't compete directly in AI area.
No
6x 3000W PSUs in a 3x3 redundant config. So 9000 watts total. So at least 240v x 50A. x2 for redundancy.
This seems fairly common e.g. https://www.currys.co.uk/products/aeg-ikb64401fb-59-cm-elect...
I am sure data centers have larger circuit breakers and chunkier cables than my kitchen appliances!
Front left: 2.3 kW / 3.7 kW
I know that I will never be able to afford such a thing (or possibly even afford to power it for more than a few minutes), but a man can dream.
I remember hearing a lot about rankings of supercomputers, but less so about what they actually achieved.
1. workload for national labs this is mostly sparse fp64 in my understanding, for warehouse-scale computing is lots of integer work, highly branchy, lots of pointer chasing, stuff like that.
2. latency/reliability vs throughput warehouse-scale computing jobs often run at awful utilization, in the 5-20% range depending on how you measure, in order to respond to shocks of various kinds and provide nice abstractions for developers. fundamentally these systems are used live by humans and human time is very valuable so making sure it stays up always and returns quickly is paramount. In my understanding supercomputing workloads are much more throughput-oriented, where you need to do an enormous amount of computation to get some answer but it doesn't much matter whether the answer comes in one week or two weeks.
3. interconnect warehouse-scale computing workloads are mostly fairly separable and the place where different requests become intertangled is in the database. In the supercomputing world, in my understanding, there are often significant interconnect needs all the time, so extremely high performance networking is emphasized.
1. Low latency network, 1-2us. Most servers can't ping their local switch that quickly, let alone the most distant switch for 1M nodes
2. High bandwidth network, at least 200gbit
3. A parallel filesystem
4. Very few node types.
5. Network topology designed for low latency/high bandwidth, things like hypercube, dragonfly, or fat tree.
6. Software stack that is aware of the topology and makes use of it for efficiency and collective operations,
7. Tuned system images to minimize noise, maximize efficiency, and reduce context switches and interupts. Reserving cores for handing interrupts is common at larger core counts.Absolutely, they contribute to research all of the time.
Some of them have pages where they list research outputs that they enabled (though this is of course limited to those authors tell them about!).
supercomputers do all the hard work in research universities all the time. Hell, astrophysics and research involving telescopes and observatories use em all the time.
If you're talking about the more stereotypical "high performance supercomputers", I think that they are still used very liberally within the defense industry. I think Lockheed Martin, for example, uses them for CFD analysis.
Besides the outcomes that were adopted by the industry, before cloud computing there was grid computing, exactly to manage such resources at scale.
Generally given there are a much larger number of less powerful computers, more accessible to much scrappier interests, one would expect more innovation to be done on them.
[1] http://research.baidu.com/Public/uploads/5e76df66c467b.pdf
But systems that can host many GPUs tend to be expensive, and electricity is expensive, so at scale the expensive GPUs make sense. For a homebrew solution you can stick four consumer GPUs in a case and might save a buck.
https://thenewstack.io/how-ray-a-distributed-ai-framework-he...
The difference is that Nvidia's software and hardware stack combined makes all these systems, all these aggregate GPUs, look like One Really Big GPU. Not hundreds of small ones. That's not only good for users because they can take existing programs and migrate them to these big machines and get improved performance, but also good because it's generally much easier to program "one big machine" as opposed to programming and orchestrating many small ones. This is an attractive proposition for many but it requires an insane amount of integration to achieve.
So, the major differentiator here isn't the lack of or existence of many discrete machines connected together. It's the programming model, at this scale, that's different. And Nvidia is way ahead of everyone else here in terms of programming models; once full heterogeneous memory management for CUDA arrives in a stable consumer driver, it'll be a massive change for others to catch up with.
What you might also be referring to is the idea of "distributed training", or what is called "ensemble learning" where you individually train a bunch of small unique models that, when combined together, perform better than if they were one giant model (or at least are as accurate/efficient as a giant model.) It's "The P2P model" of training because you can take lots of small models and collectively aggregate them. That's an open problem people are attacking but not really relevant in the case of the DGX.
Many hyperscalers, such as Microsoft and their project "Brainwave", have very complex heterogeneous AI datacenter stacks consisting of GPUs, FPGAs, TPUs and CPUs. (Google "Microsoft Brainwave" for some papers.) This DGX is positioned as an alternative to that but also as a tool for their customers to use since many want to train large models efficiently.
This makes me feel like we're close to that one terrifying short-story.
To paraphrase an analogy I've heard somewhere (in similar context) - We're building better and better ladders, maybe even lifts with this last push in ML field. But the brain is on the moon - even the best lifts won't get us there.
At the upper bound: In molecular dynamics, which is used extensively in modern day neuroscience to understand the function of ion channels and GPCRs, a single H100 can model 70ns/day of compute for 1M atoms. There are 8.64e+13 nanoseconds per day. There are ~10^26 atoms in a human brain. Therefore, an upper limit back of envelope is you need fewer than 10e+26 atoms / 10e+9 atoms * 8.64e+13 ns / 70 ns = 1.23e+29 H100 GPUs.
Calculating the lower bound is more difficult, but let’s start by saying you can get away with a fp16 for each synapse. Storing the weights of that model for 100 trillion synapses is 200 Terabytes, and if you figure weight size * 4 or so to do anything useful then this is in spitting distance. Note that this example lower bound is massively less complex than the Neuron model I suggested, as the entire field of neuromodulators, homeostatic mechanisms, glia, and more are thrown out, which are all important for modeling how the brain works under certain computational regimes.
My bio knowledge is very basic, so forgive naiviety in these two questions.
First, I'm not asking you to go through the math on the spot, but I'm guessing that lower-bound capability is well understood in 'the field', but is it documented against various species? Perhaps mapping against current / projected GPU/compute systems capabilities? (I know there's a project to model a worm's brain, IIRC down to molecular level. But I'm picturing a 'we are 3 years away from being able to emulate a basset hound, 4 years for a border collie' - that kind of roadmap.)
Second, you said upper bound is to ignore sub-atomic. I thought we had proton and electron gradients, at least in metabolism. I believe proton there is a synonym for Hydrogen (atom), but electron would imply some potential need to emulate at sub-atomic? Have I misunderstood the bounding / chemistry involved?
I believe the situation is a lot more extreme than the absurd efficiency of differentiable programming. I have been meaning to write up (but been too busy to do so) an insight where I believe training can be made ridiculously cheap computationally speaking (in a way that combines with differentiable programming, not replaces it). I am agnostic if this is what the brain does, but wouldn't be surprised at all if the brain does in fact do back-propagation (or uses the insight that I've been meaning to write up).
[1] https://en.wikipedia.org/wiki/Hodgkin%E2%80%93Huxley_model
[2] https://en.wikipedia.org/wiki/Reaction%E2%80%93diffusion_sys...
[3] Michael Levin | Cell Intelligence in Physiological and Morphological Spaces, https://www.youtube.com/watch?v=jLiHLDrOTW8
https://www.quantamagazine.org/how-computationally-complex-i...
> "If each biological neuron is like a five-layer artificial neural network, then perhaps an image classification network with 50 layers is equivalent to 10 real neurons in a biological network."
The complexity explodes quickly because each biological neuron's behavior is modulated by a large number of biochemical neurotransmitters, on top of all the dendritic connections (up to 15,000 each, apparently).
(edit: thanks for both replies already, and any others that might fit; I understand now the reference was likely to Asimov)
Why would you want to make anything close to the brain? What real scientific or engineering or humanitarian uses does doing that even have? AI is already and going forward should strive to be a groundup of redesign of intelligence.
To have models of the human brain that we can poke at and change and tinker with and etc., so that we can get better ideas of how therapy techniques, medications, ... will impact the actual real people that might benefit from them.
To your second point, we have little to no ability to understand yet what quantum effects may or may not be active in brain/consciousness function. We certainly can’t exclude the possibility.
We can fairly well exclude the possibility of interesting quantum effects in human consciousness, because the human brain is a hot, dense environment that might as well have been literally designed to eliminate the possibility. It's the exact opposite of how you want a quantum computer to be built.
Which doesn't mean there aren't plenty of quantum effects involved in the molecular physics, but that isn't what is normally meant by 'quantum computer'. Transistors would also meet that definition.
1) That 433 qubits does not make a computer and is instead “a half dozen logic gates.” I agree a half dozen logic gates is not a computer. 433 qubits is not comparable in terms of information capacity or processing capacity to a half dozen logic gates. This number is also publicly doubling annually now — I would bet the systems we don’t know about are more complex. Importantly, a computer in this context is not something you would attach a monitor to — it is just an electronic device for storing and processing data.
2) That we have any good idea of the limits of how biological systems might be influenced by quantum effects within specific temperature ranges. You certainly wouldn’t construct a human brain to interface with quantum effects given the present state of our knowledge in constructing these kinds of systems. But then we can’t even construct a self-replicating cell yet, nevermind a brain. It’s hard to imagine we understand the limits at work there.
Says who?
Sure, if you assume that “AGI is just scaling up GPT”, it will be digital and silicon. But that’s a big assumption.
For all we know, AGI will only ever, if it exists, be analog and chemical.
> We have no idea how the map from one to the other.
Plus, even if we had an easy one-to-one mapping function between them, we don’t understand the source well enough to do the mapping.
[1] https://protosupplies.com/product/pcf8591-a-d-and-d-a-conver...
I hope this improvement translates to consumer GPUs as 24GB is a big limitation.
I see no end in sight for that specific law. As we can always go vertically if needed. It’s also held through so far in 2020[1].
The folks who conflated moores law to also mean doubling of compute processing capabilities of a CPU double every 2 years were wrong.
This ceiling will be hit much earlier than what process/technique allowed.
Scaling vertically would “technically” still meet the above.
> The complexity for minimum component costs has increased at a rate of roughly a factor of two per year. Certainly over the short term this rate can be expected to continue, if not to increase. Over the longer term, the rate of increase is a bit more uncertain, although there is no reason to believe it will not remain nearly constant for at least 10 years.
If he was talking about the area of a single transistor, there would be more concise ways to put it.
Another way to say it is to count the famous founder brown: ymmoore wasn't thinking 4th dimensionally.
For shame really
Not a hardware person but heat dissipation becomes more of a problem when you go vertical IIRC.
I don't think you understand how difficult non-planar transistors are to engineer at scale
> "The original Moore’s Law came out of an article I published in 1965...I had no idea this was going to be an accurate prediction, but amazingly enough instead of ten doubling, we got 9 over the 10 years, but still followed pretty well along the curve. And one of my friends, Dr. Carver Mead, a Professor at Cal Tech, dubbed this Moore’s Law. So the original one was doubling every year in complexity now in 1975, I had to go back and revisit this... and I noticed we were losing one of the key factors that let us make this remarkable rate of progress... and it was one that was contributing about half of the advances were making. So then I changed it to looking forward, we’d only be doubling every couple of years, and that was really the two predictions I made. Now the one that gets quoted is doubling every 18 months...I think it was Dave House, who used to work here at Intel, did that, he decided that the complexity was doubling every two years and the transistors were getting faster, that computer performance was going to double every 18 months... but that’s what got on Intel’s Website... and everything else. I never said 18 months that’s the way it often gets quoted."
Anyways, See slide 13 here[2] (2021). "Pop-culture" Moore's law stated that the number of transistors per area will double every n months. That's still happening. Besides, neither Moore's law nor Dennard scaling are even the most critical scaling law to be concerned about...
...that's probably Koomey's law[3][5], which looks well on track to hold for the rest of our careers. But eventually as computing approaches the Landauer limit[4] it must asymptotically level off as well. Probably starting around year 2050. Then we'll need to actually start "doing more with less" and minimizing the number of computations done for specific tasks. That will begin a very very productive time for custom silicon that is very task-specialized and low-level algorithmic optimization.
[2] Shows that Moore's law (green line) is expected to start leveling off soon, but it has not yet slowed down. It also shows Koomey's law (orange line) holding indefinitely. Fun fact, if Koomey's law holds, we'll have exaflop power in <20W in about 20 years. Which should be enough for people to create ChatGPT-4 in their pocket.
0: https://www.rfcafe.com/references/electronics-mag/gordon-moo...
1: https://cdn3.weka-fachmedien.de/media_uploads/documents/1429...
2: (Slide 13) https://www.sec.gov/Archives/edgar/data/937966/0001193125212...
3: "The constant rate of doubling of the number of computations per joule of energy dissipated" https://en.wikipedia.org/wiki/Koomey%27s_law
4: "The thermodynamic limit for the minimum amount of energy theoretically necessary to perform an irreversible single-bit operation." https://en.wikipedia.org/wiki/Landauer%27s_principle
And my lecture notes from a class I took on semiconductor physics in college had a photo copy of a memo from Moore himself endorsing this broader conception of Moore's Law.
We’re seeing ChatGPT plug-ins for games, to provide intelligent conversation — and we’ve seen DNNs in StarCraft and similar.
To me, the “next gen” of gaming is intelligent NPCs, combining those features to create realistic behavior. That will require that consumer GPUs get closer to supercomputer GPUs:
More tensor cores and higher memory.
To clarify in case anyone else finds this confusing. The linked article suggests a 1.7x annual increase, which compounds to 2.89x every two years.
What happens if it loses a node or a link? Or some memory becomes unreliable? This thing needs some sophisticated fault tolerance.
Define "fail".
> I didn't have any insight on whether it was a hardware or software failure.
Have scripts check nvidia-smi for ECC errors and dmesg for devices dropping of the PCI bus.
For the former, replace the card. For the later, just perform a device reset (a power toggle of the device and a rescan of the bus is often enough to be back online within 5 seconds)
I contacted the CEO of Sega and suggested that they should find another partner. But I also needed Sega to pay us in full or Nivida would be out of business.
Just like failures, there are lessons for learning behind every success. Does anyone here have any insight - how did Nvidia came out of the above embarrassing and incompetency phase, and became a path-breaking, and trend-setting power house?
part 1: https://www.acquired.fm/episodes/nvidia-the-gpu-company-1993...
part 2: https://www.acquired.fm/episodes/nvidia-the-machine-learning...
In short, Jensen has an almost Elon-like appetite for "bet the company" tier risk. He's never been comfortable with a plateau, and is always looking for the next mountain to jump to, before the plate tectonics of the industry come around to form it.
There aren't a lot of CEOs and companies that oversee a company or industry-transforming shift more than once, he's definitely in that camp.
But that's a gross oversimplification, the story is more interesting. Check it out!
It seems Jensen is either reserved or media shy. He does not do as many public appearance as his contemporaries like Musk/Jobs/Gates/Bezos.
Also, there has not been any book on him. But, there are couple on the way - https://www.amazon.com/s?k=Jensen+Huang&ref=nb_sb_noss . I hope he'll choose to publish a 2k/3k pages biography. Love reading life stories of successful leaders.
The needs are more in line with consumer server hardware with user-choice on cpu/ram etc. Sounds to me like there's a market for disruption. Pity that the deep-learning community is under the choke-hold of nvidia's software.
1. Each node here is a more tightly-coupled CPU+GPU two-chip pairing, and the CPU side has a significantly larger pool of 480GB of LPDDR ("regular" RAM). So each GPU is part of a node that includes up to 480+96GB of total memory.
2. There are way more nodes: 256, up from 8.
I mean...nvidia has obviously been using DMA for decades. This isn't just DMA.
Is this memory unified like Apple Silicon? Meaning, can a model be deployed onto 574GB of total memory? Can the GPU read memory directly from the 480GB pool? Same question for CPU being able to directly access the 96GB.
Also, while the 1 exaFLOPS topline number is impressive, this has some asterisks. Each GH H100 GPU only does 34 TFLOPS of FP64 according to the data sheet. [1] At 256 nodes, this is a mere 8.7 petaFLOPS, or 0.0087 exaFLOPS. You only get to the 1 exaFLOPS number (from looking at the data sheet) if you are doing sparse FP8 Tensor Core FLOPS (3968 TFLOPS/GH, non-sparse is halved). It's worth keeping this in mind when comparing to something like Frontier (1.1 exaFLOPS) [2] or the upcoming El Capitan (expected 2 exaFLOPS) [3] - Top500 uses LINPACK, which benchmarks FP64 FLOPS. Of course, for AI training, FP8 or BF16 is probably the most relevant numbers for perf/W and perf/$... Frontier and El Capitan are each estimated to cost ~$600M, and while exact numbers weren't given, I'd expect a full 256-node DGX H200 to come in between $50-100M.
AMD will be having a "Data Center and AI" event on June 13th, so we'll get to see soon how competitive they are (the announced MI300 specs is 24 x Zen 4 cores with a CDNA3 architecture that is 8X faster than MI250X (383 FP16/BF16 TFLOPS), so about 3000 TFLOPS, with 128GB of unified HBM3 on a 8196 bit bus (6.5TB/s theoretical), which is in the same ballpark as Grace Hopper - I think it'll mostly come down to software, but with drop-in PyTorch support, OpenAI's Triton, etc, I'm somewhat optimistic that it will be worth it for big players do to some software lift (that others can benefit from), if the cost competitiveness of the hardware is there. For details already made public, see: https://www.tomshardware.com/news/new-amd-instinct-mi300-det...
[1] https://resources.nvidia.com/en-us-dgx-gh200/nvidia-grace-ho...
[2] https://en.wikipedia.org/wiki/Frontier_(supercomputer)
[3] https://en.wikipedia.org/wiki/El_Capitan_(supercomputer)
(But that number drops to only 10/s if the rewrites are in Rust)
Because of the slow toolchain, or because of the trademark lawsuits?
256 x 450W = 115KW = 82MWh = $82.000 / month at peak EU costs this winter (which will be normal next winter)
For what, something that gets everything wrong?
But correction to my comment above: These become one global memory... how slow it is and how many corruptions they will have is unknown but holy hell...
The end result of this announcement is another expensive system only available to the same incumbent of tech giants with tens of billions at their disposal.
Nope. I'm talking about efficient methods in training, inferencing and fine-tuning these AI models that doesn't require lots of data centers, TPUs, GPUs, etc. You're talking about something else.
Petrol and diesel cars are already burning the planet, but the main difference is, that there are efficient alternatives available today like electric cars to use instead.
AI (Deep learning) however, does not have any viable and efficient methods in training, fine-turning these AI models, at all [0] [1] and wastes a tremendous amount of resources, all to keep up with scalability.
So that problem is still NOT solved after a decade of using GPUs, the wastage is getting worse.
[0] https://gizmodo.com/chatgpt-ai-water-185000-gallons-training...
[1] https://www.independent.co.uk/tech/chatgpt-data-centre-water...
Exactly the types of problems future AI models could solve.
Dire climate alarms are based on the predictions made using models. As modelling advances as a field, both predictions, and solutions become more and more voluminous and accurate, along with revealing mistakes and failures of prior models.
Anyone concerned with climate should rally behind this kind of general progress. Further, it simply is progressing, and fields that don't embrace it, will be left behind. We're in the midst of an unprecedented revolution which touches all.
Which can also be archived by training more with the same amount of spent energy.
Why learn about training ("make training more efficient") on old hardware, which is more energy inefficient?
Add that towards scalability and you will realize that training AI models scales terribly with more data as it is very energy and time inefficient. Even if you replace all the hardware in the data centers it still wouldn't reduce the emissions regardless and replacing them also costs at most billions either way. That is my the entire point.
So that does nothing to solve the issue. Only ignores and prolongs it.
I mean, that's the root of scaling as a principle, right?
You could viably start training an AI on your cell phone. It would be completely useless, lack meaningful parameter saturation and take months to reach an inferencing checkpoint, but you could do it. Nvidia is offering a similar system to people, but at a scale that doesn't suck like a cellphone does. Businesses can then choose how much power they need on-site, or rent it from a cloud provider.
If a product like this convinces some customers to ditch older and less efficient training silicon, I don't see how it's any more antagonistic than other CPU designers with perennial product updates.
Certainly wasn't the case when a public research university partnership seeded by a generous donation from Nvidia co-founder/UF alumnus Chris Malachowsky was formally announced[1] shortly after DGX A100 launch[2] several years ago, never mind the handful of other academic early adopters mentioned in the press release.
Of course, we tend to conveniently forget such exogenous details.
[1] https://news.ufl.edu/2020/07/nvidia-partnership/
[2] https://nvidianews.nvidia.com/news/nvidias-new-ampere-data-c...