Nvidia Unveils Blackwell, Its Next GPU
spectrum.ieee.org
spectrum.ieee.org
I have the feeling that GPUs are on the sweet spot where smaller footprint directly translates into more executions and, therefore, higher throughput for the same chip area.
https://www.nvidia.com/en-us/data-center/products/ai-enterpr...
It is this kind of delivery that the competition misses out.
The networking alone is a huge bottleneck at scale. A competitor has to be better at networking AND chips to be competitive.
NVIDIA got big because CUDA works on the most crappy notebook GPUs up to their most powerful chips, and AMD should do the same, but focusing their limited number of driver devs on the expensive enterprise hardware makes sense IMHO.
AI/ML is a rapidly moving field etc and you know geohot is gonna leak it all on twitter as soon as there's anything to announce, which makes it far more difficult for them to pivot later, etc.
Agree and yet none of the contenders were able to work out their software play (Intel, AMD, chip startups) for more than a year which shows how corporates move slow.
Google is not selling their TPUs AFAIK and their tooling is completely focused on internal use.
So really interesting to see no one else is properly addressing the need even though they have chips (and the chip itself is much simpler than a cpu, a systolic matrix multiplier array).
https://stability.ai/news/putting-the-ai-supercomputer-to-wo...
I’m sure this was a lot of work, and Intel surely helped a lot, and there are probably plenty of kludges involved. But it worked, and there’s a lot of money on the table to do things like this.
CUDA will do whatever you want and it more-or-less just works. ROCm (after > six years) is still:
- Won't work on your hardware
- Used to work on your hardware but we removed support within a few years
- Burn 10x more time trying to get something to work
- Be perpetually behind CUDA in terms of what you want/need to do
- Sorry, that just won't work
- Performance is lower than it should be for what is often actually better hardware, to the point where a superior newer generation AMD GPU gets bested by a previous generation Nvidia GPU with inferior (on paper) hardware specs
I've been trying ROCm since it was initially released > six years ago. I want AMD to succeed - I've purchased every new generation of AMD GPU in these six years to evaluate the suitability of AMD/ROCm for my workloads. Once a quarter or so I check back in to evaluate ROCm.
Every. Single. Time. I come away laughing/shaking my head at how abysmal it is. Then I go back to CUDA and sit in wonder at how well it actually works and throw even more money at Nvidia because I just need get things done and my concerns about their monopoly, artificial market segmentation, ridiculously high margins, etc are a distant second to my livelihood.
AMD (and others) need to understand what Jensen Huang has been saying for years - 30% of their development spend is on software. As the announcements this week show, Nvidia is using their greater and greater financial resources and market share to continue to lap AMD in the only thing people actually care about: here's our product and here's what you can actually do with it.
Many people with a fundamental hate/disgust for Nvidia will come back and say "ok bootlicker, it's supported in torch you're spreading FUD". Ok, take a look at the Nvidia platform you linked and show me where the ROCm equivalent is. Take a look at inference serving platforms which are one of the things I care most about. Look at flash attention, alibi, and the countless other software components that you actually need beyond torch in many cases. Watch even basic torch crash all over the place with ROCm.
Sure, you /might/ be able to train or run local one-off inference with AMD. How do I actually run this thing for my users? Crickets -or- maybe vLLM support for ROCm for LLMs (nothing for other models). Then dig just a little bit deeper and realize even vLLM isn't feature complete, requires patches, specific versions all around, and from personal experience a lot of github/blog spelunking and pain. With CUDA it's `docker run` and flies.
With CUDA I can run torchserve, HF TGI, vLLM, Triton, and a number of others to actually serve models up for users so I can make money from my work. ROCm, meanwhile, can barely run local experiments.
AMD needs to get it together.
Reminds me of how a mic is a speaker and speaker is a mic.
Sony copied that idea for the 1st Playstation, and then folks like NVidia & 3DLabs quickly followed suit, the idea being they would enable that functionality for games like Final Fantasy.
In the early 2000s, the HPC folks realized that you could use a GPU for physics & engineering codes, and here were are 20 yrs later.
I think it's pretty obvious that they saw the writing on the wall about the value of GPUs for massively parallel computing and actively pushed to be in the right spot for when everything took off.
Their release cadence only really picked up in 1998, and just 2-3 years later they had the first 'programmable' GPUs with early shaders.
As such, saying that NVIDIA wasn't always aiming for GPU compute feels kinda like saying that SpaceX wasn't always aiming for feasibly reusable rocketry because it took them ~15 years to build up to their current version of it.
I'm not surprised here. They invested and invested and invested and now everyone is playing catch up while they nearly corner the market. In industry circles we've been lamenting their dominance in this area for well over a decade.
Google and Apple are the only companies I see that have a shot at disrupting their market but they tend to end up navel gazing instead of burning bridges. And Nvidia has bridges with everyone.
It's absolutely an industrial geopolitics mastercraft scenario. We'll be studying them for a century.
Everyone has long known that the secret to unlocking the next tier of performance is solving concurrency. Look at all the advances that have made it into production in the last twenty years. But threading models only get you so far on a CPU, and programming GPUs is comparatively very difficult. Nvidia has always aimed to displace the CPU's prominence, since day one.
It's horizontal scaling 101.
So I'm not buying it.
Between the first nvidia cards and the first release of CUDA, you have a decade.
Or get the slightly underpowered Banshee.
Well, there is PhysX which they bought in 2008 and still have; when they bought PhysX they shifted the acceleration implementation from relying on dedicated physics accelerator hardware to GPGPU.
There’s APEX which was built on top of PhysX but later discontinued.
There’s FleX…
“At one point” might be a slight misstatement.
The idea that future of computing is related to parallel numerical computing and linear algebra (HPC, graphics ,scientific computing, data science, machine learning) predates current deep learning boom.
Already in 2003 and GPGPU (General-purpose computing on graphics processing units) was a thing. OpenVidia came out in 2003.
CUDA was first released in ~2006 but started at least in 2004, largely building on top or the momentum of Cg which was released in 2002 and being worked on since 2000. There was at the time in the late 90s about general purpose parallel programming and how it could possible be done on GPUs or doing it on something like PS2 Cell.
I dont know if Nvidia was really started with CUDA in mind in 1993. But Nvidia was into CUDA like GPU usage WAY ahead of anyone else in the field.
GeForce 8 introduced CUDA because graphics pipeline similarly evolved into position where instead of forcing the use of dedicated cores for specific tasks (vector and fragment shaders) due to simple economies of chip making, the Shader Model 4 introducing yet another stage justified making unified shader cores... which could be just as easily used for other tasks.
Yeah, sure.
But because these graphs are mostly dense and involve numerical operations, matrices/tensors are a great implementation.
oh, sorry, I thought we were just asking questions
It turns out that anything related to neural networks, and similar AI approaches, is all about compute.
GPUs, as they became programmable (and maybe even a little bit before that...), started to take cues from SIMD Supercomputers. So the compute methodologies were researched first, and then GPUs (ie: applications to graphics) were applied afterwards.
I've heard rumors that the first programmable GPUs were considered because GPUs already were in SIMD-style compute and running instructions in a programmable way at the hardware/firmware level. It just needed to be "revealed" to OpenGL or DirectX programmers.
--------
You're right in that 80s / Arcade chips look to be different. Its a programmable graphics chip alright, but I wouldn't call that chip SIMD, not from what I can see from Wikipedia at least.
GPUs always had more compute / Gigaflops than traditional computers. GPUs in fact have more to do with 80s-era supercomputer architecture than normal CPUs.
Generative Processing Unit works for all cases.
Both aviation and computers use them, though the latter more often call them UPS (Uninterruptible Power Supply).
You could call them gPUs.
I seek all kinds of answers, including ones about fundamental logic, mathematical physics, etc
The most impressive thing about modern computing is that we've had exponential increase in compute speed, yet everything runs as slow as it did 30 years ago
But fundamentally technology gets better when we can do more with less.
When it has to draw that much current it limits the contexts in which it can be beneficial.
Nvidia went all-in on infiniband serDES while AMD chose pcie/CXL. But since Pcie signaling requirements are tighter, you need bigger stronger PHYs, which means you get less actual area per beachfront. The penalty is latency/power, but who cares when gpus are latency-hiding machines anyway?
https://www.semianalysis.com/p/cxl-is-dead-in-the-ai-era
https://www.semianalysis.com/nvidia-b100-b200-gb200-cogs-pri...
this in turn means that nvidia can implement more links or bigger links in their nvswitch networks, which means they can construct bigger systems and push the TCO down.
Two 7900X is still functionally a 7900X, but two 3090s is functionally a 48GB card. Nvidia has got the interconnect bandwidth to a point where it’s a significant enough fraction of the local bandwidth to be functionally one single gpu - this is the same argument as MI300X etc. Doesn’t matter whether the link is on-package or off-package, what matters is that it’s a significant fraction of the speed of your local memory or cache ports. Nvidia did that, with large numbers of gpus, not just a pair of chiplets.
Nvidia has been thinking about this one for a long time - nvswitch is on its third generation, and can switch literal terabytes of data per switch, times several switches. The Mellanox purchase too, but it goes back way longer.
And unlike AMD they actually have a driver that works and just trivially exposes these capabilities and gets out of the way. If you want to tinker and build the open alternative that’s fine, other people want to work.
This is shocking to many AMD fanboys but actually Jensen is a good engineer too, nvidia is mostly on top because they sell products that people want (to such a relentless degree they get furious if they don’t get faster every year etc) and cannot be trivially displaced by “just as good” Radeon drivers etc - just see the latest installment of the geohot saga. Nobody is trapped by nvidia, it is a golden cage - getting actual work done or just going and playing a game instead of spending hours playing with regedit hacks to disable dxnavi to fix DX11 shader compilation stutter is what you’re buying.
https://twitter.com/__tinygrad__/status/1770160392389771305
https://old.reddit.com/search/?q=Dxnavi+stutter+&include_ove...
Nvidia is on top because of relentlessly competent engineering and savant-level business direction, and as much as people scoff at the idea… that’s literally the reason you hate him lol. He is a Jobs-like visionary figure that can see what the tech can be and drive the engineering and business factors to align along the long-term to get him where he wants to go, while also providing the funding and profit in the short term.
https://m.youtube.com/watch?v=Xn1EsFe7snQ&t=1034
The only company with comparable parasocial negative attachment is apple and it’s for the exact same underlying reason . People are also systematically unable to understand that apple users are not “trapped” or in need of rescuing either. People buy apple because it does what they want it to really well, and they don’t care about installing Linux on their phones. And nerds resent that deeply. It’s not a coincidence there’s this axis of warfare around both Nvidia and the App Store with the EU etc. Nerds cannot abide someone choosing the “wrong” hardware. They are right and you will buy the same thing as them or they will get the EU to outlaw your product, or change the symbol licensing to prevent you running on Linux, etc. If you don't like the same filesystem as me, obviously that means I get to relicense some symbols that have been there for 20+ years and break your filesystem. Btrfs is better, the council has spoken.
It keeps happening for a reason, folks, lol. Nerds can’t tolerate others making different choices. And those users disproportionately self-select to “nerd” platforms like android and AMD.
Just with current AI models, the amount of value that is waiting to be created (take technology X, add AI to it) is incredible. Casual things that used to take years for a team to build, can now be solved by throwing a GPU at it with a generic model that is fine-tuned a bit. Basically, things that were unpractical 2y ago are now on the table.
The bear thesis is that compute will stop being scarce. Which is plausible, since in capitalism, the best cure for high prices tends to be high prices.
Something crazy to think about is that Accelerando by Charles Stross is starting to look like a prophecy being slowly fulfilled.
"That AI isn't all that that and won't make much money" seems to be by far the biggest one. So far the applications are impressive and a little scary, but not actually something that anyone is going to pay for. Apple makes a zillion dollars because people want its phones. Google makes a zillion dollars because people want to sell junk to folks on the internet.
You need to posit a product built out of compute that does more. Maybe replaces a bunch of existing workers in an existing industry, something like that. So far the market is still looking.
Although, even if that is the ultimate result, there's still going to be plenty of money changing hands on the way to that conclusion. See also all the blockchain/web3 companies, when there was (to me at least) clearly a lot less substance/potential there.
For example, being able to feed a potential customer's invoice into GPT and ask it to see what kind of services we can offer to beat its price. Our sales people had to spend hours doing this before. Now it's done in 2 minutes through an engineered prompt. And it's incredibly accurate.
The problem with GPT4 API is context size and price. That's it. Both are bottlenecked by faster and cheaper compute.
That's my bull case for more compute, not bear case like OP asked.
Also bearish on programmers keeping up on their fundamental algorithms, rather than trying to throw NNs at every problem.
Besides interviews, at least 90% of developers have no need for 'fundamental algorithms'. The library being used uses them in some way sure, but the vast majority of devs simply need to know how to use the tool, not how the tool itself is developed.
Thinking back to a recent-ish discussion here about the (very elegant and efficient) algorithm behind Shazam, and some of the comments here being along the lines of "haha, that's so quaint, nowadays you could just use a neural net". Nevermind how that would even work as well.
Or programmers excited about neural networks forgoing the much simpler, and in many cases completely sufficient, computer vision algorithms built into OpenCV, in favor of trying to train their own model from scratch.
And the other smaller problem was using a hash array instead of a hash map. You don't really need a mental model of multiple algorithms to know o(1) is faster than o(n).
The whole thing is more of a sign of how poor coding practices must have been at Rockstar for such a bug to not only make it in, but persist for years.
But once we understand a little bit about the problem we can model 80-90% of its behavior with a handful of parameters. Add in some bias and noise parameters and you have an accurate trainable machine learning model that’s orders of magnitude more efficient.
Take for example a spring which can be modeled by 1 or 2 parameters. But its impulse response looks like a sin curve multiple by exponential decay.
If you just train neurons to match input/outputs from a spring you need a ridiculous number of model parameters to describe that shape.
CNNS have seen an enormous amount of success due to this fact: a lot of processes can be modeled by convolution.
1) AGI won't happen because we are on the wrong path
2) AI being a big part of our lives is still a theory. Aswath Damodaran has some brief thoughts on this.
But the biggest bear case has to be that the technology won't get better. Essentially, everyone assumes that it will without reservations.
Nvidia is ceding the low-end GPU market to anyone who wants it. Not only does it allow a competitor to establish a reliable source of revenue for their R&D department, but it could cut off the sale of the binned chips that are inevitably produced on the expensive, tiny processes that Nvidia uses - which would hurt their margins to some degree.
bro there is like $10 of margin in your idea for a $200 gpu lol, nobody is "ceding" anything (actually 4060 is a more advanced card than 7600 on literally every front, for ~10% more money) but the cost floor has climbed to the point where $200-300 gpus just don't progress that much anymore.
There's very good reasons for this - shrinks are the least effective on low-tier cards (because memory controllers don't shrink), and you simply don't gain much actual savings from shrinking a 200mm2 die - congrats it's 150mm2 now, on a more expensive node, meaning your $10 chip is now $9. And meanwhile gamers want more VRAM every year, manufacturing and testing and shipping costs have gone up (and cost the same for a 4090 as a 4060), etc. The economics of low-end cards is literally terrible and they are simply falling off the edge of profitability.
Intel is willing to lose money hand-over-fist just to get into the market, but AMD and NVIDIA are pretty much charging fair-ish prices, and gamers just are too emotionally immature to accept that moore's law really really actually is dead for realsies and things aren't going to progress 40% perf/$ per gen anymore.
It's so weird, nobody cries about the CPU market like this. A 1600AF went for $85, a 3600 went for $160, nobody said "boo" when the 5600X increased that to $330 or whatever. Nowadays you are spending at least 50% more on your CPU than you did 5 years ago, sometimes closer to 2x. The enthusiast market is buying $250-400 cpus now, not $85-160. And obviously everyone understands that upgrading your CPU every gen is terrible value too, especially when prices have drifted upwards. But they don't have a half-decade of negging from reviewers telling them that this is a market in crisis, and that they should feel bad about buying a CPU, etc.
The literal half-decade of warfare from reviewers against the GPU market is so tired at this point. Bro, things are going to slow down, it just is how it is. GPUs are the processor that's most dependent on moore's law providing growth in transistors at the same cost, and wafer price increases hit them the hardest. Go complain to TSMC instead, or ASML, or the brick wall - it's ultimately a physics problem. But there's a hell of a lot of clicks and youtube ad money to be made whining about it in the meantime.
At least reviewers are finally coming to jesus on DLSS - mostly because they know AMD will finally have a decent upscaler within a year tops, and that RDNA4/5 will be pushing forward on tensor etc. The writing was on the wall as soon the specs leaked for PS5 Pro, which is basically adopting RTX features wholesale. https://www.youtube.com/watch?v=CbJYtixMUgI https://www.youtube.com/watch?v=BG-7vyw2YRg&t=1625s
a) Humans at some point between here and eternity become more efficient easier, than scaling compute is hard. Seems unlikely.
b) Compute is overrated, now or will be in the near future. I will be happy to donate to the church of "compute is overrated" if that makes people get off of gpt-4+ and let me cook. Read that as: I doubt it.
I don't see a c)
There's still the secondary question of how compute heavy it will be, and I don't think anyone knows. But Sam Altman, in a recent speech he gave in Korea, expressed confidence that there isn't a limit in sight for returns from GPT scaling.
Nvidia GPUs are pretty flexible in terms of computation and extremely power hungry. It may be that a next generation of more specialized hardware, such as TPUs or something, outperforms Nvidia GPUs on machine learning tasks to such an extent that those GPUs are obsolete for those tasks. This next generation could come to market sooner than Nvidia anticipates.
Another possibility is that ML researchers figure out some ways to radically reduce the amount of compute required for good training and inference on _less_ specialized hardware. It's really impressive what you can do with llama.cpp. If open source models running on consumer grade hardware ever get to 90% as good as ChatGPT (which, to be clear, is absolutely not the case currently), then those top end GPUs are overkill for most use cases.
I don't think either of those scenarios is particularly likely, but they're at least plausible.
just like the creation of radically simpler internal combustion engines led to us spending a lot less on internal combustion engines, right? /s
(seriously though, don't call it a "GPU" when rendering takes the back seat)
I'm curious if the technology is now vastly out preforming the demand here or if the demand for compute is outpacing the tech.
"up to 30 times the inference performance, and up to 25 times better energy efficiency"
Quite interesting.
It will be interesting when chips such as this percolate down to single folks using one of these to just run their home AI node.
When every building has just one of these in their core building AI system that allows for all the regular talk to your smart home and have it intelligently accommodate your inferred needs/intentions.
Its possible today - but I mean on a wide scale.
Instead, I expect those buildings to invest in having the fastest, most stable internet connection with added redundancy and everything will be fully centralized in datacenters.
EDIT: the 10-year is just a random number.
And while these chips will obviously eventually be obsoleted as far as their cutting-edge - that doesnt remove the usefulness of such a chip for as long it can logic electrons.
So even by any future standard - as few of these thrown in more a consumer space will be adding of value for a long time, one would think. I'd be really interested in knowing about fab re-tooling.
I recall one night at Intel in 1997 or so - I stepped onto the balcony to have a smoke at about 1am.
There was another guy there who was in finance and we had a similar smoke schedule - and we would chat. He was lamenting about the difficulty in his work was to re-work a lot of DB schemas because some of the numbers he had to entire as far as finances, were too large for the fields.
I never forgot that - or a common trope that was thrown around intel at the time - I was in my early 20s and was focused on video games when at Intel - so I didnt get to follow along with this: "Its cheaper to just build a new fab than to retool one for the next iteration of processor"
-- so I wonder just how much ancillary waste happens these days with each new chip iteration/evolution. Meaning - all the amounts of resources that went into the fab to make such a chip - and where these wind up in their lifecycle.
The newest machines to make the ultra EUV chips are ~$380 million a piece (which is nothing these days on a scale - but on a unit basis, thats a Fton.
https://www.cnbc.com/2022/03/23/inside-asml-the-company-adva...
I'm sure there's a few people that have e.g. an intel itanium from 10 years ago in their home lab, but those don't hold a candle to current-day consumer grade CPUs.
Don't tell me it's because of familiarity with the word GPU. Nvidia could coin a new acronym and write a PR release and the entire world would circulate it and even discuss it in here and every other vendor would scramble to play catch-up.
It's because of familiarity with the word GPU...
Because names are sticky, and no one wants to start evangelizing a new term (“Matrix Math Processing Unit”) for it, preferring to put energy into things with value.