Hacker News Reader

Comment by insane_dreamer | Hacker News Reader

Not sure that would have mattered once LLMs emerged. NVidia's success is mostly a matter of being the only major advanced GPU player at the time when LLMs exploded onto the scene. That won't be the case for long. The big players are already heavily investing in high performance AI chips to wean themselves off a dependance on NVidia.

View on HN

Nvidia’s success is due to a decade+ of investment into their software stack (CUDA and friends), built on a world view that more software can and should be massively parallel.

They were a key cause for LLMs being a thing in the first place.

This. I worked in HPC between 2010 and 2013, and people were trying to compete with NVidia in the GPGPU space even back then, seeing the way the wind was blowing. Compute bandwidth, and FLOPS/watt has steadily been more and more important the last 15 years, even before AI.

NVidia has continued to stay ahead because every alternative to CUDA is half baked trash, even when the silicone make sense. As a tech company, trading $$ for time not spent dealing with compatibility bugs and broken drivers pretty much always makes business sense.

> half baked trash

> dealing with compatibility bugs

> broken drivers

Describes my experience trying to use CUDA perfectly.

We have a long way to go and we haven't even started yet.

Given that experience, think about what state the alternatives must be in!

To be fair, Cuda has improved a lot since 2014 or so. I messed up my Linux box multiple times trying to install Cuda but the last time it was just apt install and maybe setting ld library and it all just worked.

It’s pretty bad. just when you think you can order AMD chips since there is no shortage, and use a translation layer and have a cheap AI datacenter, it turns out AMD is fumbling the ball at every step of the way

It’s interesting. They have had plenty of time and resources available to mount solid competition. Why haven’t they? Is it a talent hiring problems or some more fundamental problem with their engineering processes? The writing has been on the wall for gpgpu for more than 10 years. Definitely enough time to catch up.

Its a commitment problem IMO. NVidia stuck with CUDA for a long time before it started paying them what it cost. AMD and Intel have both launched and killed initiatives a couple times each, but abandon them within a few years because adoption didn't happen overnight.

If you need people to abandon an ecosystem thats been developed steadily over nearly 20 years for your shiny new thing in order to keep it around, you'll never compete.

Have you tried Vulkan Compute or OpenCL? I recommend evaluating this relatively, vice compared to an ideal.

Wasn't Intel's commercial compiler also dominated HPC space around that time period? I remember back then people would use cuda and icc for hpc stuff.

I believe their compilers still dominate in CPU-bound workloads.

I can't speak to modern hpc, since I've been out of the game for a few years but icc was absolutely the preferred compiler for almost any workload and Intel procs were the desired hardware to run on. AMD was joked about as the only reason Intel wasn't considered a monopoly at the time and that Intel would happily take wheelbarrows of cash over to AMD to prop them up, just to ensure they didn't even appear like a monopoly. But no one in the field was buying AMD procs for their workloads.

Most super computers were still Intel/IBM, but around that time is when the shift to GPU clusters started. #1 super computer spot was taken by an Nvidia cluster in 2012, but I remember other big projects were done before as well.

Nvidia did a lot of things right. But it sure didn't hurt that the crypto bubble gave way to LLMs with almost perfect timing. Luck favors the prepared and all that but Nvidia also would have had trouble timing market changes better.

Yeah, exactly, it has never been clear that we would find a use for parallel computation outside of niches like graphics and HPC. And in fact most of the history of computing has been evidence of the opposite: useful computation has been predominantly serial in nature. But here we are now.

Nvidia did ride a perfect storm of gaming + bitcoin + AI.

But AMD and others could have done the same, had they been better at riding that wave. There’s a reason it was Nvidia who won out.

More than a decade.

I was using it as soon as it came out in 2007 and my dinky desktop workstation was out performing the main frame in the basement.

> a world view that more software can and should be massively parallel.

Twenty years ago I was thinking we'd be speccing machines in kilocores by now.

https://en.wikipedia.org/wiki/Connection_Machine

128 and 512 cores are normal now, so not that far off.

4090 has 16384 cuda cores, so there's that!

Nvidia's marketing is misleading. Those "cuda cores" are more SIMD lanes than cores. Number of SMs is a more appropriate equivalent to CPU core count.

you're right

Are you sure that isn’t what @mullingitover meant?

> Number of SMs is a more appropriate equivalent to CPU core count.

What do you mean by this? Why should an SM be considered equivalent to a CPU core? An SM can do 128 simultaneous adds and/or multiplies in a single cycle, where a CPU core can do, what, 2 or maybe 4? Obviously depends on the CPU / core / hyperthreading / # math pipelines / etc., but the SM to CPU-core ratio of the number of simultaneous calculation is in the double digits. It’s a tradeoff where the GPU has some restrictions in return for being able to do many multiples more at the same time.

If you consider an SM and a CPU equivalent, then the SM’s perf can exceed the CPU core by ~2 orders of magnitude — is that the comparison you want? If you consider a GPU thread lane and a CPU thread lane equivalent, the the GPU thread lane is slower and more restricted. Neither comparison is apples to apples, CPUs and GPUs are made for different workloads, but arguing that an SM is equivalent to a CPU core seems equally or more “misleading” when you’re leaving out the tradeoff.

I’d argue that comparing SMs to cores is misleading and that it makes more sense to compare chips is by their thread counts. Or, don’t compare cores at all and just look at the performance in, say, FLOPS.

A single Zen5 core can do 32 single precision FMAs per clock.

That's using SIMD, but so is Nvidia for all intents and purposes. Those "cuda cores" aren't truly independent: when their execution diverges, masking is used pretty much like you'd do in CPU SIMD.

A lot of the control logic is per-SM or perhaps per-SIMD unit -- there are multiple of those per SM. You could perhaps make a case that it's the individual SIMDs which correspond to CPU cores (that makes the flops line up even more closely). It depends on what the goal of the comparison is.

> An SM can do 128 simultaneous adds and/or multiplies in a single cycle, where a CPU core can do, what, 2 or maybe 4?

https://images.nvidia.com/aem-dam/Solutions/geforce/news/rtx...

An SM is split into four identical blocks, and I would say each block is roughly equivalent to a CPU core. It has a scheduler, registers, 32 ALUs or FPUs, and some other stuff.

A CPU core with two AVX-512 units can do several integer operations plus 32 single-precision operations (including FMA) per cycle. Not 2 or 4. An older CPU with 2-3 AVX2 units could fall slightly behind, but it's pretty close.

That doesn't factor in the tensor units, but they're less general purpose, and CPUs usually put such things outside the cores.

I would say an SM is roughly equivalent to four CPU cores.

Yeah I totally forgot to consider CPU SIMD. Brain fart. The other comment corrected me too, you’re both right. When I said 2 or 4, I was thinking of the SISD math pipe and not AVX instructions.

Yes, considering CPU SIMD, maybe comparing a CPU core to a CUDA warp makes some sense in some situations. The peak FLOPS rate is still so much higher on Nvidia though, that the comparison hardly makes sense. So yeah like I and the other commenter mentioned, it depends entirely on what comparison is being made.

100%. If you ever tried to build pytorch (or tensorflow) gpu accellerated using NVidia vs AMD it was absolutely chalk and cheese for the longest time.

Nvidia/CUDA process: Download package. Run the build. It works. Run your thing- it's GPU accelerated. Go get a beer/coffee/whatever while your net runs.

AMD process: Download package. Run the build. Debug failure. Read lots of articles about which patches to apply. Apply the patches. Run the build. It fails again. Shit. OK ok now I know what to do. I need a special fork of the package. go get that. Find it doesn't actually have the same API that the latest pytorch/tf relies on. OK downgrade those to an earlier version. OK now we're good. Run the build again. Aw shit. That failed again. More web searches. Oh ok now I know - there's some other patches you need to apply to this branch. OK cool. Now it compiles. Install the package. Run the thing. Huh. That's weird. GPU accelleration isn't on.... sigh....

I think this undersells the story. NVidia's success is built on innovating and scaling for 20 years. Vastly oversimplifying it:

- CUDA conception in 2006 to build super computers for scientific computing.

- CUDA influencing CPU designs to be dual purpose, with major distinction of RAM amounts (for scientific compute you need a lot more RAM compared to gaming)

- Crypto craze driving extreme consumer GPU demand which enabled them to invest heavily into RND and scale up production.

- AI workload explosion arriving right as the crypto demand was dying down.

- Consistently great execution, or at least not making any major blunders, during all of the above.

It’s more no major blunders + no real major competition, they have not consistently “executed great”, it’s dumb luck + no extremely stupid decisions.

It doesn’t mean they didn’t make a bunch of mistakes its that when they did there was no competition to to realistically turn towards, and they fixed a lot of their mistakes.

As someone who's intimately familiar with both cultures, I'm convinced Intel would have killed any innovation Nvidia had going for it before it had a chance to really take off.

Management at the two could not be more opposite if they tried.

Intels 3rd wave management by MBA would have ruined nvidia way back then forcing them into something dumb and blocked all of the R&D they did since since that's what they also did at intel.

Other companies are capable of making big GPUs; they aren't the only TSMC customer. Intel themselves have perfectly fine GPUs. Their issue is that their management never allocates enough space in their chips to let them perform well.

Nvidia's advantage is that they have by far the most complete programming ecosystem for them. (Also honestly… they're a meme stock.)

Other companies don't have CUDA.

This seems to have cause and effect backwards. LLMs emerged because of Nvidia.

LLMs absolutely did not emerge because of NVIDIA. you’re the one imho who is mistaking correlation with causality.

The first transformer models were developed at Google. NVIDIA were the card du jour for accelerating it in the years since and have contributed research too, but your statement goes way too far

I think their point is that easily accessible hardware capable of supporting the research is the reason the research has come as far as it has, and I would tend to agree. At the very least, GPUs keeping the PCI ecosystem going has played a major role in allowing the specialized accelerator market to flourish.

But that could apply to any of the GPU manufacturers. CUDA made for an easier ecosystem but if it didn’t exist it would have been any of the other APIs.

The first transformer models didn’t even use CUDA and CUDA didn’t have mass ecosystem inroads till years later.

I’m not trying to downplay NVIDIA but they specifically mentioned cause and effect, and then said it was because of NVIDIA.

First transformer != LLM. Imagine a world where you had to use AMD or CPUs, there would be no AlexNet, there would be no LLMs. Nvidia seeded universities with gifted hardware accelerators for a over a decade. Nvidia built the foundations for modern ML on which transformer lies, it's just one step on a long road to LLMs.

AMD had GPU compute APIs at the time as well. They also used to contribute GPUs (albeit in much smaller quantities). They just ended up killing them in favor of OpenCL which then withered on the vine.

NVIDIA absolutely contributed to the foundation but they are not the foundation alone.

Alexnet was great research but they could have done the same on other vendors at the time too. The hardware didn’t exist in a vacuum.

Alexnet could not have been done on AMD, nope.

Please, this is just revisionism. There’s nothing inherent to AlexNet that relied on NVIDIA hardware or even software. It was just what happened to be available and most straightforward at the time.

To say it wouldn’t have been possible on AMD is ludicrous and there is a pattern to your comments where you dismiss any other companies efforts or capabilities, but are quite happy to lay all the laurels on NVIDIA.

The reality is that multiple companies and individuals got us to where we are, and multiple products could have done the same. That's not to take away from NVIDIA's success, it's well earned, but if you took them out of the equation, there's nothing that would have prevented the tech existing.

The crux of the issue is this:

> It was just what happened to be available and most straightforward at the time

AMD made better hardware for a while and people wanted OpenCL to succeed. The reason why nvidia became dominant was because their competitors simply weren’t good enough for general purpose parallel compute.

Would AI still have happened without CUDA? Almost certainly. However nvidia still had a massive role in shaping what it looks like today.

Many people here are heavily invested in NVDA stock, which certainly clouds the judgement.

That’s my take away too, even though I’m invested myself. The discussion around them has transcended into mythology. Imho part of it is because GPUs themselves are so foreign to many people as a development target and so they’re not really as aware of the landscape

OpenCL itself is kinda fine, but the libraries never existed because every academic from postdoc up could get a free NVidia card. You literally filled a form out and they sent you it.

I agree the library situation for OpenCL never stood up. Khronos tried at the start but then lost steam.

It’s one of the projects I think the Khronos group mishandled the most unfortunately.

Do they have a project that achieved success?

OpenGL, Vulkan, WebGL?

Here’s a list of projects

https://www.khronos.org/

If I were to categorize the successful ones:

glTF, KTX, SPIR-V, OpenGL and its variants, WebGL

People will say Vulkan but it has the same level of adoption as OpenCL, and has the same issue that it competes against vendor specific APIs (DX and Metal) that are just better to use. It’s still used though of course as a translation target but imho that doesn’t qualify it as a success.

OpenCL was and is a failure of grand magnitude. As was colada.

> The first transformer models didn’t even use CUDA and CUDA didn’t have mass ecosystem inroads till years later.

I graduated college in 2010 and I took a class taught in CUDA before graduating. CUDA was a primary driver of NN research at the time. Sure, other tools were available, but CUDA allowed people to build and distribute actually useful software which further encouraged the space.

Could things have happened without it? Yeah, for sure, but it would have taken a good deal longer.

I was around the space before Alexnet came out. Without NVidia the last 15 years of AI would not have happened.

Deep learning was only possible because you could do it on NVidia cards with Cuda without having to use the big machines in the basement.

Trying to convince anyone that neural networks could be useful in 2009 was impossible - I got a grant declined and my PhD supervisor told me to drop the useless tech and focus on something better like support vector machines.

I was around then too and people forget that AMD also had GPU compute APIs.

The difference is AMD killed theirs to use OpenCL and NVIDIA kept CUDA around as well.

Just like how ROCm is supposed to be competitive today and it isn't unless you have an army of grad students to look after your data center cards.

I tried using AMD Stream and it lacked documentation, debugging information and most of the tools needed to get anything done without a large team of experts. NVidia by comparison could - and was - used by single grad students on their franken-stations which we build out of gaming GPUs.

The less we talk about the disaster that the move to opencl was the better.

I agree ROCm was a mess and only recently is even really usable.

That said ROCm is quite a recent thing borne out of acknowledging that OpenCL 2 was a disaster.

OpenCL 1 had a reasonable shot and was gaining adoption but 2 scuppered it.

Did you ever train anything past mnist on AMD? I try every five years or so and promise myself never again each and every time.

Not with Rocm since I’ve moved my personal stack to NVIDIA (for rendering) and Macs for day to day use.

I did write quite a bit of OpenCL prior to that on Intel/AMd/NVIDIA, both for training and for general rendering though, and did some work with Stream before then.

Was it OpenCL1? That's the only one I hadn't tried out for AMD GPUs. Everything else I have and can say with absolute certainty that you spend more time fighting the hardware than you did writing code.

Cuda by comparison JustWorks^tm.

Both 1 and 2. I haven’t done much with 3 as OpenCL is effectively a dead api at this point.

1 was definitely a lot easier to work with than 2. CUDA is easier than both but I don’t think I hit anything I could do in CUDA that I couldn’t do in OpenCL, though CUDA of course had a larger ecosystem of existing libraries.

Damn, I missed the best one.

Thanks for the trip down memory lane.

I was not a PhD, but I studied both around that period in university and can confirm that neural networks were only seen as a kind of theoretical plaything. All the libraries were pushing SVMs.

Surely intel would have killed off CUDA

You are completely ignoring the impacts of Covid and GPU cryptocurrency mining on Nvidia profits.

It could have been AMD/ATI profiting from such random events, as they were the ones that financed AI development.

BTW, when GPUs were used for Bitcoin mining (up until year 2013, obsoleted by spacialized chips after that), AMD chips were used exclusively, because they had much better integer math performance, compared to Nvidia cards, which focused on floating point performance.

Nvidia will make general purpose GPUs while the AI players will make ASICS. I guess most data center will prefer GPUs so customers can run whatever AI model they need.

Nvidia was a $750B company in 2021 (pre-ChatGPT). Cause of gaming and all of pre-LLM machine learning using nvidia gpus.

And when the LLM hype dies NVDA will be the hardest to fall.

Reply on news.ycombinator.com