Nvidia reveals new A.I. chip, says costs of running LLMs will drop significantly | Hacker News Reader

Nvidia reveals new A.I. chip, says costs of running LLMs will drop significantly

cnbc.com

73 pointsTalktoCrystal13 comments

The GH200 was announced in May. What’s new is a variant with HBM3e memory which increases capacity (96GB -> 144GB) and bandwidth (4TB/s -> 5TB/s). The posted article butchers this. Here’s a much better source: https://www.anandtech.com/show/20001/nvidia-unveils-gh200-gr...

Although this is obviously great news, it is increasingly troubling how AI innovation (from the hardware sector) is pretty much limited to Nvidia.

I haven't really gotten too much into it, so I'm not sure how has Nvidia come to absolutely dominate this market; AMD GPUs are quite good in the gaming sector... though I guess... with just two real players in the GPU market it's difficult to really get anywhere.

The article talks about AMD announcing a chip supporting 192GB memory.

AMD’s hardware is essentially irrelevant for non-gaming applications because the software stack isn’t there. Unfortunately same goes for just about everyone BUT NVIDIA. It’s a situation that could be fixed, but it would require cooperative investment from, well, the rest of the semiconductor industry.

The fact that a large number of people care about a small number of ML models right now means that AMD has an opportunity to port just a few things and take market share.

By having modern tooling to program GPGPU, allowing several languages to target them instead of plain C dialect, IDE and graphical debuggers to track down issues on GPGPU just like on the CPU, and a myriad of good quality libraries.

Meanwhile Intel and AMD, never delivered something at the same level as CUDA for OpenCL, and when they finally decided to react with SPIR and C++, everyone was already too busy to care, and it isn't as if the tooling has improved that much.

Hence why OpenCL 3.0 is basically OpenCL 1.0 rebranded.

Nvidia in a very real way built the market by providing not just the hardware but the APIs and the tooling and so on. The rest of the market has been reacting to them for a long time.

Can't say they don't deserve their success in a capitalist sense. Can't say their success doesn't alarm me.

They literally built their success and shaped the future of humanity… Our architectures are what they are because they run well on nVidia GPUs, same for our optimisers. What run well set a direction, the hardware got optimised and so did the software in a beautiful feedback loop.

> The rest of the market has been reacting to them for a long time

The rest of the market has not been reacting to them for a long time.

Competition will have to come from somewhere else than amd since the CEOs are literally family.

>I'm not sure how has Nvidia come to absolutely dominate this market

They had the foresight to see the coming wave and ensure they have both best hardware and best software stack.

What puzzles me more is why AMD isn't throwing money at their software side to sort that out. Rocm was launched in 2016. 7 years later running pytorch on their flagship GPU still requires a janky laundry list of crowdsourced instructions:

https://github.com/AUTOMATIC1111/stable-diffusion-webui/disc...

Regarding NVIDIA domination, I want to promote that the Google project https://github.com/openxla/iree exists and IREE acts as a way to turn Tensorflow, Pytorch, and MLIR workflows to compute on cpu, vulkan compute, cuda, rocm, metal and others.

https://github.com/RechieKho/IREE.gd -- RechieKho and I collaborate on making this work for Godot Engine, but IREE.gd is at a proof of concept stage.

JM2C

lood_in_4bit=True will let you run Llama2-7B variants at 6.3GB VRAM.

Reply on news.ycombinator.com