Hacker News Reader

Comment by PhilippGille | Hacker News Reader

Dev kits:

It will be interesting to see their inference performance, compared to graphics cards. Will they be interesting for home labs?

I found one interview with unboxing a preview version (if I understood correctly), with some background info, but no performance numbers: https://morethanmoore.substack.com/p/unboxing-the-tenstorren...

View on HN

Memory seems very low for inference on bigger models?. Memory bandwidth is also massively lower than something like a 4090 or 7900xtx?. Am I missing some fairy dust magic here?

  This is purely a developer kit, letting people get used to the hardware configuration and the software stacks before the big hardware comes later in future generations.

I'd like to know what different precisions/ quantizations if any are supported. For LLMs 8GB is fine for playing around if weights are quantized. And as the article mentions it's more than enough for lots of computer vision models.

Grayskull supports FP8, FP16, BFP16, VFP2, BFP4, BFP8 [0], and some sort of 19-bit floating point in the SFPU. [1]

I don't know how and with what performance those are supported.

FP4 has 221 TFLOPS on the Grayskull e75, and 332 TFLOPS on the Grayskull e150. [0]

While wormhole has 82 INT8 TOPS per card. [2]

I haven't found any other numbers.

Edit: Some more info about data formats: https://docs.tenstorrent.com/tenstorrent/v/tt-buda/dataforma...

[0] https://docs.tenstorrent.com/tenstorrent/add-in-boards-and-c...

[1] https://tenstorrent-metal.github.io/tt-metal/latest/tt_metal...

[2] https://tenstorrent.com/systems/galaxy/

That's a lot of formats for FP8, FP4, and even FP2! (never heard of). I wonder how they did it. Nice.

Cool, thank you!

The website indicates each Tensix core ~1TOP theoretical. So not amazing.

It will have to compete on price/memory in the crowded inference space.

This card isn't competing on price - it's a dev kit.

Maybe power usage ?

Why are they so skimpy on memory capacity? I guess this is a dev kit for anything but LLMs?

Grayskull is make before LLMs being a thing. And their plan is like Groq, to distribute the compute graph across multiple processors to get higher effective memory and throughput by pipelining. But better-ish by having RAM so you can fit models on much less cards. Grayskull doesn't have this ability. The next generation Wormhole does by having 100GbE interfaces on the cards.

Also the CPUs on Grayskull is 32bit. Memory is addressed through the bank address so it works for now. But they'll have to upgrade to 64bit soon.

What's the memory interface like? Is an extreme NUMA thing with several orders greater potential for parallelization?

I'm probably dreaming; new CPU core designs are one thing, a completely new memory design is likely high fantasy..

I mean it says LPDDR4, nothing about how many channels but I would be surprised if it's anything more than dual-channel. Big cache but not anything not already seen in the AMD 3XD chips.

The quoted ~100-120GBps is going to be quad-channel LPDDR4

Will graphics cards eventually get renamed?

Are the optimizations for inference basically identical to graphics?

I imagine the workloads are different, and that inference has been the main market for years now...

The new things are already getting renamed, but the dust hasn't settled yet.

NPU: https://en.wikipedia.org/wiki/AI_accelerator

TPU: https://en.wikipedia.org/wiki/Tensor_Processing_Unit

why bother renaming? just re-acronym to "greatly parallel unit"

These specifications rather look like some AI acceleration ICs, and not like GPUs (recall that "GPU" stands for Graphics Processing Unit).

That’s why the article characterizes them as an “alternative to GPUs […] for AI development“.

That being said, it’s a Grayskull Processing Unit. ;)

I wonder, is the dual-use of GPUs a significant engineering bottleneck for Nvidia et al? My understanding is that for libraries like Tensorflow and PyTorch, much of the complexity is supporting the many-to-many mapping of model structures to GPU silicon in a way that reduces “compile time”, with all the tradeoff landmines you might expect there. I would imagine that abstracting this at the hardware layer is really valuable. Are these non-GPU architectures the future of machine learning, and can we expect the big vendors to start competing here or is this a business risk for them? (robbing Peter to pay Paul, like Google investing in AI that cannibalizes their search revenue)

I doubt it. Toolkits like TensorRT are specifically optimised, and NVIDIA already segregated their HPC vs gaming dies (and corresponding, die space).

The H100s feature a minuscule amount of ROPs and TMUs (mainly for graphics); and doesn’t come with NVENC/DEC, etc. DirectX, Vulkan is not supported.

You’ll just see more and more tensor cores, optimised CUDA cores for bfloat, etc.

H100 does have NVDEC and jpeg ingest accelerators

https://www.servethehome.com/wp-content/uploads/2023/10/NVID...

Tbh it’s mildly surprising they even removed NVENC considering the overall size of the chip (in the absolute we are only talking about low-single-digit mm2 savings) and then H100 is still advertised and has features targeting VM graphics/visualization still… remember they also still put a full graphics pipeline with ROPs/TMUs on the chip, just no actual display hardware.

It's a tradeoff with verification and validation, not so much die-space. If you include the IP, you need to run an expensive (time-consuming) simulation and emulation for the IP. Then, you need to validate it on first silicon and pray that it doesn't impact yield in HVM.

These days, simulation runs basically nonstop on all the FPGAs the company is able to throw at it until tape out. Then, in post-silicon validation, you have engineering teams working 24/7 to validate the device on the bench. Finally, test engineering tries to achieve yield in HVM.

Anything that you don't need on the spec sheet or the designer insists on is cut to save time to market.

These can also be used for machine learning actually (see Dali for data loading for instance)

> I wonder, is the dual-use of GPUs a significant engineering bottleneck for Nvidia et al?

Not much. GPUs became array processing engines in the early 2000s and the amount of residual fixed function graphics hardware is tiny now.

The biggest thing is that a lot of network evaluation is being done at lower precision (BF16/ FP8 / ternary) now, but my understanding is that training the networks still requires FP32. As these companies are really targeting the training market the high precision modes needed for graphics can't be thrown away. If a company was to target the end user, then you could optimise for lower precision, but end users find dedicated neural net acceleration a luxury they can get for free by buying a GPU.

The dual-use of GPUs is what's driving Nvidia's demand, for the most part. They bet big on more outlandish designs (eg. CUDA/Tensor cores) and mixed-precision computation, and did a lot of the hardware/software work to get it shipped out years ago. There wasn't much interest in CUDA outside niche research/industry applications for a while, it's a small miracle they kept it around long enough to see the crypto and AI booms. Now their bet is paying big dividends, and other companies are trying to figure out fast ways to leverage raster compute for general-purpose acceleration.

Apple initially developed OpenCL with Khronos as a similar-ish analog for GPU acceleration a while ago. There were a few partners that invested in it, but it suffered the same lack of demand as CUDA and languished for a while. Now Apple doesn't support OpenCL, so the purpose of a multi-platform acceleration library has kinda been scuttled. Nvidia played their cards well, and their competitors are going to feel the pain for a while unless they work together again.

Sure, no one actually cares about the graphics part anymore in these contexts. Many Nvidia Datacenter GPUs are also no longer GPUs in that sense.

I know it isn’t the what you mean, but as an aside (rather than a contradiction), the gaming marked is still much bigger than the ML model training one, right? So presumably people care about the graphics part in the sense that development there funds the development of the cards, haha.

I always assumed that was what killed Xeon Phi. Hard to justify a GEMM card in and of itself. However clever you get, NVIDIA will be back next year with twice as much bandwidth.

According to my googling, data center has been bigger than gaming for nvidia since 2022, and almost x2 in Q2 2023 following a crazy nosedive in gaming sales.

Woah, that’s really something

Nvidia burnt quite a lot of goodwill with the 40xx series cards and pricing.

Melting connectors, obscene pricing, marginal performance increase, zero availability; then when hardly any of that was addressed, Ti versions release which are equally expensive, and equally lacklustre. Combine that with patchy support for frame generation and 30xx series cards still going really strong, and you’ve got not a lot of incentive to upgrade.

In dollars, for sure.

In units sold?

I think dollars was the right figure to defeat my point. My thought was that they would be impossible for their more datacenter type competition to beat, because their R&D budget came from a much larger gaming segment. But, if they are around the same size, or the data center type users actually pay more, it is a level playing field.

I’m sure they don’t mind having both revenue streams of course! But on the other hand, they have to serve two sets of interests.

No, datacenter GPU revenue has surpassed gaming revenue some time ago. At least for Nvidia and I fully expect this trend to continue.

I'll bet against that. NVIDIA's margin is so large that Google/Microsoft and others can fund / are funding large teams to make TPUs and other dedicated hardware competitive. I also believe LLMs are part of the hype cycle just like crypto. In ten years the gaming market will return to being NVIDIA's primary market.

A couple factors to consider:

1. The hardware isn't nVidia's primary moat. CUDA is. Currently all machine learning tooling assumes you have a CUDA device.

2. The hype cycle may be real, but nobody really knows how much better AI will become. The uncertainty is driving the hype (which is obviously on the optimistic side), but it could well be that models become so good that everyone "needs" an AI chip like how we "need" a smartphone today.

3. The gaming/graphics market is probably going to slowly shrink. There's only so many pixels you can push to the eye. There's very real diminishing returns there. At some point people literally won't be able to tell the difference between the newer/better/faster graphics and the older generation, at which point the GPU vendor's margins drop significantly.

> 1. The hardware isn't nVidia's primary moat. CUDA is.

People really don't appreciate this. So many times I see people comment that AMD should implement CUDA and not try to get software to move standard.

AMD have a massive task ahead because they NEED to get CUDA unseated just to play on a level playing ground, let alone win.

If by "LLMs are part of the hype cycle" you mean "people are hyping them more than their current capabilities entirely warrant" maybe. If you mean "LLMs are a hype technology that is going to peter out without leaving a major impact on society" as kindly as I can phrase it, what on earth are you smoking, or are you Amish?

To be fair, GP compared the LLM hype with crypto... And as niche crypto may be in the future, I don't think it will also "peter out without leaving a major impact on society"...

Crypto will not be niche in the future. Bitcoin has a market cap of 1,349,125,982,176 (https://coinmarketcap.com/currencies/bitcoin/) and recently surpassed the market cap of gold itself and has large investors from Black Rock to Apple, ans recently got approved for ETFs which means the general population's 401k's will have bitcoin in them starting this year. Other crypto currencies may die out, but bitcoin is here permanantly.

If you haven't yet, buy some bitcoin and stop letting the government steal your hard earned money through inflation.

Many of them don't even have video out ports, they are purely for the compute power.

The hardware is still on the chip.

Yep just need to load the model. It can and probably should be headless.

Obviously they are graphicless processing units.

They are "inferrence only" according to the article

grayskull is "inference only", but wormhole will also "support" inference.

I put those in quotes, because theoretically both can do training, it's just that grayskull isn't well suited for it, because of the internal fp19 format, and no good support for scaling to multiple cards. wormhole will have internal fp32, and is designed to scale out with across multiple cards, and servers.

At least that's how I understood it.

FP19 is perfectly fine for training - llama is trained in fp16, many train in bf16, with microscaling exponents, you can do 6 bit training : https://arxiv.org/abs/2310.10537

I think the broader context here is that it's "fine for training" in the sense that you can successfully train a model, but it's not "fine for training" in the sense that it can only train small models due to the lack of scaling across cards, which directly cuts against where ML has been trending over the last several years.

In LLM-land we've rapidly gone from training bespoke models to doing fine tuning to RLHF to zero-shot prompting. The better the underlying model the more you can do without additional training, so hardware that fails to scale up to the largest training runs will have limited practical utility despite technically supporting training.

Wormhole does support scale out though via it's built in networking? And there seem to be links for something on top of the board, NVLINK style.

And yes, I know, I've been working on LLM pretraining for about 4 years now, since 2020. The number formats themselves so far are mostly scale invariant or improve with larger scale - you can quantize a larger model and see less performance drop than a smaller model.

Matrix-Math Coprocessors.

8087 - vector edition.

GPGPU is a thing.

> GPGPU is a thing.

Indeed.

GPGPU means "using a GPU for things that are General Purpose (GPGPU)". The aforementioned IC is not a GPU that is used for general-purpose computations, but a specialized chip for AI computations.

Isn't a GPGPU just a PPU?

Reply on news.ycombinator.com