Hacker News Reader

Comment by qwertox | Hacker News Reader

I think it's more of a subtle publicity stunt for Tenstorrent's Grayskull [0] cards, which are highly optimized for inference.

With CUDA you get a lot of flexibility which you don't get with Grayskull, at the expense of having a more layered/patched API which has been evolving for many years while R&D came up with new solutions to new problems.

I wonder what Nvidia's capabilities are in creating new, highly optimized cards, similar to Grayskull, but I wouldn't be surprised if they don't have any interest in creating them since their current products are already consuming all their resources.

It's great to see that companies like Tenstorrent are offering more optimized and pricewise more accessible products. It would also ease the situation with how hard it is to get Nvidia cards, because even companies doing only inferencing are buying chips which are developed to be able to do much more than that.

[0] https://tenstorrent.com/cards/

View on HN

> which are highly optimized for inference.

My understanding is that grayskull is a devboard for inference, Wormhole is supposed to do both, because you can scale it out with multiple cards.

> With CUDA you get a lot of flexibility which you don't get with Grayskull

Do you? From what I can tel tt-metalium gives you quite low level access.

Grayskull/Wormhole/... basically have a bunch of tensix cores, which each have 5 small rv32 cores, where some are used for data movement and some for computation. IIRC two (could be one) drive a SIMT compute unit.

From what I can tell there is no "matrix in matrix out" style fixed accelerator, the SIMT unit looks decently flexible, the grayskull one is a bit limited isa wise, wormhole is supposed to improve on that.

tt-buda is supposed to be the high level API with pytorch, tensorflow, ... support, but idk how much compatibility there is at the moment. I assume there is a lot of software work left.

Wormhole is an accelerated NIC, like what Nvidia got access to when it bought Mellanox. It's meant for high-speed routing.

Yet you got me to reading the following:

> t-Series Workstations

> Our t-series workstations are turnkey solutions for running training and inference on our processors, from a single-user desktop workstation in the t1000 up to the t7000 rack-mounted system designed specifically to function as a host with our Galaxy 4U Server

> 8 Grayskull Cards

> 16 Grayskull Chips

So these cards can also do training.

https://tenstorrent.com/systems/aicloud/

What does it mean by "inference" in this context?

Neural networks have two different compute costs: training and inference.

These are roughly analogous to compile time vs runtime for compiled programming languages.

Training is in general a more intensive task. However, in an ideal scenario training is run once and inference is run millions of times, so the lifetime cost of inference is bigger - this is why it might make sense to optimize for intense.

Inference consists of a single forward pass, which is easy to compute. Meanwhile training has both a forward pass and backward pass (back propagation). The minimum required precision for training is higher since you want to be able to have both fast and slow rates of change. Bfloat16 is preferred for training, while int8 tends to be good enough for inference.

You’re trying to get an output from giving an input to a bunch of matrices (tensors if you will). You do not touch anything in the matrices themselves which happens during training. Essentially, they’re great from deploying models but not making one.

Reply on news.ycombinator.com