Hacker News Reader

HNHacker News

Top New Best Ask Show Jobs

Comment by Veedrac | Hacker News Reader

Parent Full thread

Veedrac·

18GB is huge! An NVIDIA V100 has 6MB of L2 memory. HBM is off-chip, and vastly (~100x) slower.

View on HN

That's true, but it doesn't match their claim of keeping all of the model on the chip.

An 18 GB cache is huge for sure, but that's not what they claim.

18GB of very fast memory will still be just as hard to keep fed with data as that 6MB cache

The idea is that the whole model resides in the fast memory, so you don't need to ‘keep it fed’.

44K/core is very little memory

Indeed, but cores are only responsible for small fragments of the network, so don't need huge amounts of memory.

Unless you need to multiply large matrices, where you need access to very large rows and columns...like in...ML applications

That's what the absurdly fast interconnect is for. You send the data to where the weights are.

Absurdly fast != Single cycle

It will be physically impossible to access that much memory single cycle at anything approaching reasonable speeds. I suppose you could do it at 5Hz :)

A core receives data over the interconnect. It uses its fast memory and local compute to do its part of the matrix multiplication. It streams the results back out when it's done. The interconnect doesn't give you single-cycle access to the whole memory pool, but it doesn't need to.

I think it is telling that in one sentence there is a claim that it is faster than nvidia, and in another, a claim it does tensor flow. I do not think this architecture could do both of these at once. It could not do tensor flow fast enough (not enough local fast mem) to compete even with a moderate array of GPUs

Hey! You seem knowledgeable, mind emailing me at tapabrata_ghosh [at] vathys (dot) ai ?

Reply on news.ycombinator.com