Hacker News Reader

Talaas have done this with a llama 3 model. Runs at like, 16k/tokens a second oror something obscene. Very little power draw too.

Doesn’t need hbm or lots of memory, because the hardware can just forward the data straight to the next layer and you don’t need to round trip through memory.

They claim to be working on an approach to make the underlying hardware a bit more reusable between models.

View on HN

Yeah, if you have a fixed llm topology, you can just effectively burns 2 top layers of the chip as Rom (model weights) - which has a per area density even better than dram - so it’s just attention and kv streaming that is hbm to sram transfer.

Most big model weights will not fit a single reticle sized chip - so you’d have prob 30 different chips to split the model .

And you’d need super fast chip to chip comms for the all-reduce and similar.

So scaling to 1T models is hard - and a long lead time - but can be very power efficient.

Reply on news.ycombinator.com