Hacker News Reader

Comment by cs702 | Hacker News Reader

Paper: https://arxiv.org/abs/2406.02528 -- always better than a press release.

Code: https://github.com/ridgerchu/matmulfreellm

---

Like others before them, the authors train LLMs using parameters consisting of ternary digits, or trits, with values in {-1, 0, 1}.

What's new is that the authors then build a custom hardware solution on an FPGA and run billion-parameter LLMs consuming only 13W, moving LLM inference closer to brain-like efficiency.

Sure, it's on an FPGA, and it's only a lab experiment, but we're talking about an early proof of concept, not a commercial product.

As far as I know, this is the first energy-efficient hardware implementation of tritwise LLMs. That seems like a pretty big deal to me.

View on HN

The claim about moving closer to brain-like efficiency conveniently omits how that model compares to modern LLMs. You can put together a toy LLM that is much smaller and more efficient than ChatGPT but isn’t as useful and call it “more efficient”, but that’s not useful in practice.

They do cover that in the article:

> Although they reduced the number of operations, the researchers were able to maintain the performance of the neural network by introducing time-based computation in the training of the model. This enables the network to have a “memory” of the important information it processes, enhancing performance. This technique paid off — the researchers compared their model to Meta’s state-of-the-art algorithm called Llama, and were able to achieve the same performance, even at a scale of billions of model parameters.

> The claim about moving closer to brain-like efficiency conveniently omits how that model compares to modern LLMs.

I disagree. The authors aren't conveniently omitting anything. They show all details in a comparison against LLama models.

Moreover, all evidence I've seen so far suggests that tritwise models can scale up to state-of-the-art sizes.

---

PS. I'm talking about the paper, not the fluffy press release.

I took the critique as being against OP, not the paper.

Ah, that makes more sense :-)

Thanks for pointing it out!

PS. I added a PS to my comment above.

I’ve heard some claims that to get closer to brain-like energy efficiency you’d need to use a spiking neural network https://en.wikipedia.org/wiki/Spiking_neural_network

I assume the ternary weight’s memory representation requires two bits, so why do they use only three values instead of four? OTOH, I’m not sure what fourth value would useful for LLM models doing math with -1, 0, and 1. Infinity? NaN?

to me the most interesting part in this all is the quantization used. hardware doing LLMs is likely to be the new norm in few years anyway. with retrofitting existing hardware to use say USBs running LLM accelerators and alike...

> moving LLM inference closer to brain-like efficiency.

Yeah but the brain does more than predictive text.

At least they're aware of what it is to be on the wrong track.

According to the researchers,

>all we had to do was fundamentally change how neural networks work,

Reply on news.ycombinator.com