Hacker News Reader

Comment by behnamoh | Hacker News Reader

I wonder if the transformer will remain the de facto arch for LLMs in a couple years. We already have Mamba, RWKV, etc. which are huge improvements over transformer in terms of memory requirement and speed. I wonder why there's still so much work (and hacks and workarounds) on transformer to make it go faster and consume less VRAM whereas the same effort could be spent on other architectures that solve some of those problems fundamentally.

View on HN

Theoretically transformers are fundamentally more powerful than stuff like Mamba and RWKV because they don't "forget"; the newest token can attend perfectly to the oldest token if it wants. Mamba and RWKV on the other hand compress the old state, based on information available at the time of compression. If it turns out the new token wants some information from the old token that during the compression was excluded due to the compression thinking it wouldn't be needed, the new token has no way to access it.

Yet this all-to-all approach of transformers is bottlenecked on compute, while Mamba can take in a context length of a million tokens, that might have a positive effect on long range tasks.

What use is to have infinite context if new tokens can only access a lossy compression of past tokens?

There is definitely a use for this kind of model, but this also shows why Transformers are still the main architecture we use today.

I know almost nothing about this space but “lossy compression of millions of past tokens” feels a lot more like how actual human memory works than “perfect access to a small number of recent tokens”.

RNNs and LSTMs from the past did this as well (but cannot be trained in parallel as each token has to be compressed sequentially). Transformers ate their cake.

Newer methods are going back to similar concepts but trying to get past previous bottlenecks given what we've learned since then about transformers.

> feels a lot more like how actual human memory works

Beware of feel-good traps like this.

If you were to map the human connectome to a computational neural network down to the ion channel, it'd be at least 500 quadrillion parameters*. That's at least 5-6 orders of magnitude beyond what is currently possible with SOTA ML which means that even if the human brain was 99% devoted to compressing those tokens, that 1% that could actually do work with them is still a thousand times bigger than GPT4. There be emergent dragons.

* This is a fascile argument to begin with since biological neuron signals aren't quantized and ion channels are far too complex to map to a single static parameter

> biological neuron signals aren't quantized

Perhaps, but they inherently have imprecision due the nature of being biological. Expose the same neuron to the same input N times, and you'll get a range of output. The effect of noisy analog data is, broadly speaking, similar to the effect of low-resolution digital data.

Expose the same neuron to the same input N times and you won't get a randomly distributed range of outputs, you'll get a signal that is either attenuated or potentiated. That's not noise, that's the simplest form of neuroplasticity at work (which is why static parameters can never represent ion channels).

Maybe against HN rules, but: thank you. Very helpful comment!

I didn’t know those alternatives had those weaknesses. I did have a whole list of alternatives to check out. Having already looked at some, do you know any transformer alternatives that don’t forget things but were still sub-quadratic? What rabbit holes are worth digging deeper into?

Doesn't sub-quadratic attention fundamentally imply forgetting things?

This is fair -- the newest token can attend perfectly to the oldest token, within the context window.

but also, on a broader scale, if a transformer model is presented with a long input that does not fit in its context (e.g.: you are building a chatbot, and you have a very long chat history), it must "compress" or "forget" some of that information (e.g.: repeatedly summarizing historical messages, dropping them and appending the summary at the beginning of the input).

Mamba/RWKV/other "recurrent" architectures, can theoretically operate on unbounded input lengths; they "forget" information from earlier tokens over time, but is that not comparable to what a transformer must do with input lengths greater than their context window?

The jury very much still seems out on this. Computationally speaking, I believe Mamba is Turing complete, while transformers aren't (they can't do loops), so technically Mamba is more expressive. But of course, the question is always whether it ends up with lower loss.

(well actually its not turing complete, but it definitely seems closer)

There’s hacks and workarounds because billions of dollars have been poured into transformers by the biggest tech companies in the world.

Your assumption that the same effort could be given to MAMBA or RWKV isn’t necessarily true. I’m sure there’s research arms looking to see if they scale to the level OpenAI or Google have their transformers at now, but they’re still very much in their infancy.

This all ignores the risk of them not panning out and the lost time not focusing some energy on transformers.

The current best-in-class performance is from transformer models so of course research is focusing on this. You're creating a false dichotomy between "work on transformers" and "work on something new and fundamentally better." All of the current work on optimizing transformer architectures is exploring and better understanding the space, and so it IS leading us to better architectures. This is the most fertile ground for new insights that will outperform the current approaches.

New technologies aren't generally just created out of whole cloth by some genius, they're build up of layers of incremental improvements, each of which is modest in its own right but which when taken together are groundbreaking.

The speed of transformers is memory bandwidth limited. I think that it's possible to speed them way up with different chip architectures, and I know there are lots of people working on this e.g. https://etched.ai making asics and https://untether.ai making chips will co-located memory and compute cells. So unless some architecture really starts beating transformers badly on language tasks, I think the speed problem is going to disappear as silicon architectures adapt.

Yup, the company I work for has the world's fastest LLM appliance, based on a hardware design co-locating memory and compute. It requires compilation innovation as well in a sort of hardware-software codesign. We don't see transformer models as being an impediment in terms of memory requirements or speed in the near future.

[I would be happy to say more but I just got a top-level comment flag killed, presumably because they though I was advertising, so I won't mention the company name.]

Are you guys hiring? I'm actually looking for a job in this industry.

Yes we are! (I put links in my profile because I'm worried if link here or mention the name I'll be flag killed again.)

Awesome, thanks! Is there someone there you could connect me with for an introductory chat? I put my contact info in my profile, I completely understand if it's not possible. Thanks!

Yes, sure! I've emailed a recruiter to introduce you. Thanks for reaching out and showing interest.

If I understand correctly, Groq chips have 220MB SRAM and the next best level is DDR4? How many chips are needed to run Llama2-70B at those speeds?

Cool that you know the tech specs of the GroqChip! Yes, that's right, 220 MB of SRAM per chip. I think the demo where we first broke 200 tokens / sec was running on 1 GroqRack, so 64 chips. The live public demo that's currently running at 275 tokens / sec I think might be running on two GroqRacks, so 128 chips. I'm not certain of either of these figures so please don't quote me! But those are the right ball-park.

This article from less than a month ago says that it is on 576 chips https://www.nextplatform.com/2023/11/27/groq-says-it-can-dep...

Thanks, looks like you're right and this demo is running on 9 GroqRacks (576 chips). I think we may also have an 8 rack version in progress. We've tried a variety of different configurations to improve performance, which is possible because of the high level of flexibility and configurability of our architecture and compiler tool chain.

Link to HN discussion about the appliance: https://news.ycombinator.com/item?id=38739199

There is still so much performance to gain by applying software-based optimizations for superior performance of transformer LLMs on GPUs. From ~ 30% GPU utilization to over 90% is possible. I wrote a white paper on this, page 5 and 6: http://tinyurl.com/uujr3z4f

> why there's still so much work (and hacks and workarounds) on transformer to make it go faster and consume less VRAM whereas the same effort could be spent on other architectures

Probably because the pace of development is now no longer determined by ML researchers but by CS engineers. ML is "easy to pick up" until you have to understand the reasons certain architectures work or don't for different applications.

The best technology doesn't necessarily win. First to market matters, especially when models on the scale of gpt-4 cost 100mln to train.

"best" is subjective, but I do think the industry will eventually converge onto an architecture that is significantly more cost effective than current state of the art. Regardless of who is first to market, everyone is incentivized to continue down this path with their research on improving LLM performance.

While the first mover advantage exists, I don't think it's a sufficient moat in the long term.

OpenAI's moat is the high cost of training as you mentioned but this might be obsolete in a few papers down the road.

They themselves realized this by trying to turn it into a platform but I don't think it's enough.

technical debt

Reply on news.ycombinator.com