New attention mechanisms that outperform standard multi-head attention

I’m certainly not a domain expert, but one thing I have read repeatedly about Transformers is that not all tricks scale the same.

Can't believe that FNet paper flew under my radar all this time—what a cool idea! It's remarkable to me that it works so well considering I haven't heard anyone mention it before! Do you know if any follow-up work was done?

The 2D Discrete Fourier Transform (DFT) can replace attention because the 2D DFT looks a lot like attention. The algorithm loops over each pair of values in the input matrix, calculating first a linear combination of each pair that scales based on the distance between the values in the matrix. Closer values have more influence on each other than more distance values. This linear combination is fed into the famous e^(i*pi) equation that transforms it into a sinusoid. The sinusoids are summed up, giving you an output matrix that is a sum of sinusoids representing the frequency domain.

By comparison, the attention algorithm also combines each pair of values in the input matrix, but that linear combination involves learned weights, rather than simple fixed basis functions (sines and cosines) in the DFT.

The FNet paper suggests that the DFT works as a replacement for attention because in making the pairwise calculations to generate the 2D frequency representation of the input, information on these pairwise relationships is surfaced and made available to later feed-forward layers. As a handy add-on feature, the DFT also makes positional embedding unnecessary, because positional information is encoded by the DFT at each layer. That being said, in the paper they still applied positional embeddings so that they could make a direct comparison with the BERT architecture.

Amazing how these old signal processing ideas have made their way into neural networks, still proving their effectiveness and efficiency after 100 years or more (Gauss devised the Fast Fourier Transform algorithm in 1805 but we left it mostly on the shelf until it was reinvented by Cooley and Tukey in 1965).

There are sporadic attempts at making things more dynamic, like the Neural Turing Machine. It doesn’t seem to buy much actual power.

The answer might just be scale for all we know.

The answer is simple: AI systems aren’t just one technique, agent, or even agency — any somewhat anthropomorphic ones will be ensemblematic on an extensive and fundamentally-recursive level. LLMs are a groundbreaking technique that solve the “Frame Problem” by emulating human subconscious generative networks.

To paraphrase an old comment on here: the problem isn’t a chatbot gaining sapience inside a browser window, the problem is when billions of dollars are allocated to a self-administering ensemble of 10,000 GPT agents, each specialized for some task (aka functions). That, plus Wikipedia, Cyc, WolfraAlpha, YouTube, and Google Books at its fingertips.

“General” doesn’t even begin to cover what we’re already capable of, IMO.

See: Marvin Minsky, 1991; https://ojs.aaai.org/aimagazine/index.php/aimagazine/article...

Look at JEPA and Modulo-LLM.

Also AGI is a poor term to use because we as humans have no notion of what general intelligence is, does GI have morals and ethics, does it make decisions like we do based on executive functioning or does it work more like how ants do?

xLSTM has a working memory and seems to outperform transformer architectures: https://arxiv.org/abs/2405.04517

Current models can already argued to have something like working memory by storing information in little-used parts of the tokens. If placeholder tokens are handed to them that they can use as working memory, performance improves.

https://openreview.net/forum?id=2dnO3LLiJ1

https://news.ycombinator.com/item?id=40329675

It’s going to take time. I can’t speak to the actual quality of mamba other than to say the authors are extraordinary and should be taken seriously.

But training a large model requires a huge amount of capital so the biggest runs are designed around risk minimization. And remember, many of the decision makers of these runs made are in their positions by doing transformer-centric work. The true value of mamba is still unclear to me with very long context techniques being effective for transformers.

I believe the attention mechanism we use now was introduced in 2014 by Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio in their paper titled "Neural Machine Translation by Jointly Learning to Align and Translate."

2014. It took almost a decade for the potential of this technique to be realized and come to the attention (heh) of most developers. I don't know what researchers are doing with Mamba and RWKV, but we should let them cook.

> I doubt it because there's $BILLIONS in this market and everyone wants a piece of the AI cake, so it doesn't make sense to ignore promising methods.

I also doubt this result. The "why have $BILLIONS not already invested" question is interesting in its own right though. Generally, the literature on the theoretical bounds of swarm optimization is pertinent. Those $BILLIONS aren't being invested by a single omniscient entity, so they're subject to interesting constraints.

As one of many explanations, fragmentation is common. If $BILLIONS are split between greedy, mostly non-interacting entities (e.g., competing companies each trying to replace the transformer in a bounded number of hours and dollars while securing their market dominance), you expect, probabilistically, for each of them to converge on the same strateg(y/ies), especially if the "best" alternatives are obvious or globally known for some reason (e.g., some solutions intuitively feel "natural" or your researchers publish early results or you have employee movement between companies or whatever). Riskier strategies won't be touched, and you'll have $BILLIONS spent duplicating the same most likely alternatives when $MILLIONS would have sufficed.

The normal counterpoint is that a few big players dominate the spending, and they would have higher internal coordination. Interestingly though, they don't usually, except when that coordination would tend to enforce the same strategies smaller competition are pursuing. How often do you hear about stories like the misaligned Google+ integrations resulting in employee bonuses for poor customer experiences vs a forward-thinking executive actively devoting funds to a meaningful number of competing solutions? Approximately never. It's career suicide if you fail and depend on other people for your position, you _are_ actually more likely to outdo the competition with your increased resources if you just lean into the "best" alternatives, and for a whole host of reasons very few executives (except for people with real power) will coordinate a more comprehensive strategy, certainly not one orthogonal to the competition's just for the sake of allocating the global $BILLIONS more efficiently.

Separately (going back to the merits of the preprint), I'll probably read the full thing later, but a few points stuck out as suspicious on an initial skim. Notably, they seem to mix linear transformations in different domains. E.g., `xa` is linear in both `x` and `a`, and `vx` is linear in both `v` and `x`, but `xax` is _not_ linear in `x`, even if you try to "prove" that idea with `v = xa`. Linearity in `v` isn't enough to make the composition linear in `x`. A lot of their results seem to rely on eliminating those "redundant" computations, even though the things they're replacing with linear computations are actually higher order polynomials. On an initial skim, the other "novel" ideas also don't seem well grounded.

Their experimental results are decent. That could mean a lot of things (normally that the authors made more errors in their competitors' than in their own work), but it's probably worth looking into for a few hours despite my other complaints.

> Why, then, we barely see any non-transformer production-ready LLM these days?

Because having a 5% better non-transformer model doesn't help you if as a result you can't use the 10% improvements people publish that only apply to transformers. Very quickly you'll be 5% worse than those who stuck with transformers, and have wasted a ton of time and money.

FlashAttention is completely orthogonal to this. This work is about speeding up the computation of Q, K and V vectors while FlashAttention is about speeding up the attention algorithm itself.

You could combine the two.

You could provide the quote in full("In addition to providing rigorous mathematical comparisons,") so that the author's work in proving their point is not hidden by your effortless snark.

Sometimes it doesn’t need to. You might have a problem that isn’t web scale and where transfer learning is hard. We also need techniques for small datasets even if they are slower to train or are outperformed after 5 billion tokens.

Yep, came here to say this. The big thing about the results here that might not be obvious to someone not in AI is that the models being trained in this paper are very many orders of magnitude smaller than the LLMs we've all heard so much about recently, and they're also being trained on specific tasks instead of general language modeling.

So I'm not expecting this will find its way into a LLaMA near me any time soon, but maybe this is an interesting result for people working in the specific domains represented in the evaluations.