Hacker News Reader

Comment by crystal_revenge | Hacker News Reader

The most clarifying post I've read on attention is from Cosma Shalizi[0] who points out that "Attention" is quite literally just a re-discovery/re-invention of Kernel smoothing. Probably less helpful if you don't come from a quantitative background, but if you do it makes it shockingly clarifying.

Once you realize this "Multi-headed Attention" is just kernel smoothing with more kernels and doing some linear transformation on the results of these (in practice: average or add)!

0. http://bactra.org/notebooks/nn-attention-and-transformers.ht...

View on HN

It's a useful realization, too, since ways of approximating kernel functions are already well-studied. Google themselves have been publishing in this area for years, e.g. https://research.google/blog/rethinking-attention-with-perfo...

> To resolve these issues, we introduce the Performer, a Transformer architecture with attention mechanisms that scale linearly, thus enabling faster training while allowing the model to process longer lengths, as required for certain image datasets such as ImageNet64 and text datasets such as PG-19. The Performer uses an efficient (linear) generalized attention framework, which allows a broad class of attention mechanisms based on different similarity measures (kernels). The framework is implemented by our novel Fast Attention Via Positive Orthogonal Random Features (FAVOR+) algorithm, which provides scalable low-variance and unbiased estimation of attention mechanisms that can be expressed by random feature map decompositions (in particular, regular softmax-attention). We obtain strong accuracy guarantees for this method while preserving linear space and time complexity, which can also be applied to standalone softmax operations.

So do we think that some form of this is what they are using internally to get those long context lengths by already using sub-quadratic architectures [1] in their deployed models?

1: "Context Is The Next Frontier by Jacob Buckman, CEO of Manifest AI" (https://youtu.be/wJyl6kBCwmY?si=ruxMWdENjazu3rp6)

For those who don't know the term "kernel smoothing", it just means

    ∑ᵢ yᵢ · K(xᵢ, xₒ) ⁄ (∑ⱼ K(xⱼ, xₒ))

In regular attention, we let K(xᵢ, xₒ) = exp(<xᵢ, xₒ>).

Note that in Attention we use K(qᵢ, kₒ) where the q (query) and k (key) vectors are not the same.

Unless you define K(xᵢ, xₒ) = exp(<W_q xᵢ, W_k xₒ>) as you do in self-attention.

There are also some attention mechanisms that don't use the normalization term, (∑ⱼ K(xⱼ, xₒ)), but most do.

> ∑ᵢ yᵢ · K(xᵢ, xₒ) ⁄ (∑ⱼ K(xⱼ, xₒ))

That clarifies things...

It does, for anybody who studied math at the level you need to understand Attention (some linear algebra). Please no low effort comments, ask if you don't have the math background and people will gladly help. This is sum notation, see https://en.m.wikipedia.org/wiki/Summation

No, it doesn't really clarify things. I had the best linear algebra grades in my year at my university, and if you don't know anything about kernels, this is not helpful (what are xi and yi in the first place?).

It's all described in the referenced link. No need for everyone to get antsy.

> 0. http://bactra.org/notebooks/nn-attention-and-transformers.ht...

In kernel methods the kernel is typically given, and things like positional embeddings, layer normalization, causal masking, and so on are missing. Kernel methods did not take off partly due to their computational complexity (quadratic in sample size), and transforms did precisely because they were parallelizable, and thus computationally efficient, compared with the RNNs and LSTMs that came before them.

Reductions of one architecture to another are usually more enlightening from a theoretical perspective than a practical one.

Interesting read. But to me, describing attention as "a weighted mean of a continous learned k-v map lookup" is direct and descriptive, while saying "it's just kernel smoothing" is opaque and referential.

Maybe take the best of both worlds: "It's just kernel smoothing, which is a weighted mean of a continuous learned kv-map lookup."

Now you got the definition, and the alternative naming you can use to search for resources.

Wow, thanks for referencing that. What a very detailed and long read!

So do we think that some form of this is what they are using internally to get those long context lengths by already using sub-quadratic architectures [1] in their deployed models?

1: "Context Is The Next Frontier by Jacob Buckman, CEO of Manifest AI" (https://youtu.be/wJyl6kBCwmY?si=ruxMWdENjazu3rp6)

Reply on news.ycombinator.com