Hey I like your exposition of attention in terms of the kernel trick, but the big huge difference is that kernel methods use the inner product which is a commutative operation -- it's bidirectional (and both tokens are projected into the space by the same function prior to being dot-producted).

This means that it can't capture unidirectional relationships, like "ball" is the object on which the verb "threw" acts in the sentence "I threw the ball". This relationship is true in only one direction; it isn't true to say "threw" is the object on which the verb "ball" acts.

I do think it would be fair to say that transformers generalize the kernel trick to noncommutative relations by applying a different projection function (W^Q and W^K) to the two tokens being considered. This makes the overall operation (project then dot product) a noncommutative operation.

And the thing that comes after harnesses/efficiency in terms of progress should be obvious if you understand all of the above.

Nirvana? Singularity? Paperclips? Vernor Vinge rising from the dead? I'm curious; please share!