> For a sequence with N = 100,000 tokens, it would mean cost dropping by a factor of 100,000×

I'm not sure I understand the intended interpretation of this. Concretely speaking, if it cost CoolAI 100k seconds of compute to process a sequence of length 100k, it would not take them 1 second now.

I agree that as sequences become longer, the quadratic component will become more important. But as models get bigger, the attention component also becomes less important.

For example, to take a concrete model (say Llama-70B), it takes about 1.4e16 MLP FLOPs (70 billion * 100000 * 2) to process 100k tokens. The attention component takes about 6.5e15 FLOPS (80 [layers] * 100k [sequence length]^2 * 8192 [hidden dim]).

So even if attention turned constant it would reduce runtime by about 30% with today's model at 100k sequence length.