Show HN: Long context window for LLMs

It's where RoPE fails to generalize to longer contexts, because this method introduces additional data - a data required to encode position.

FFT method on the other hand does not introduce any additional data - it learns the position from the available data (input) using long convolutions with wide kernels (it generates those kernels from the data).

If you scroll down the notebook, you'll see the generation output made by RoPE and generation made by FFT and what happens to the text generated with RoPE for longer and longer sequences.