Unlock Massive Context Windows: Positional embeddings are just training wheels

Would this generate the same completion for 'the cat sat on the' as 'on the cat sat the'?

ref: https://x.com/hardmaru/status/2010565943269999091