The Impact of Positional Encoding on Length Generalization in Transformers

> Our findings reveal that the most commonly used positional encoding methods, such as ALiBi, Rotary, and APE, are not well suited for length generalization in downstream tasks. More importantly, NoPE outperforms other explicit positional encoding methods while requiring no additional computation. We theoretically demonstrate that NoPE can represent both absolute and relative PEs, but when trained with SGD, it mostly resembles T5's relative PE attention patterns.

Interestingly, position embeddings seem unnecessary for decoder only architectures going against what is conventionally thought. Wondering, if this result is as strong as it sounds.

> Overall, our work suggests that explicit position embeddings are not essential for decoder-only Transformers to generalize well to longer sequences.