> The Transformer models, used in this experiment, all have a single attention layer with model dimension and context length 32.

I think we are going to need to see more experiments here, especially because the theoretical motivations here are weak