Transformers are very bad at counting in one feed forward pass, you need to explicitly tell them to use a counter in autoregressive fashion like here:

https://chatgpt.com/share/6775cb37-4198-8007-82cb-e897220827...