They can't even learn parity checks.

https://arxiv.org/abs/2207.02098

You should check out the work referenced in the abstract, the most recent here: https://twitter.com/lambdaviking/status/1630581475425828864

There are limitations for what a Transformer can compute if we do not allow for Chain-of-Thought type of output. By allowing the model to "show its work" allows it to effectively use the output as an "infinite tape". I'm simplifying but that's the basic gist of it.

I'll shamelessly plug my blog post from a week ago for a simpler take on the matter: https://blog.wtf.sg/posts/2023-02-03-the-new-xor-problem/

you'll be limited right away by the context lenght of 4096 tokens

Of course.