Yes, I have read that paper and have forwarded it eagerly to colleagues. Respectfully, please read the paper you have linked, especially section 8.2.

Look at section 8.2 in the paper you linked:

"8.2 Lack of planning in arithmetic/reasoning problems"

> However, it seems that the autoregressive nature of the model which forces it to solve problems in a sequential fashion sometimes poses a more profound difficulty that cannot be remedied simply by instructing the model to find a step by step solution

Nowhere in the paper is an evaluation on standard reasoning datasets (https://tptp.org) because they are much much much tougher than the simple examples that GPT-4 struggles with.

LLMs are a huge breakthrough but let us not muddy the science.