Large language models have a reasoning problem

Interesting study. Just reading outputs from GPT3 or other models makes it seem almost lifelike. Sometimes it is easy to forget it is just a well fit function with a lot of parameters.