That's not correct. Try asking a question that requires multiple steps of reasoning and add ", think step by step" to the prompt. This not only changes the output, but also often improves the quality of the result... like you'd expect it to happen with people.
You definitely need to know a little and be able to push back, it feels like. But it’s been an absolute champ in describing why things are going wrong in a general sense when I’ve been having issues, especially with generics and templates in C#.
> ML cannot conceptualize of things in the abstract like people can
And:
> They cannot offer reasons, a train of thought like a person
Are very different claims! The first one just seems wrong: LLMs require abstraction to work, and early work in interpretability suggests they build rich world models during training (i.e. see https://thegradient.pub/othello/).
What is true is that often those models aren’t very legible, and it would seem current LLMs are incapable of introspection, and so can’t make those models more transparent.
The second one is a tricky one: you can often get it by explicitly prompting for a chain of thought, but it’s true current LLMs don’t seem great at this yet. The big jump in this capability when going from GPT 3.5 to GPT 4 makes me thing that this is just a limitation that will be overcome relatively soon.