The thing to keep in mind - we know very little about how the human mind works.
Thinking that somehow humans are special and then making conclusions about LLMs truly work based on that it just pointless IMO.
Who knows. Maybe consciousness itself is a next word predictor.
All we know now is that LLMs exhibit some emergent behaviors that allows them to do very useful kinds of reasoning that was previously only possible by humans. Engaging in heated debate derails the eventual goal of putting these capabilities into use.
This is like “computers are just calculators”.
My team has been neck-deep in applications this past year. We have nobody on the team with ML experience and largely do not know how any of this works under the hood. But what we have built over the past months is considerable familiarity with the “personalities” of LLMs. Like this emotional context improving performance is a basic learning at this point.