Researchers have only trained these LLMs on more data and have even less understanding of what these LLMs do internally since their architectures are with in a massive black-box with unexplainable numbers and operations going on.
That isn't helpful to researchers or even serious professionals in high risk industries. It makes LLMs less trustworthy for them are is incredibly unsuitable for their use-case in general.
I suspect that complex intelligence, that cannot be directly attributed to structure of the underlying LLM, has emerged. I am guessing it has to do with the use of language itself and at a sufficient enough size, this property exist in both humans and models.
Seeing how everyone is so divided on this really highlights how it's almost purely a philosophical argument about what intelligence actually is.
https://www.reddit.com/r/ChatGPT/comments/12l9nwx/really_imp...
Not even the researchers who created it can get it to transparently explain it decisions.