The Clever Hans Effect, Iterative LLM Prompting, and Socrates' Meno
aalokbhattacharya.substack.com
aalokbhattacharya.substack.com
I like to point this out with the analogy of a Ouija board [0], where the manufacturers are reassuring their customers with: "Sometimes it channels the wrong spirits and ghosts from the beyond, but we're working to fix that."
Much like LLM "hallucinations", the wording implies something about "normal" operation which isn't quite true.
In my experience of using LLMs for various tasks, in _a lot_ of use, the output is very much in line with reality. Coding is not a domain where it shines in this. Natural language based domains, though: they're really good at saying true things and do so _way_ more often than not, to the point one could easily call the former "normal" operation.
> Iterative prompting, like Socratic questioning, may never produce true intelligence in the systems it engages with, but it reveals the collaborative nature of intelligence itself—an emergent property shaped by context, guidance, and the interplay of minds, human or otherwise. Through this lens, the promise of AI shifts from creating autonomous intelligences to augmenting human inquiry.
This reads (like the rest of the post) as if the author is curiously unaware that, in fact, lots of research has been published on what LLMs can do "zero-shot" or without iterative prompting, vs what takes iterative prompting, and that the number of things that they can do without anything at all Socratic in shape is both non-zero and increasing predictably with scale.
IMHO what LLMs have most obviously revealed about "true intelligence" is just how attached many people are to the idea that it's a special possession of humans and the tenuous intellectual lengths to which they will go to hold on to that idea.
We are mimicking machines and halluciate everything just the same.
Those kind of discussions are getting now a bit boring to be honest.
Not sure that's enough to start the witch-hunt: It was common long before LLMs came around. :p
> Mentions the boy, the Socrates effect etc repeatedly but does explain what that means until very late in the post.
When I highlight some words and zoom out, it's explained within two paragraphs of being introduced, and only because the intervening one is describing the other half of the comparison.
In fact, the big labs should post-train their models to extract as much information from questioners about their questions before attempting to answer them.
These interaction traces would also be useful during later training runs.