1960s chatbot ELIZA beat OpenAI's GPT-3.5 in a recent Turing test study
arstechnica.com
arstechnica.com
And I think that part of the reason that both Eliza and GPT work so well is that the observer really, really wants it to, because that's a crucial part of how this activity is structured, as described by Grice and others:
Language has operations and rules that construct the space of all possible thought, and not all languages are in the set of all possible thoughts. The host form navigates a coordinate space of a constrainted possible thoughts. Whenever a language is used to communicate across interface boundaries (bodies) then some level of mapping takes place to exchanging mutual possible thoughts.
Also, under Perl set up cpanminus if you don't want to play with Emacs:
curl -L https://cpanmin.us | perl - App::cpanminus
~/perl5/bin/cpanm -n local::lib
echo 'eval $(perl -I ~/perl5/lib/perl5/ -Mlocal::lib)' >> ~/.profile
echo 'export PATH=$HOME/perl5/bin:$PATH' >> ~/.profile
cpanm -n RiveSript
~/perl5/bin/rivescript ~/perl5/lib/perl5/RiveScript/demo/
> The study also showed that participants' education and familiarity with large language models (LLMs) did not significantly predict their success in detecting AI.
Also, humans only score 61%, and the best chat gpt prompt gets 41% (but 49% in the graph?).
I’m definitely having trouble distinguishing between content farm crap written by humans and chatgpt output. They’re usually of comparable accuracy.
I’d be curious to see an extended study where they had a dozen human participants or so, and got statistically significant percentages for each individual. Is the worst human below 41%? Is the best close to 100%?
M-x doctor can fix all your woes.
On the other hand, it did offer (correctly) to use the word "fucking" the other day for me instead of trying to change that to "ducking", so maybe we're making a bit of progress!
Aside from the fact that ELIZA was tuned to present as human and GPT3.5 was not, the validity of the basic Turing Test still seems a key issue. Fooling ordinary humans is a common occurrence, as Richard Feynman pointed out, "... and the easiest one to fool is ourselves.".
It doesn't test "Can a machine think" so much as it tests "Can a human be tricked". There is a reason why ML research mostly ignores the Turing "Test".
But, alas, its name has a nice ring to it, it is known (by name at least) to pop culture, and easy enough to understand for everyone, and so it gets far more attention than it deserves. Pretty much like Asimovs three "laws" of robotics :D
Like, most people would say that animals can think, but clearly it can't pass the Turing test. Would you rather have unconscious GPT-4 or a conscious pig "working" at your job?
I disagree.
In my experience human customers don't care much about whether its obvious when they are communicating with a robot. In fact, in several use cases it's considered good practice, and may even be legally required, to tell them when they are.
What they DO care about, is whether their requests, inquiries, complaints, etc. are handled quickly and efficiently.
We don't create value for our customers by giving them good chatterbots. We create value by giving them highly functional automated systems.
Maybe we just disagree on what we mean when we say "think"?
Chatterbots, including the 60s-era ELIZA, which are mostly useless for this kind of task, have repeatedly been shown to be able to trick people into chosing the machine during the Turing "Test", so I'm not sure how you come by that conclusion.
> at least if you require consciousness as a requirement for "thinking".
I don't require consciousness, or an abstract and not-even-clearly-defined "thinking" capability from an advanced ML model. I require fitness for purpose, whatever that purpose is, and in most use cases I ever encountered, "trick the humen into believing I'm a human" isn't one of them.
> Maybe we just disagree on what we mean when we say "think"?
I don't think we really disagree, I think the problem is that "thinking", "consciousness", "intelligence" and a lot of other terms that often get thrown around when talking about AI, are really not well defined in technical terms.
This is not actually clear; people commonly feel that they are having meaningful conversations with animals.
The answer to "can a human be tricked" is always yes, regardless of the particular context.
Of course it can't be proven one way or the other, we can't prove humans are conscious either.
Humans think they are conscious because they have an internal sense of self. How do you determine if a machine has that? If it can mimic a human was one hypothetical test.
There are more robust versions of Turing Test than this one, so can't toss it out as an idea.
But importantly, Turing didn't just mean it'd be tested on ability to make small talk. He chose the format because he considered text to be a sufficiently general interface for testing any particular intelligent behavior, and gave examples such as feeding in chess moves.
If it can perform all such intelligent behavior at human level (as far as human interrogators can tell), then that's a significant result and - depending on what philosophy of mind you subscribe to - may or may not be sufficient to demonstrate that it can "think".
And there's the problem again.
The "Test" isn't evaluating a machine. It evaluates an interrogators response to that machine.
The core problem of answering the question "can machines think" isn't evaluated by Turings method. The "Test" doesn't test what it is supposed to test.
And that has nothing to do with "what philosophy of mind you subscribe to", that is simply a fundamental flaw in the tests design, and the reason why it's pretty much pointless; it defines "thinking" as "the ability to trick humans". Well, there are optical illusions that can trick people into seeing things that don't exist.
There are definitely those that would reason "Using the tools and expertise available to us, this behaves indistinguishably from a human - therefore we should believe it is intelligent, like humans are".
It's not the same question, and Turing is clear that it's a surrogate question due to the ambiguity of the original question, but it's not unrelated either.
They can believe whatever they want, that doesn't mean I have to share their beliefs. Fact of the matter is: Without a definition of intelligence that doesn't involve pointing at ourselves, there is no accurate test of intelligence for something that isn't us.
> this behaves indistinguishably from a human
Several problems with this:
a) Something that behaves like something else, isn't necessarily the same thing. Nature doesn't do duck-typing. A porpoise isn't a fish.
b) Without a clear definition of "indistinguishable from human", this, again, is more a test of the discriminating factors (the interrogator) ability, than of the qualities of the thing that it is actually supposed to test. Case in point: Once upon a time, people didn't know that there is a difference between stars and planets, as both were just bright dots in the sky.
I don't claim you have to, and noted that whether you're willing to make such an inference depends on the views you subscribe to.
> Fact of the matter is: Without a definition of intelligence that doesn't involve pointing at ourselves, there is no accurate test of intelligence for something that isn't us.
Say people have varying, largely ambiguous, definitions of what it means for a webpage to be responsive. You propose measuring time from user click to the new content being presented.
For some the test is entirely sufficient as it already encompasses what they mean by responsive, for others it's a partial indication but not on its own conclusive, and for some it may be a separate concern to what they mean.
The test wouldn't settle the issue of definition, and generally isn't equivalent to the original question, but is still useful, relevant, and can actually be carried out.
> Nature doesn't do duck-typing. A porpoise isn't a fish.
I don't think nature really does any typing like this at all. "Porpoise" and "fish" are our own labels, and we may choose to define them based off of observable characteristics or behaviors.
> Without a clear definition of "indistinguishable from human", this, again, is more a test of the discriminating factors (the interrogator) ability,
The test can be repeated with many interrogators to help factor out the variance in individuals' ability.
The proposed test for responsiveness actually does test an aspect of responsiveness. The problem here is only in the definition of the variable to test: Yes, "responsiveness" isn't well defined.
The Turing "test" however doesn't test intelligent behavior at all, so the problem is BOTH in the definition of the tested variable, AND in the actual test.
That's all that was really required for the purpose of that analogy - arguing that there can still be useful/relevant tests without agreement on definition, as a response specifically to your "Without a definition [...]" claim.
But, I do think you could equally say that the responsiveness test is actually testing speed of the computer or speed of the connection, in addition to some defining responsiveness in a way that isn't related to click-to-new-content speed.
Better way to have done it would have been to put them in a chat room with x other humans and just have a normal conversation/meeting, without the humans being told that AI is even involved. GPT would have aced it, this article is just scammy clickbait.
it was eventually found
And the whole article can be shortened down to “GPT 3.5 has been conditioned by OpenAI specifically not to present itself as a human.” and it does it with success :)
Nearly fooled me
That's exactly what a human would say.
* Are you a human
> Would you prefer if I were not a human?
Eliza wins this round! %-P