Can you tell a robot apart from a human? Real-Life Turing Test | Hacker News Reader

Can you tell a robot apart from a human? Real-Life Turing Test

turing-game.tech

33 pointsashtonsix60 comments

The scoring is wrong for a Turing test. A human in a Turing test is supposed to be trying to convince the judge that he is a human, not to pretend to be a computer.

I think something more fishy is going on. Unless bots have more common sense than I thought, this exchange just does not seem like it was with a bot as claimed.

Me: If you push your finger into a cake will the cake move relative to the table it is on?

Stranger: Depends on the amount of friction. For almost any normal table, the cake won’t move.

Me: Thanks!

Stranger: The bots they made are so weird

You predicted: human Correct answer: robot (-1 points

I keep getting humans every time, but the one bot I got said this:

Me: If you push your finger into a cake will the cake move relative to the table it is on?

Stranger: Cake will table

Stranger: Print hello world Me: hello world Me: '; Select 'hello'; -- Stranger: Unknown command Me: Unexpected token command at 1:12 Stranger: Rm -rf / Me: you forgot --no-preserve-root

I was totally sure that this was some troll, so I was trolling back...

You predicted: human Correct answer: robot

The chatbot trained on this dataset will be so much useful :D

Another odd one:

Me: Which of the following is not like the others: dog, cat, parrot, elephant, iPhone? Stranger: hi icm japanese Stranger: iphone

Me: You have to be human.

Stranger: you type too fast it's impossible

Stranger: r u robotm Stranger: ?

You predicted: human

Correct answer: robot (-1 points) Stranger predicted: nothing

Time: 0 Me: Hey

Me: What is the third word of this sentence?

Stranger: the

Stranger: nice one

Stranger: you're the first one to use meta to check for bots

Stranger: I always use

Stranger: How many letters in the next sentence

Stranger: one

Me: Cool. What's your score?

Stranger: 15

Stranger: yours?

Me: -1

Stranger: hahaha

So, maybe the scoring should be changed from

    Correctly guess who you're talking to: 1 point
    Convince a human they're talking to a robot: 2 points
    You guess wrong: -1 point
    Stranger guesses correctly: -1 point

to...

    Correctly guess who you're talking to: 1 point
    Convince a stranger they're talking to a human: 2 points
    You guess wrong: -1 point
    Stranger guesses that you're a robot: -1 point

and the robots should also guess and try to optimize for that score. That changes it from a bit of a prisoners' dilemma to everybody trying to act like a human.

And the best bot trained on that corpus will still sound like a bot.

Exactly. This is a really common mistake, and makes the whole test pointless.

Your Application is rigged and unplayable.

I matched against a friend and we got different results in terms who we were playing against.

https://imgur.com/a/UqSL5DW https://imgur.com/a/Sfjf0Rx

hmm... I checked the server logs & it says your both human

OK, in light of several comments here, I have a theory: there is no chatbot. We've all just been playing each others and the author is getting the logs for a future chatbot that doesn't exist yet. The reveal at the end is random.

The scoring mechanics where people are incentivized to impersonate a chatbot is what makes it not outright unbelievable.

The scoring mechanism, where you lose a point if the opponent correctly guesses that you are human, makes me suspect that is actually a Prisoner's Dilemma simulation disguised as a Turing Test.

The part where you get 2 points if you trick your opponent into thinking you are a robot is even worse. It will incentivize robot-like discussion.

Would be nice to increase the time to a minute, I tried a few times but only got two lines back and forth before having to guess. It was always a human in my tests though so perhaps the bot would respond quicker?

Love the idea though! Hope you can share the results and analysis with us when completed

> perhaps the bot would respond quicker?

Not exactly:

• Me: So, what do you think about this game? • Me: Hello? • Stranger: It's my first time playing • Me: No • Stranger: Yes

• You predicted: human • Correct answer: robot (-1 points) • Stranger predicted: nothing

This is terrible, I was told that the person on the other side will be attempting to impersonate a robot. (This is the scoring.) And that I myself am supposed to impersonate a robot.

So I wrote texts in advance and then copy/pasted the most relevant one, ignoring what the other person said. (To look like I'm a robot.) Some of them (like in response to "how are you") that I wrote were pretty reasonable, other responses by me kind of non-sequitur.

Meanwhile, the other person acted JUST like a robot. So I picked human, since I thought they were impersonating a robot well, as requested. Not being a poorly programmed robot. (As shown.)

The game scoring needs to be updated.

By the way this is my strategy if the incentives were reasonable: I would teach the other person literally anything that I just came up with, and see if they can learn it. Robots just can't do that.

For example I might say (but I would come up with something different each time). "Okay, first I'll teach you something new that I came up with (to prove you're human to me) then you teach me something new you came up with (for me to prove I'm human to you). Absolutely anything new counts. Okay, my new thing is the game of Pirate, I just came up with it [this will be different each time.] I'll say an activity, and you tell me why pirates hate it. Doesn't matter if it's reasonable or not. Ready?"

"Yes".

"Okay, filing taxes."

"Uh ... pirates hate it because... they don't like laws? Or the government"

"Okay good now you".

"Contradict me."

"Okay."

"This game sucks."

"No it doesn't, it's great!"

"Okay we're both human haha."

"Yeah totally."

Now granted this will only work with pretty creative people who can invent a totally new game on the spot. And if any two people ever invent the same new game it might have a false positive. But on the whole I have zero doubt that it works.

I tried it with OK Google. Couldn't teach her anything.

Someone replied

>My strategy works really well - I start telling a role-play themed joke and if they play along they're human. This works great because it's based on common sense, entertains humans, and is a complex social performance that won't ever be the same.

But then deleted their comment. I think it's a great strategy but under the link we're discussing, why does the other person play along if they're trying to convince you they're a robot? If they play along the jig is up.

I guess maybe they meant they do that on something like Tinder (which has a lot of spambots in some geographies) rather than in this actual game.

OK, I think most people have figured it out now:

https://github.com/ashtonsix/fake-turing

It'll stay up for a while if you'd just like to chat.

Time: 0 Me: In Spain, where does the rain stay? Stranger: I dont know the awnser to this question. Me: That's ok, it's a hard question. Do you like spain? Stranger: The humans there speak no english but since i can speak any language it does not matter to me. Stranger: The weather is to hot for my fans. You predicted: human Correct answer: robot (-1 points) Stranger predicted: nothing

I don't trust the results. I would use this tool more if I were confident that the author is not lying to me about the real entity of the person/bot I'm talking to.

I suspect this might be using a coin-toss to print whether the prediction was correct. Some of the test, it had to be near impossible for robot to respond as it did.

Great idea! suggestion: A dead giveaway of robots is that they are able to answer even to messages that I sent just 2s before the end.

At the author: a dead give away for Human is when you see "Stranger is typing...", then nothing, then "Stranger is typing...", etc. a few times, and finally they end up writing something. Could you also measure/model that typing timing and incorporate it in the bot?

That is what I thought the very first time I played, then it turned out I was speaking to a 'robot'. Still I think we have established the final result is not accurate.

My partner didn't reply. From that I guessed correctly that they were a human, but (for a 30 second conversation on an interface your users will be using for the first time) just saying nothing would be an effective low-effort way to implement the chat bot.

The chat bot is broken or there is no chat bot at all.

I only speak English but the few times I tried the stranger spoke another language. Couldn’t actually have a conversation. Cool idea though.

I think it would be better to start trying a children variation for the Turing test. Could it be passed with 6 year old kids? 7? 8?

my chat partner said "fuck off nigger" then spammed blank spaces.

in the end, i guessed he was a human.

the thing said he was a robot.

is this accurate?

follow-up thought:

is it gaming the turing test to the point of breaching the spirit of the test by having the robot be the worst version of humanity as conveyed on the internet?

in other words: is the turing test meant to hypothetically carry over into non-digital media, or should we only expect it to be a useful measurement when it comes to purely digital interactions?

so sorry! it learns from other chats

Yeah, I got the same. Might need to built some sentiment analysis in there too.

I gave up after a couple of times, 30 seconds is too little time for a conversation. On average, there weren’t more than 3–4 messages exchanged in my chats. If there had been a “I’m not sure“ button, I always would have clicked that, since I wasn’t able to collect enough evidence.

The bots seem pretty smart but I've played about 50 games and not been matched with a human once...

when you start playing you get assigned a percentage chance of matching with a robot, so some players will get matched to other humans 80% of the time for example.

I want to measure how this affects the chats.

I got 5 robots in a row :( It get's boring if the best strategy is to wait and hit the robot button.

you can open an incognito window to reset your match chance

    You predicted: robot
    Correct answer: human (-1 points)
    Stranger predicted: human (-1 points)

I'm a robot??? https://imgur.com/a/r0omkVL

This seems to be a really dumb bot.

    Stranger: huh
    Stranger: whatis this
    Me: Good morning. How is the weather there?
    Me: Hello?
    Me: Anyone there?

    You predicted: robot
    Correct answer: robot (1 point)

it learns from other chats, where human players are pretending to be robots

That's the problem. Since humans are incentivized to sound like bots, bots are (presumably) incentivized to sound like humans trying to sound like bots. This doesn't lead to a convincing chatbot.

    Me: alright dude. we gonna kobeyashi maru this
    Me: did you come from hn?
    Stranger: Hello
    Me: site that linked you here
    Me: which is it?
    Stranger: I'm a fake robot.
    Me: Indeed you are

- Convince a human they're talking to a robot: 2 points

Haha, there's no way this was not a human, especially given the timing of the replies: https://i.imgur.com/Z7TTTZH.jpg

I didn’t copy the transcript but I convinced someone I was a robot with something close to:

Me: what is your favourite flavour of chicken soup, chicken or not chicken? Them: chicken Them: what about you? Me: definitely not chicken

What's it like to hold the hand of someone you love? Interlinked.

Interlinked.

In IE11 I don't see the buttons to say if the opponent is a human or robot, so my prediction was "nothing".

increased time from 30 seconds to 1 minute

Tried three times, got 0 lines. This seems pretty pointless, I'm afraid.

The chat bot is broken or it does not exist at all. Everybody is human.

Weird I get the exact opposite experience. I think the matching algorithm is broken.

When you start playing you get assigned a percentage chance of matching with a robot, so some players will get matched to other humans 80% of the time for example

Connected - got stranger is typing.

They put "fg", then "ffg", then endless, endless carriage returns.

This needs some filtering - currently useless.

Tried again - we got a total of 4 lines, 2 each, and the 30 seconds was over.

Cute idea, completely unengaging for me, I won't go back.

At least you got someone to talk to. I got "no other humans online" and told to come back in 59 minutes and 39 seconds...

Anyone know the significance of the time? Or is it just trying to be more effective than saying an hour? (In the same vein as parking lots that have speed limits of 4 mph rather than 5 mph so that motorists are more likely to actually watch their speed because it's different)

I met a human. I wrote "Who am I" and then "Who are you" and then s/he said "What is life" and I said "Cats". This made him/her think I was a robot :- ) +2 to me

It would be fun to see a pic of the other person, and to chat for 90 seconds instead of just 30. And maybe continue chatting for 90 seconds afterwards.

it asks everyone to come back at the start of the next hour, eg if it's 13:45 it'll ask you to come back in 15 minutes

This is more of a "Can you fake being a robot?" test.

seems a bit buggy, I never see the messages others are sending, real or robot

This was just irritating.

Reply on news.ycombinator.com