Creators: Watson has no speed advantage as it crushes humans in Jeopardy | Hacker News Reader

Creators: Watson has no speed advantage as it crushes humans in Jeopardy

arstechnica.com

29 pointsevo_927 comments

Here's the meat: "They're not waiting for the light to come on," Welty said; rather, the human players try to time their buzzer presses so that they're coming in as close as possible to the light. Though Watson's reaction times are faster than a human, Welty noted that Watson has to wait for the light. Dr. Adam Lally, another member of Watson's team, noted that "Ken and Brad are really fast. They have to be."

What a deceptive little statement. If you set the "attempt answer" bit in Watson before the light comes on, and you have the humans attempt to press it as soon as allowable, they will never beat Watson.

Actually, they may beat him a few times per hundred-thousand attempts (assuming Watson gets the "light" and buzzes in via tens-of-microseconds-scale ethernet), but would have to be willing to tolerate what would normally be an unacceptable error rate (if you press early, you can't press again for ~2 seconds if I remember right).

If you set the "attempt answer" bit in Watson before the light comes on, and you have the humans attempt to press it as soon as allowable, they will never beat Watson.

There were at least a couple of questions for which Watson had 90%+ confidence for the right answer and also way above its buzz threshold but could not buzz in.

Heres one with 65% confidence: http://www.youtube.com/watch?v=HR2_M8kL_3o&t=3m58s

97% confidence: http://www.youtube.com/watch?v=HR2_M8kL_3o&t=4m26s

Watson doesn't decide instantaneously; did it pass its buzz threshold before or after the 'light' went on?

Indeed. And I suspect the light comes on as the result of some human process – a person offstage following Trebek's cadence – so it has an inherent variance.

If that variance is above a certain threshold compared to Watson's light-triggered reaction time, then even a 'perfectly' accurate (on-average) prediction of the light-activation time could be dominated by Watson's precision, and rarely win any buzzes. (The lock-out for being a little too early – a mistake Watson can never make – makes it even harder.)

Watson presses a mechanical buzzer, it’s not Ethernet.

(I don’t know the reaction time on that and would like to know it. If read about anything between 1ms and 200ms.)

Actually, although he buzzes in mechanically, he gets the "light" signal electronically. That would give him about 100-200 ms advantage over a human nervous system reacting to a visual stimulus.

Well, obviously, but implying that it’s all ethernet is simply wrong.

The humans can predict when the light will come on before it actually does based on when Trebek finishes reading the question. Watson can't, and that's what Welty meant. Also, Watson buzzes in via the same button the humans do, not electronically, so there is a window after the light comes on but before Watson can physically depress the button.

That said, it was obvious last night that in practice Watson has a significant advantage in buzzing in. The supposed advantage of predicting when the light will come on doesn't cancel out Watson's reaction time advantage. I'm guessing the reason for that is there's a significant variation in the time between Trebek finishing reading the question and the light coming on (because the light is triggered by a human), making it hard to predict.

Well, opinions differ then, because Ken Jennings said halfway through an NPR interview yesterday Watson has the "advantage of speed which no human reflexes can match."

Plus, come on, everyone who watched the match so far knows they're just trying to put some spin on this. The reason why humans don't wait for the light is because their nervous systems are too slow!

Watson doesn't have that problem. If timing and rhythm were so important, they would have Watson do that too. But we all know they don't need to.

Speaking of reflexes, it reminds me of those games where your supposed to click/hit a button once it says go. I did a quick search and found this pretty simple flash reaction game: http://www.opendb.net/media/content/426.swf

Best I could do was .26 seconds =/

I bet someone with a Lego Mindstorm set could whip up a robot that would do this much faster.

Got .03 here

But that was after I figured out that you could just continually click the button and score much lower. Advantage humans.

You should have kept trying. I got .01 second by using your technique. http://i.imgur.com/PG92y.jpg . I recently had to go to a nerve specialist and we talked about the speed of the electrical pulses that go through your body. He told me that it is nowhere near the speed of electricity in wire - more on the order of meters per second, not miles per second.

The reaction timer was interesting -- I managed to get 0.16s in one attempt out of ~15, with most attempts ranging from .25-.35. It reminded me of exercises they had us do in driver's ed in high school to show how reaction time+braking distance was affected by alcohol.

Nice timer. I managed to hover around .25 by staring at the background color rather than the button itself. Unsettling to notice the time gap between my brain perceiving the color change and moving the finger/clicking.

I think the bigger issue is the fact that they 'feed' Watson the question via a text file.

That was probably the most (only?) disappointing thing they revealed. It would seem that voice-recognition is far enough along where that shouldn't be necessary. Maybe adding that would delay Watson just enough to seem far? Or perhaps I'm over simplifying the difficulties of adding voice recognition to Watson?

The interesting question is whether the computer can do natural language processing, not whether it can recognize spoken words.

Even the human contestants don't rely on the host reading the questions; they read the board. I guess you could argue that Watson should do OCR of the board, but that's not actually a hard problem, so it wouldn't really add much to the challenge.

They're focusing on language processing, not environmental sensors.

Right, perfect - that is actually more fundamental and exactly what I was getting at.

Is the game currently fair? It's hard to say but watching it seems like there is an advantage there for Watson.

I do like the way Jennings downplayed it though. I think he was being almost too kind but compared to how Kasporov handled himself, this is hugely better.

I haven't seen the video yet but I think the facts will show that the game is reasonably fair, given the different species participating.

Whether or not there is an advantage for Watson is, I suppose, rather the point of the exercise ;-)

Of course it's not fair. The question is whether Watson's advantages are offset by its disadvantages.

Humans on the show don't necessarily have to do voice-recognition either. The text is presented to them on the screen, and I'm sure they can read it faster than Trebek.

Exactly. It pops on the screen instantly, even if there's a video attached.

A contestant with a bad eyeglass prescription is a goner. Waiting to hear to the clue puts a player at a major disadvantage, especially on really short clues.

However, there is one very notable exception: Eddie Timanus.

http://en.wikipedia.org/wiki/Eddie_Timanus#Jeopardy.21

i wish watson had to do speech/text recognition of the answers. it would slow down watson's response and put it on par with the human contestants who don't have the answers hard-wired directly into their brains and still have to spend time reading or listening to alex.

I doubt that OCR on the nicely formatted question screen would slow it down appreciably.

I suspect they could have done this, the technology certainly exists to read the screens and parse text from an image. They could also have tried to recognize when a human gives a wrong answer, like with the 20's question, and presumably the same logic used to play Jeopardy could interpret that statement and not give the same answer again. Watson took four years to develop and is the first of its kind. I suspect that something fancier never would have been finished, and too many features would have been a distraction from what IBM has actually accomplished.

No, it wouldn't.

At the highest levels of play, all players know most answers, and are prepared to ring in as soon as the capability to ring in is given to them. In other words, at this level of play, all humans are racing for the buzzer. The fact that Watson may know the answer before the contestants are allowed to ring in is not an actual advantage for Watson.

(The mechanical button-pushing device is.)

One thing I haven't seen mentioned is that Watson is going to be consistent for the whole match. A human is going to vary in response time if their attention drifts just a little or their hand slows (maybe an itch, maybe a cramp).

Reply on news.ycombinator.com