Hacker News Reader

Comment by emelski | Hacker News Reader

For all the talk of the difficulties of playing Jeopardy! due to the "nuances of natural language" and "puns and double meanings in the clues", that did not really seem to be a factor in the second round -- most of the questions were quite plainly worded with answers easily discoverable just by searching. Accordingly, Watson performed dramatically better today than yesterday, when a larger portion of the questions did have nuance and plays-on-words in the phrasing. Note too how spectacularly badly Watson performed on the Final Jeopardy! question, where nuance _did_ play a much bigger role.

So today, we learned that machines can push buttons faster than people, and search is a great way to find answers for trivia questions. I doubt the former is a surprise to anybody alive in the past 50 years; the latter shouldn't surprise anybody who's ever used Google.

View on HN

This. A thousand times this. Watson absolutely CRUSHED the human players on pretty much every question that was basic facts. I know Watson can probably generate answers faster than humans on simple search stuff, but it seemed so bad at some points that I wondered: is Watson not wired in with some sort of delay that mimics the delay that humans have between deciding to buzz in and actually buzzing in? A lack of such a system would seem to skew the results somewhat.

I was at a watching party with a couple of IBMers who worked on Watson, and one thing they said is that it's not a question of speed, but of timing. Players time their pressing to an estimate of when Alex will finish the question, and Brad Rutter in particular has been clocked at under 2ms with shocking regularity. The advantages Watson has are consistency and the emotional perturbations in its opponents. You could see them getting frustrated, and that likely only served to harm their ability to hit that window between the end of the question and Watson's button press.

You're right: consistently being 6x faster on the buzzer than the common case for your opponent is going to let you destroy them. Their only hope is that you can't come up with a response before Alex finishes reading the question.

I arrived at the 6x approximation by googling around for avg. ethernet latencies. I'm consistently seeing numbers of .3 - .35 ms for an ethernet ping/pong. I think it's fair to assume that with the money IBM has invested in this, Watson is on at least ethernet quality connections.

When a question appears on the screen, Trebek reads it. A human decides when he's 'done' reading it and pushes a button, which makes a light appear. Contestants aren't allowed to buzz in until they see the light, and are penalized if they buzz in too quickly. Watson is also notified of this metaphorical gunshot to start the race, and won't try to start buzzing in before that.

My point is that a human nervous system is MUCH slower than Watson's equivalent. It is probable (don't know, but it seems likely) that it is more variable, as well. As such, under the current rules, Watson has an overwhelming advantage. If all the contestants know the answer before the question is "read", then Watson will consistently beat the humans to the buzzer. The only hope for the humans is that Watson hasn't decided an answer by the time the question is finished being "read".

Without some way to account for the fact that the human nervous system CANNOT beat Watson to the buzzer on any sort of consistent basis, the game is far less compelling than it should be.

Ken Jennings (and I) disagree: http://live.washingtonpost.com/jeopardy-ken-jennings.html#qu...

He seems to be arguing that you shouldn't handicap the computer by removing one of its advantages, but they've already handicapped the humans by removing lots of their advantages: at least in the parts I've seen, there have been no audio or visual clues, and it's my understanding that Watson gets the text of the question over the wire, not even having to OCR the same text that the humans get to read while waiting for Alex to finish reading the question.

I'm an IBMer and I think Watson has been extremely impressive, but as a Jeopardy fan it gets tiresome to see Watson win the race to the buzzer this often.

Finally, I'm hoping that tonight features more of the wordplay-heavy clues that I hear were present on the first night (why oh why was that on Valentine's Day???), because that is the element that excited me most when I heard about the challenge.

http://lesswrong.com/lw/im/hindsight_devalues_science/

That essay focuses on social science, but I think it's still relevant here. Experts in the area did not think they could do this, and even the people involved weren't sure. It's easy to dismiss this as "yeah, it's just a big search engine" once you already know it's been done. Besides not accurately characterizing the approach Watson takes, that sentiment misses the fact that this was an open question.

Paradoxically, people would probably be more impressed if Watson did worse and the game was more competitive. It's like watching an NBA team play a high school team. The NBA team is so good that it looks easy despite the fact that they're that good because of decades of practice.

(Disclaimer: I work at IBM Research, and have associated biases.)

So today, we learned that machines can push buttons faster than people

I'm not gonna lie... it was pretty entertaining watching Jennings squirm every time Watson beat him to the buzzer.

Also, there wasn't any mention of Watson having adaptive artificial intelligence but I would guess it's safe to say IBM was smart enough to include something like that. That in itself would be crazy hard to implement given the magnitude of what it's already doing... but not impossible. Maybe there are a few corrective algorithms in there somewhere.

It doesn't do any training during the game, except that it uses the correct answers already seen in a category to give weight to a specific interpretation of the category title.

Reply on news.ycombinator.com