Watson crushes the competition in second round of 'Jeopardy'

But the fact that bulk correlation mining can answer even some 'vague and indirect' questions isn't that remarkable. Jeopardy clues are a very constrained domain: short clues in English with some distinctive idioms, and short answers that are drawn from some well-defined and constantly recurring classes.

With true natural language understanding, offline searchable copies of Wikipedia and Wiktionary – 64GB, tops? – could be used to answer almost every question. Instead Watson uses 15TB of RAM and 2880 cores.

The best case I can make for Watson is that perhaps the alternatives shown are each actually the top option based on totally different understandings of the question. So in fact many other plausible answers are folded up just behind its right answer. The shown alternatives are meant to be: if the question means something else entirely.

¹ eg Wolf Blitzer: http://www.youtube.com/watch?v=DVC28oemocA

I arrived at the 6x approximation by googling around for avg. ethernet latencies. I'm consistently seeing numbers of .3 - .35 ms for an ethernet ping/pong. I think it's fair to assume that with the money IBM has invested in this, Watson is on at least ethernet quality connections.

Without some way to account for the fact that the human nervous system CANNOT beat Watson to the buzzer on any sort of consistent basis, the game is far less compelling than it should be.

Here are the pictures: http://www.wired.com/epicenter/2011/02/watson-jeopardy/?pid=...

You'd have to learn it to deal with thick accents like this one: http://www.youtube.com/watch?v=5FFRoYhTJQQ . Honestly, I don't know if that's possible, no matter how much training you'd put into the machine.

Also, this would be pretty artificial (no pun intended), but they could analyze previous all-human Jeopardy! episodes and figure out average buzzer response time, and perhaps incorporate that into Watson.

If Watson has confidence at the moment ringing-in is allowed, it seems it will always win. So how much time it has to achieve that confidence, as a function of how the text is fed, may be more important than buzzer mechanics.

Based on the same general idea of a common starting point that motivates waiting until after the entire question has been read before allowing any ringing in, I could see there being a tiny 'common period' where all buzzes are considered as coming in simultaneously, with the person chosen to answer then being chosen in round-robin fashion (or for maximal drama, favoring whoever is behind). After the tiny common period, it would be strictly based on first-to-buzz.

It would take away a twitch-timing factor that has been important for human champions, too, but offer more fairness with regard to computers and even people with slight ticks or timing problems.

It's less impressive, just like a computer sorting 1000 integers faster than a human is less impressive.

edit: added "not" after the first word

I mean, if they had changed the ring-in system to work in a manner befitting a man-versus-machine match, you're telling me that wouldn't be entertaining? I would be fascinated by that match. This one was a letdown.

But instant button pressing is not terribly impressive and yet is Watson's critical advantage. This match is framed as a battle of knowledge, not a struggle for humans to overcome the massive disadvantage of their meat-based nervous system, which it is.

This is largely glossed over. Watson would lose horribly if one of its engineers was pressing the button. If its electronic advantage on the button was taken away, it would be the kind of fascinating match I and a lot of people were expecting.

Therefore they've made a machine that plays Jeopardy about as well as an average Jeopardy contestant.

That's still an impressive feat but not quite one I would have thought was out of reach five years ago.

They have a ton of priors, a ton of specific optimizations, and a lot of resources.

It's pretty impressive but in a they-clearly-worked-a-lot-on-this way, not a what's-their-secret kind of way, you know?

Whether or not we can make a more interesting or "fair" game of Jeopardy involving such a machine is an entirely separate, and to my mind far less interesting, question.

Where do we go from here? It would seem silly to say "we make a more interesting Humans vs. machines Jeopardy game." Rather, it seems more prudent to figure out ways to expand on this research and use the underlying technology to solve more interesting, and practical, problems.