IBM's "Watson" finally ready for prime-time Jeopardy
hpcwire.com
hpcwire.com
When Deep Blue beat Kasparov I was a kid in an Italian high school, trying to explain to disinterested fellow students why that was a big deal.
Not even in my wildest dreams I would have imagined to be working for the company that made that and this event possible, in a different country than my own.
Sorry for the slightly off topic comment, but I just wanted to remind everyone that the future can be a beautiful and surprising time if you stick around long enough to witness it.
Some like the raw power of the Saturn V, I think the multi-talented Lunar Module is much more amazing.
The reason is that when multiple contestants know the correct response in Jeopardy!, it all comes down to reflexive timing. It's no surprise that a computer could buzz in faster than a human being, and there's no evidence that Watson knows more correct responses than the human competitors.
THATS FUCKING HUGE!!!
Again, I fully understand and appreciate what an advance in natural language processing and sheer computation Watson represents. Even being able to match a below-average human Jeopardy! contestant would be impressive. However, the true game-winning technique in Jeopardy! (when all three contestants are highly skilled) is buzzer reflexes.
It's clear that this is a huge achievement. However, it is different than a computer beating the human chess champion. All this will prove is that a computer is about the same as the best humans, not strictly better than them.
(And, that it is strictly better at timing its buzzer response, which is completely not impressive for a machine to excel at.)
Think of it this way: if you had a human standing in Watson's spot, with a monitor displaying Watson's suggested response, I don't think that human would perform as well against the other human contestants. Sure, Watson would often come up with the correct response while the clue was being spoken by the host, but the human would still have to buzz in when the light turns on. If you buzz in too early, you get locked out for something like a quarter of a second. Watson would still be extremely awesome technology, but I don't think a human with access to Watson's output would compete at anywhere near the level that Watson did in the video.
Do you see what you're saying.. "contestants of this skill level"? It's Ken Jennings, Brad Rutter, and WATSON. Discussion over. The fact that there is a computer that can answer questions in 3s that put it at the level of the two greatest trivia players in history is THE story.
Some jeopardy answers:
Anagramed Animals -- A furry little pet: the rams
Some say the Bush administration's domestic spying conflicts with FISA, the Foreign Intelligence this Act
"R"2"D"2 -- More rubicund
Half the people in the US probably couldn't answer any of these at all, much less necessarily even know what the answer was referring to, period.
The fact that the computer may have an advantage at pressing the buzzer against the two best buzzer pushers of all time is a pretty small deal. IMHO. :-)
This is the biggest innovation in computers in my lifetime and kids are like, "but its no Twitter".
Watson's human-level performance at Jeopardy comes from the combination of highly superhuman data retrieval with highly subhuman language processing.
Watson is more a spectacle than an innovation.
As soon as AI succeeds at something, it is simple.
Marvin the Paranoid Android, in HHGG
The additional fact that this computer can beat human players is not impressive to me, because I know how Jeopardy! works. To sum everything up: it's a great feat of artificial intelligence and computer performance for Watson to generate correct responses so quickly; it's not a great feat for Watson to be able to buzz in faster than human competitors. I think IBM could have chosen a better sort of competition to truly show off Watson's abilities than a competition with reflexes as the final layer of competition.
"The best human contestants don’t wait for, but instead anticipate when Trebek will finish reading a clue. They time their “buzz” for the instant when the last word leaves Trebek’s mouth and the “Buzzer Enable” light turns on. Watson cannot anticipate. He can only react to the enable signal. While Watson reacts at an impressive speed, humans can and do buzz in faster than his best possible reaction time."
I suspect if anyone is good at anticipating the buzzer it is Jennings and Rutter -- the two best Jeopardy players in history. At least until next week...
The eye to finger path for humans is about 200ms. It probably takes about 100ms for Watson to physically press the button. So Watson is about 100ms faster. But that also gives humans about a 100ms window in which to beat Watson. This means that you need to start your press 100-200ms before Trebek finishes his last word.
That's pretty good sized window for most people given you are reading the question along with Trebek. If the person who turns the light on is very consistent, I think a human who is good at this could consistently beat Watson.
Chess, in contrast, is a totally deterministic game and the game state can easily be fed into a computer. That computers are able to beat humans at that comes as no surprise.
Some of the early matches I got data from were downright funny. Lots of nonsense answers and weird correlations that kinda made sense but made it obvious Watson didn't really understand the problem space.
In case you're interested the avatar gets realtime data from Watson and visualizes both Watson's internal state and the game state. The core bit is a collection of "threads" that swarm around the surface of a sphere. The speed, variability, color, and length of the threads are all tied to the data we get from Watson. The colors roughly correspond to confidence, and when the threads bunch up it has to do with what Watson is "doing" (i.e. if he gets an answer wrong the threads will go slow and gather to the bottom of the sphere, if one of the other contestants is answering the threads will gather on that side of of the sphere, etc).
The designer and my team agreed early on that there would be exactly 42 threads. ;-)
Oh, and when Watson speaks the threads push off the surface of the sphere to the intensity of the audio. It also makes a subtle glow in the center of the sphere brighten in an homage to Hal.
Watson can do amazing NLP (and presumably Machine Learning), which is something that the general public perceives as straight up "AI". NLP has been lagging far behind expectations for decades, but with Google's new Translate apps and Watson competing on Jeopardy, it seems like NLP is pretty close to being fully solved.
Very exciting. Very, very exciting.
Edit:
> Eventually the machine will prevail.
This sends chills down my spine.
Cleverer and cleverer puppets will not become intelligent machines. A dedicated effort to build a massive neural cluster simulation might become intelligent, but it will think such incomprehensible thoughts as to be fairly useless to us (think intelligent rhododendon; what would you have to talk about?)
We will know when Watson is dangerous when it feels fear, angst, want. Not just an algorithm to sort facts and simulate speech.
> We will know when Watson is dangerous when it feels fear, angst, want. Not just an algorithm to sort facts and simulate speech.
Nah. Look at good science fiction to see why this is false. The most dangerous AI ever conceived might be something like Skynet or HAL 9000; machines that are cold, calculating and have no emotion whatsoever. But they do hold weapons, they do speak English and they are intelligent. I'm not suggesting we'll ever build anything like what we see in science fiction, but good writers often are able to predict - or even influence (see Clark, Asimov) - the future.
Algorithms to sort facts are indeed the beginnings of machine intelligence.
Small bits of it can be algorithmically simulated. Large processes can be algorithmically simulated. But to call the algorithm "intelligence" is sympathetic magic.
Algorithms work in a different way; they break in a different way; they are hard-coded so don't change. They are a simulation.
I was introduced to Watson when I took a sub-project from IBM related to Watson, I had no idea what it was untill taking the contract, at which point I was introduced to Watson and it is an impressive feat, it is with out a doubt the state of the art in NLP AI. Sufficient to say, AI algorithms differ from the static and rigged structures that most developers write for business apps, web or mobile apps (even the good developers that decouple stuff), the scale of difference is orders of magnitude.
In many cases machines learning has the ability to generate new code based on learning and build new connections and new algorithms to deal with learned problems. Applications literally generate new and novel applications and then connect themselves to these new nodes.
This is not stuff usually broached when you just need to credit a line in an account payable or get the sales volume for last month. There is a huge gap between the code used to write business applications and the structures used in AI.
This is not meant as a critique of you or anyone in general, and is not intend to belittle business app, mobile or web developers but rather to help inform people that, those who have not worked in the field of AI, that their understanding of software development has little relevance when applied to the field of AI. It is literally a different world where the concepts are totally foreign to a non-AI developer.
One of my favourite thought experiments in the area being Searle's Chinese Room:
Searle claims through this experiment that strong AI does not exist. The robot in the room doesn't know how to talk Chinese, it just matches up symbols by using an elaborate dictionary.
Turing defined intelligence by the appearance of intelligence. If the Chinese room can make you think it houses someone that speaks Chinese, than that means the person in the room is intelligent.
To Searle everything is weak AI. Just calculation, without knowing what they really calculate. Intelligence is more than appearance, like a hologram of someone, is not that person self.
Pragmatists and functionalists play around Searle's conclusion. For them it is about the behavior, not the system itself. In Searle's vision something like the China Brain (every person in China uses a radio to act as a neuron) is ridiculous. In the functionalist view the China Brain is intelligent and self-aware.
Using Gödel's incompleteness theorems one can argue that no set of algorithms is capable of perfectly modeling human consciousness. A logically correct algorithm can not give faulty output, yet internally conclude that output to be correct. We do not make the same mental steps as a set of algorythms: I can't say this post is correct with a 97.77% accuracy. In fact I wonder if you will respect me more or less after this reply, if you'll believe I lack understanding of machine learning... not if I passed the Turing test for intelligence or not. Calculation != Intelligence.
Like the robot in the Chinese room is still a puppet, that doesn't really understand Chinese. Attaching a radar to a flying drone doesn't make it feel or act like a bat.
Much of AI is still Advanced Informatics.
Small bits of it can be algorithmically simulated. Large processes can be algorithmically simulated. But to call the algorithm "intelligence" is sympathetic magic.
Algorithms work in a different way; they break in a different way; they are hard-coded so don't change. They are a simulation.
By the structuring of his description, it is apparent that he is reasoning from an application or "computer" if you will, developers perspective. My point was that it is flawed to look at AI software as rigid structures, applying traditional development patterns is flawed and does not reflect the realities of AI development. Put simply the description of AI as hard-coded paths and developer generated (implied) algorithms is in no way factual.
As an aside: like Penrose chose quantum physics (a mysterious thing) to explain consciousness (another mysterious thing), and therefor didn't succeed to convince others, so we should guard against using a fuzzy, complex, black-box, dynamic system (a mysterious thing) to explain (or fully model) consciousness and human intelligence.
We just replaced the wonder with another wonder :)
Machine learning is really computational statistics - it applies fairly standard and well understood techniques to fit a function to a noisy data set. Genetic algorithms and neural networks are really fancy words for optimization algorithms - they're merely a set of tools (not unlike hill-climbing) for searching a large space. The de-facto books on AI are PAIP and PPAI. I've read both, and example programs there, while very interesting, are not much different than a combination of reasonably clever techniques.
"Generating new code" is the same thing as generating a data structure and running a predefined interpreter over it. These systems do that, but in a much more restricted way than you imply. They certainly don't design new algorithms in an intelligent fashion, merely use a set of predefined inference rules, not unlike any other rewriting system.
I don't know anything about Watson, but it is a well understood fact that every AI system to date is nothing more than a clever marionette (and it's very unlikely that this will change for a very long time). You can't just throw terms around - show an example. In every case so far a result that initially appears impressive, when understood, is immediately disappointing. They're all clever, but they're a far cry from "self-learning systems" for any reasonable definition of the word "learning".
I don't think that I have exaggerated, the fact that currently AI systems are built by developers with a predetermined set of rules in not in dispute, those rules constrain what an AI system will generate this is analogies to the function of serotonin in the brain, it's level directly affects a factor of our state (happiness, empathy). Machine learning employs the same factor, basic here is serotonin (data) and here is the serotonin regulation mechanism (algorithm) but the machine learns (abuse drugs) to defeat the regulation mechanism, none the less it has to work within the constraints of the system, just as our minds have to work with the constrains of their biological functions. It works with in the constraints of the system to find ways to adapt the system. Whether this is impressive or not is subjective to the observer (I personally think it is). When contrasted to a biological level, it is pretty primitive reward logic is pretty low level when it comes to biology. None the less I think it is impressive.
I think when talking about AI a lot of people confuse consciousness with intelligence. While evidence suggest that intelligence is a prerequisite of consciousness the converse cannot be said. I think we have made great strides simulating the constructs of intelligence on a mechanical level. As for consciousness, we have to master the former before we will know how to tackle the latter. And when most people think about AI they think about the latter which sets a pretty high bar when measuring the state of the art.
I muse that there is a race of AI type intelligences where genius is measure only in their ability to create moving works of arts and any fool can perform advanced mathematics and exceedingly complex computations involving a vast amount of variables.
For example let us consider that based on time of day and knock you can guess who is most likely at your door. How why, can you know this? Have you learned or figured out anything there?
Or say, I tell you that a person is of gender X, race Y and lives in city Z. You will automatically generate an idea of what this person is like. And it will be different from what I would generate and these data points would likely mean nothing of significance to a 3 year old. Why? Because we have learned a model from our past experience/data. Machine Learning also uses generative models to infer situations. And in fact, we humans perform a very weak form of machine learning. It goes by the name of stereotyping or profiling.
When you are trying to figure out something. The process is not some clean logical step by step deduction. It is more like a search with dead ends (local optimums), back tracking and restarts. Trying and throwing away different ideas. Or when trying to learn a new sport, dance or flip. You do not consider the physics of the situations to try and figure amount the correct amount of impulses to apply. You try again and again to learn or statistically generate a satisfactory approximate local optimum of the correct physics model for the situation at hand.
As for systems which generate code. We can look at it most literally in terms of those which evolve rules in some way or loosely by considering that all machine learning does is use lots of data to prevent the programmer from hand coding a giant restricted system. Regardless of your stance, these systems differ from mere rewriting in that they are not deterministic. They interact and respond to different situations in varying ways. The more sophisticated methods can develop new algorithms - a set of rules - that were not programmed and make no sense to the developer to develop behaviours to cope with their situations. It is true that we provide a base, but that does not mean some limited form of learning is not occurring. What Machine learning cannot do that we can is introspect, abstract and generalize across domains.
I am the reverse of you. Before I picked up machine learning I thought the brain was something special. But now I cant help but feel that we are just clever marionettes and that whats going on is simply mundane mathematics by clever co-opting of physics by nature. I find this fact to be amazingly beautiful.
"I deny that Watson IS algorithms. It is transistors assembled into logic gates. Right?"
My statement is obviously a silly thing to say. Watson is both algorithms and transistors, depending on how you care to think about it. If you believe that your statement is not equally silly, please explain.
It can actually answer Jeopardy questions. As good or better than the best Jeopardy player on the planet. (We will see.) Another task, previously only conquered by evolution, now conquered by algorithms written by humans. Watson doesn’t pretend to be good at Jeopardy, it just is.
I don’t see the simulation. I see a machine that is genuinely good at playing Jeopardy.
I'm pretty sure that Jeopardy performance as such has not been strongly selected for among the ancestors of present day H. sapiens.
(And the related things that may have been -- effective language use, good memory, etc. -- are not so obviously "conquered by algorithms written by humans" however well Watson does.)
I think you mean, it can actually question Jeopardy answers.
Sigh... I always hated that about the show.
No, I most definitely mean that it can actually answer Jeopardy questions. Formulating sentences oddly doesn't turn questions into answers and answers into questions. You could define "question" and "answer" as sentences with a certain syntax but that would be an extremely stupid definition.
Anyway, the answer to this type of conundrum is simple and well-known: http://en.wikipedia.org/wiki/Turing_test
Of course it should be noted that we aren't there yet for Watson. Even in the restricted domain of a game show, when Watson fails, it does so with a different mode, compared to humans, so we can identify that it isn't actually a human.
There's also no guarantee we wouldn't be able to brute force it in time, just as deep blue was mere brute forcing. Impressive technologically, disappointing intellectually.
When I think of intelligence I think of intuition, creativity, flexibility and the ability to learn. The turing test is a simple test along a long path.
I don't think the answer of your conundrum is simple and well known.
Also, what makes you think that what goes on in your own brain isn't just another form of "brute-forcing"? What makes one implementation of thinking "thinking" and another form just "brute-force"? And then, brute-forcing is the technique of trying every possible solution - a Turing Machine must be able to move in a near infinite solution space, so brute-forcing is not actually possible.
The answer to insight into a system's qualia is not a Turing test:
Watson, tell us, do you feel fear? Crunching through some look-up tables Yes!
What did that prove about Watson feeling fear or not? We could have Alice talking with Watson for hours on end about fear and daddy issues, without them even knowing the taste of fruit.
That is why the Turing test is an incomplete test for human-like machine intelligence. You seem to equate feigning intelligence, with possessing intelligence. Feigning fear with possessing fear.
Also, where on Earth do I equate feigning intelligence with possessing intelligence. You're just making that up.
I'm sure the question was asked: Can machines think? But passing the Turing Test doesn't answer this question. It answers a different question: What will happen when a machine takes the part of A in this game? Will the interrogator decide wrongly as often when the game is played like this as he does when the game is played between a man and a woman?
Its up to you if you take an affirmative answer to those questions to also answer the question it replaces: "Can machines think?" The imitation game does not aim to proof anything to this point. It aims to support its replacement questions, which we have to take at face value: that imitation is just as good (for all sense and purposes) as the real thing.
Also, you write: >And how will we know that Watson actually feels fear? The answer to this type of conundrum is simple and well-known: a Turing test.
If I make a chatscript that tricks you into thinking it was a person feeling fear, by your own logic, you would deem that chatscript to be thinking and actually capable of intelligent conversation, even if you were told afterward it was just some lines of code, feigning you with a (if input, output: yes), it still passed the test, so your answer is: those binary bits must feel fear (or you agree that you apply the Turing Test where it makes no sense).
What entities are really feeling and how to communicate this, has no simple known answer. There have been century-old debates about Qualia. The answer certainly isn't an imitation game, if you insist it is, you are equating feigning with truly possessing. (Or you are suggesting I am a P-zombie and I take offense to that!)
If your chatscript can negotiate this type of test, then it would satisfy me that it was thinking.
If you can have an interview with a computer, and an interview with a human, and be unable to determine which is which (accurately, over several trials), how can you argue that the computer isn't human-like? Most versions of the test have ways of eliminating everything but the straight communication of information (because whether you can build a convincing robot or amazing TTS software is a different problem).
How do you know that I'm not a computer?
I prefer the Voight-Kampff test:
1. It’s your birthday. Someone gives you a calfskin wallet. How do you react?
2. You’ve got a little boy. He shows you his butterfly collection plus the killing jar. What do you do?
3. You’re watching television. Suddenly you realize there’s a wasp crawling on your arm.
etc.
The best bots can already carry on conversations better than a low-intelligence human who isn't concentrating on the task.
It's generally possible to distinguish bots by asking more probing questions. The problem with that is as the Watson project shows it is quite possible to build a computer system that can do well on hard questions too.
That leaves emotional-response type questions, but many of those a culturally specific, and so can really only identify something as being either a bot OR a person from a different culture.
My view is that the Turing test is much closer to being passed than people think, if you specify a Turing test that all humans can pass.
This is precisely the reason why I think AI has so much potential.
It doesn't have to be smarter than the smartest human to be of any use. If it's at least as smart and knowledgeable as a fairly dumb or ignorant person, then it could be incredibly useful. What's the use of building a fleet of robot servants if we can't load them with an AI which is at least smart enough to carry out chores, follow orders and communicate about everyday objects. We don't have to load them each with Einstein AI, just Bubba AI would be a big win.
Too often researchers think that AI must be perfect under all conditions. The truth is that we adapt our usage of tools to their limitations all the time, which makes me think that AI-powered vision systems that work well in some limited circumstances would still be very useful.
Watson will answer questions put to it for as long as the power is on. But it will never do anything else. It has no other desires, no motives, no interests. Its planning skills are limited to game strategy. Its conversational skills are limited to preprogrammed social niceties which probably don't extend beyond the context of the game.
You can make a decent case that it's intelligent --- but it's an intelligent machine. You'd have to add all of the above, and probably more, in order to get something that you could interact with even on-line as a person.
[EDIT: I don't mean to deny the possibility of a machine that I would feel comfortable calling a person --- but I saw the Watson demo run, and I haven't seen one yet.]
I played a lot of College Bowl (and its local high-school equivalent) back in my younger days. And there were many running gags about the strategy. For example, if a question contained the words "Name this artist..." there was a strictly limited set of possible answers, defined roughly as "artists whose names appear in a typical high school art appreciation class". Similarly, "name this composer" is more likely to be cluing Beethoven or Bach or Mozart than Grieg or Saint-Saens, and almost certainly not anybody more obscure than that. And Grieg is not that obscure.
There were a lot of common themes: It was a good idea to memorize the names of all the Greek, Roman, and Norse gods; the names of all the types of clouds; the capitals of countries and the names of political leaders. That sort of thing.
In College Bowl (unlike Jeopardy) it is legal to press the buzzer and interrupt the questioner at any point. The questioner just stops reading until you've taken your guess. And so every College Bowl geek knows a story like this: Someone accidentally presses the buzzer before the person reading the questions has said more than one or two words. So, with nothing else to do but guess, the player shouts something out. And that guess is correct.
I've never seen a zero-word guess work in person, but I've seen a few two and three-word guesses succeed. Given just a hint of the topic ("this scientist...") the odds are surprisingly good. The search space is just not that large.
One reason why trivia is easy is that questions that require even the smallest amount of actual thought tend to be too ambiguous for good trivia. Trivia questions are generally read, answered, and judged by people who have no actual expertise in their subject matter, so it's important that the answers be very clear and unambiguous. I used to dread the physics questions, because I was a student physicist, and I understood the questions too well. So it was sometimes difficult to avoid giving answers that were technically correct, but that did not match the one on the card.
It's all about having enough breadth that the other bits of information presented can help whittle down answers by way of association and relationships. Good trivia players rely quite a bit on intuition.
Ken Jennings mentioned (in his book) how Jeopardy! often has questions about rivers, so it was a fairly simple matter to just memorize river names and some basic associations with which to answer questions.
It seems like Watson is mostly a really sophisticated algorithm on top of lots of brute force (2880 processors?!). Though I guess it could also be said that humans are just ugly bags of mostly water.
That said: in something like medicine (differential diagnoses), such a system could also be extremely useful, given the frequent occurrence of a fairly limited number of "answers".
Arguably Ken Jennings is mostly a really sophisticated algorithm on top of lots of brute force (billions of neurons?). And computers are just pretty blocks of mostly silicon. :)
I think you overestimate human interaction. :-)
Look at how many people will have full on conversations with bots. And these aren't even good bots.
An IRC based Turing test will be passed in the next 15 years. The corpus that we have with Google/Bing/Wikipedia add to that emails from GMail/Yahoo/Hotmail, IMs and text message, Facebook messages and statuses.
The computer will not only be able to have conversation, but it will be able to go from talking to you in the Queen's English to ebonics. And will be the most knowledgeable person you've ever met.
Allow me to rephrase that: Eventually the machine will better diagnose your symptoms. More accurately then a human doctor could.
I'll be in my bunker.
That being said, a really really clever jeopardy playing machine just doesn't seem "intelligent" to me. Huge bounds forward, we're making progress, I'm not denying that but Watson isn't going to slurp in the works of Shakespeare and write down some original thoughts on it, comparing it to current events. Or contrast Wordsworth and Keats. Or suggest a new experiment to further identify envelope proteins on a virus. Or invent a new way to etch semiconductors even smaller than we currently are. Will we get there? Maybe, hopefully, maybe even in our life times, but this is comprehension and search, it's not inventing or creating yet. It seems like half the problem, maybe the easier half.
Still awesome, maybe in a few years we'll have Watsons we can access from our phones or something.
There is an archive of past Jeopardy questions here :
Try choosing a question and typing the category + the clue verbatim into Google. I've tried this a few times and in most cases the correct answer was in the top couple of sites (usually in the summary text on the Google search page).
Of course there's still the problem of actually extracting the answer from the page and presenting it in the proper form.
On PBS last night - http://video.pbs.org/video/1786674622
The "Making Stuff" series on Nova that's been airing the last week or so is very interesting: Making Suff Smarter / Stronger / Cleaner / Smaller. Watson was first mentioned during the episode of "Making Stuff Smarter" but I guess it merits its own segment.
4TB disk? something is wrong here.
"Several readers of my blog have asked for details on the storage aspects of Watson. Basically, it is a modified version of IBM Scale-Out NAS [SONAS] that IBM offers commercially, but running Linux on POWER instead of Linux-x86. System p expansion drawers of SAS 15K RPM 450GB drives, 12 drives each, are dual-connected to two storage nodes, for a total of 21.6TB of raw disk capacity. The storage nodes use IBM's General Parallel File System (GPFS) to provide clustered NFS access to the rest of the system. Each Power 750 has minimal internal storage mostly to hold the Linux operating system and programs.
When Watson is booted up, the 15TB of total RAM are loaded up, and thereafter the DeepQA processing is all done from memory. According to IBM Research, "The actual size of the data (analyzed and indexed text, knowledge bases, etc.) used for candidate answer generation and evidence evaluation is under 1TB." For performance reasons, various subsets of the data are replicated in RAM on different functional groups of cluster nodes. The entire system is self-contained, Watson is NOT going to the internet searching for answers. "
What a great ending line. The NOVA special last night was great!
http://www.pbs.org/wgbh/nova/tech/smartest-machine-on-earth....
Now, won't somebody please design a viable successor to the Concorde?
In reality, Ken Jennings was beaten on the final answer.
I'm interested in seeing whether Watson has an "aggressiveness" algorithm that allows it to respond before the answer is fully spoken. Humans have an advantage in this regard because it goes right to the heart of intelligence. If the game boils down to reaction time, Watson will probably win.
You cannot buzz in until the question is completely read out.
I wonder if it's able to say to itself "I think I'm in the right ballpark, so I'll buzz now" and then take the 3-4 seconds it takes the host to recognize the buzz and ask for an answer, to finish its processing to get to the answer it's most confident of.
In other words, it doesn't need to actually have the answer to buzz. It just needs some confidence that the answer is close at hand.
Here's the link: http://www.nytimes.com/interactive/2010/06/16/magazine/watso...
Going forward, the other real questions will be: is Watson overfitted to the problem of solving Jeopardy questions, and how practical is the technology? The former is a real risk to the general applicability of Watson's technologies, the latter is a question of who can afford it. The article mentioned on commodity hardware, Watson takes about 2 hours per clue. They only achieve reasonable response times by using about 3000 cores. That limits the potential audience.
Either way, I'm very interested to see what happens next week. I watched the demo videos on YouTube and it was quite cool.
I'm usually able to read any given question in about a second. Granted, I used to practice speed-reading trivia questions for an hour or so per day (one of the leagues we competed in projected questions via powerpoint), but there's at least one reason most Jeopardy! contestants are able to buzz in almost immediately on most questions.
Wouldn't a more fair match be a series of individual 1-on-1 matches with Watson and Jennings / Rutter?
The current configuration means the two humans will both share the questions that are naturally difficult for computers, but Watson will dominate all the questions naturally hard for humans.
Alternatively, to make it fair, we would need a 2nd copy of Watson competing, and if the two Watson's buzz at the same time, randomly pick one to answer.
,Just to be clear I don't work for IBM, and I do not know their intentions for the project, but I do take projects from IBM and have taken projects related to Watson, but I do not know their plans to monetize Watson, the former is just pure speculation on my part.
That is an impressive amount of parallelism! This is very back of the napkin (and I realise I'm comparing apples and oranges), but a rough estimate for the time taken if the problem was parallelised with 100% efficiency would be:
(2 hours) / ((2880 * 3.55) / 2.6) = 1.83098592 seconds
IBM is more likely to apply Watson technology toward analytics and data mining. There are huge amounts of clinical data locked up in unstructured text reports. If they can analyze that data in a useful way to draw correlations between symptoms, patient demographics, medications, treatments, and outcomes then that could add a lot of value for medical researchers.
http://www.youtube.com/watch?v=3G2H3DZ8rNc
The video goes into some detail, and looks at how Watson analyzes particular questions.
It feels like AI is starting to become what people once thought it could be.
http://www.quora.com/Does-IBM-Watson-Jeopardy-match-contribu...
Also, as pointed out in that thread, this is no different than a regular Jeopardy game. One game consists of dozens and dozens of questions. If Watson answers them all correctly and the humans get zero points, that is indeed statistically significant.
If the score is close, then repetition in games will be necessary. The problem with the assumption in that thread is that one Jeopardy game is vastly different than the next. In reality, you could string together a bunch of games and call it one game. Or take one game and split it up into 50 "games". The point is, inside one game, there are enough different questions to definitely count as statistically significant.
Even on the question level, a disclaimer should be given if Watson doesn't answer enough questions for statistical significance.
Finally, the issue here is with contributing to the statistical ignorance of people watching the two game match. Sure, Watson has played many games already before the match but that's not what's being shown on TV.
I guess from their shift to broadcasting opinions that the "news" outlets saw this coming...
Watson won.
"Alan Turing, meet Alex Trebeck."