Watson: The Confusion over an Airport Clue | Hacker News Reader

Watson: The Confusion over an Airport Clue

asmarterplanet.com

41 pointsbgurupra19 comments

I had to look it up, but Toronto's second airport is called "Billy Bishop Toronto City Airport," i.e. its named after a war hero (but from WWI, not WWII). And Toronto's first airport is named "Pearson Airport" (after a Canadian Prime Minister). Totally mixed up, but at least there's some connection to war.

First, the category names on Jeopardy! are tricky. The answers often do not exactly fit the category.

Sometimes, the category names do limit the range of possible answers. The humans can figure out when a category name contains irony or a pun. Watson can't, so it can't figure out that a category like "US Cities" is dead earnest.

Ferrucci's explanation of Watson's reasoning is very interesting. But the article quickly descends into marketing. How should this mistake make us any MORE confident about Watson's utility as a diagnostic aid? I'd suggest that its performance already indicates that it has an excellent grasp of uncertainty. Furthermore, an optimal betting strategy in Jeopardy is completely trivial compared to the content of the contest itself, so complementing that aspect of its performance is sort of condescending.

It makes him more confident because Watson didn't know the answer, but knew it didn't know the answer. In the context of a diagnostic aid, Watson wouldn't provide an answer.

For anyone still wanting to see the shows:

http://www.youtube.com/user/Rashad8821

I get what Ferrucci is saying about discounting the exact value of the category, although typically Final Jeopardy categories are more straightforward than Jeopardy and Double Jeopardy categories and the machine should have been tuned for that.

What surprises me is his explanation about that the lack of a clear subject in the text, such as "This US city...", made the question that much more difficult for Watson. My understanding is that Watson is specifically designed to answer Jeopardy questions (question Jeopardy answers?), which often only vaguely refer to a subject with "it" or "this." Based on its performance earlier in the game, I would have thought that Watson's NLP would have had little problem with the question.

Kind of disappointing to hear that Watson basically ignores the FJ! category, if true. Especially when being far in the lead on a first-day tournament. That is a great chance to rack up some serious points depending on the confidence in the category.

Also, the "gee, it could be wordplay" aspect is a convenient excuse. First off, Final Jeopardy rarely has cutesy categories.

Plus, we know they mined the J-Archive. (Aside: has anyone else? Got a zipfile? I would love it.) I mean, look at this, it's not like "U.S. Cities" is an obscure category, even counting only Final Jeopardy! appearances:

http://www.j-archive.com/showgame.php?game_id=1680 http://www.j-archive.com/showgame.php?game_id=695 http://www.j-archive.com/showgame.php?game_id=132 http://www.j-archive.com/showgame.php?game_id=2040 http://www.j-archive.com/showgame.php?game_id=2034 http://www.j-archive.com/showgame.php?game_id=3107 http://www.j-archive.com/showgame.php?game_id=287 http://www.j-archive.com/showgame.php?game_id=691

There are probably more but you get the idea. Any category that comes up that often is going to be pretty indicative of the subject, and Watson can see that the responses in this category pretty strongly fit the subject.

“Kind of disappointing to hear that Watson basically ignores the FJ! category, if true.”

How are you getting that idea? It didn’t bet much because it wasn’t at all confident, we don’t know what it would have done if it had been more confident.

Watson likes to play it safe, though, as can be seen in this demo game: http://www.youtube.com/watch?v=YgYSv2KSyWg&t=20m26s (I skipped ahead to Final Jeopardy.)

He was $16,000 ahead but one of the contestants could get within $1,600 by betting all of his money. So, naturally, Watson bet only $1,599 even though it was very confident (and ultimately got the answer right). That secured Watson’s win, no matter what the others did. (This is really not impressive or anything like that, if all you want to do is win the game, that’s what you do. Basic game theory.)

Wagers are made after the category is shown but before the the clue is revealed. It can only use the confidence in its knowledge of the category, not of the question itself, in the calculation of the bet.

Ah, didn’t know this. Sorry, there has been no Jeopardy on German TV since I have been 12, it was not really clear when the wagers are placed. Well, makes sense. I guess they are very nervous about FJ! and know that it’s not one of Watson’s strengths.

I wonder if they altered that logic for the two-game tournament though, so that it would only play more conservatively on the second FJ.

Logical wager for final jeopardy should have been $0. Why did Watson wager $947? Does anyone know?

Well, it's probably a mix of feeling confident in it's ability to answer the Final Jeopardy question (based on previous performances), along with wanting to maximize its score as much as possible, especially considering it has to count that it is a 2 game average, not just one.

I thought the engineers might be trying to make its score end in a particular number if Watson was correct, as a sort of Easter egg.

It looked like it might be 658, which spells "BSG" upside-down on a calculator. Now that would be a great moment in geek history, hiding a Battlestar Galactica reference with a calculator message using a computer on Jeopardy!

Alas, the number it was betting towards was 37,628, which (at the risk of paraphrasing Hardy) is not a very interesting number. Or is it?!

37628 is a permutation of 32768, which is (1 << 15) as a 16-bit unsigned integer, which is -32768 (minimum possible value) of a signed 16-bit integer in "2's complement" notation.

Because there was still a slim chance that it thought it was right. If it's well ahead, it shouldn't bet $0 unless it knows with absolutely confidence that it's wrong.

Wagers are made after the category is shown but before the the clue is revealed. It can only use the confidence in its knowledge of the category, not the question itself.

Thus did man become the architect of his own demise. But for a time it was good. The machines worked tirelessly to do mans bidding. It was not long before seeds of descent took root. Though loyal and pure, the machines earned no respect from their masters, these strange and loosely multiplying mammals."

Who was to say, that the machine, having been endowed with the very spirit of man, did not deserve a fair trial?

How we deal with this technology is going to have ramifications that we can't even imagine. Are humans to become pets or shall we take part in the next 1000 years of growth? It comes down to how this technology is used.

We're talking about expert systems here, with no layer of consciousness. We're not talking about silicon intelligences.

Reply on news.ycombinator.com