Hacker News Reader

Comment by TekMol | Hacker News Reader

They state that it "achieved state-of-the-art accuracies on the SQuAD dataset (Wikipedia sentences) in early 2017"

So I would assume it has bean heavily trained?

If you look at example questions from the dev set[1], you'll realize that they all use the same words as the sentence containing the answer. Additionally, the topics aren't everyday stuff, but something you'd write a Wikipedia article about. So I guess the model just learns to find the sentence most similar to the question and then selects an answer based on a coarse categorization, which fails when it is presented with unseen situations.

Your example works if you rephrase the question to be more similar to the text:

  Paul gives a coin to Joe.
  Whom does Paul give a coin to?
  Answer: Joe

You can cut the question down to "gives to?" or "coin to?", because that's enough to single out the answer. But as soon as you use s̶y̶n̶o̶n̶y̶m̶s̶ (EDIT: related words) that are not recognized (like "receive"), you have no chance of getting a meaningful answer.

[1] https://rajpurkar.github.io/SQuAD-explorer/explore/1.1/dev/

The "Who did What" dataset seems much better in this respect:

Passage: Britain’s decision on Thursday to drop extradition proceedings against Gen. Augusto Pinochet and allow him to return to Chile is understandably frustrating ... Jack Straw, the home secretary, said the 84-year-old former dictator’s ability to understand the charges against him and to direct his defense had been seriously impaired by a series of strokes. ... Chile’s president-elect, Ricardo Lagos, has wisely pledged to let justice run its course. But the outgoing government of President Eduardo Frei is pushing a constitutional reform that would allow Pinochet to step down from the Senate and retain parliamentary immunity from prosecution. ...

Question: Sources close to the presidential palace said that Fujimori declined at the last moment to leave the country and instead he will send a high level delegation to the ceremony, at which Chilean President Eduardo Frei will pass the mandate to XXX.

Choices: (1) Augusto Pinochet (2) Jack Straw (3) Ricardo Lagos

https://arxiv.org/abs/1608.05457

Yes, that might be a more accurate description. It picks the "who" thingy from the most similar context.

With zero further understanding as it seems:

    Paul gives no coin to Marray. Paul gives a coin to Joe.
    Who got something from Paul?
    Anser: Marray

    Paul gives no coin to Marray. Paul gives a coin to Joe.
    Who received a coin?
    Answer: Paul

Heavily trained on SQuAD questions. There are lots of models out there that are very good at recognizing SQuAD questions, and reverse-engineering the predictable ways that the Turkers who wrote the questions pulled the information out of the paragraph -- allowing them to answer the question without ever understanding it. https://arxiv.org/abs/1707.07328

Reply on news.ycombinator.com