Hacker News Reader

Comment by RyanZAG | Hacker News Reader

Natural language parsing is not 100% accurate and can make mistakes - breaking news!

All kinds of automated data collection such as this will have errors that need to be fixed manually by a human. This will be taken care of if/when we eventually have real artificial intelligence. The data parsing scripts Google and others use are fairly advanced, but they still break down to a couple rules and finding words in close proximity. There is no real intelligence involved here.

EDIT: The important takeaway here is to make sure we continue to educate people (especially government) as to the issues and unreliability of these systems. Death date showing up on Google isn't a train smash, but if some government agency decides that Google is smarter than them? Then we are in trouble.

View on HN

> This will be taken care of if/when we eventually have real artificial intelligence

Considering that "non-artificial intelligence" makes the same kind of mistakes, I find that an odd claim.

Mistakes happen. Quite frankly far fewer of them happen in the modern world, due precisely to technology like this. What has changed is actually the converse: automation has become so good that we expect it to be flawless. No one would have found it "surprising" (annoying, sure, unless it was in the context of a zazy madcap comedy film) in 1962 to be mixed up with someone else's biography based on a name confusion.

You're completely correct, great points. It's very easy to fall into the trap of 'human-like intelligence will fix it' when thinking of better machines, but it's fairly clear that human-like intelligence has a lot of failings too.

even if the hole knowledge graph would be filled out by humans, there would still be errors in it, thats just how it works with this sort of big data. No way you could design a computer algorithm which does this job without a single error

I find perverse pleasure in your ironic error, i.e. a hole in your knowledge (or potentially typing skills) related to the distinction between "hole" and "whole".

I actually tried to imagine a hole-based knowledge graph... it sounds really interesting.

there would always be some holes in the graph

AI make more mistakes than humans at some things, but only because, like humans, they are just beginning to understand. I might make similar mistakes reading articles written in German, because I am not yet fully fluent in the language. Similarly, NLP AI is currently not "fluent" in English.

I'm reading Jaron Lanier's _You Are Not a Gadget_ now (came out two years ago), and the phrase "but if some government agency decides that Google is smarter than them?" resonates. A major insight of this book is that, in fact, the dominant technological ideology does advance the idea that Google and Wikipedia are smarter than actual people. And we are, in fact, in deep trouble because of this.

It is interesting that google almost seems to take Wikipedia bio boxes as gospel, which is a questionable decision given that Google "rich snippets" in the search results now have such high placement. I wonder how much Wikipedia specific logic is used in the googlebot...I'm guessing that longevity of article and number of edits is a factor in taking an entry seriously.

> It is interesting that google almost seems to take Wikipedia bio boxes as gospel

FTA, I don't believe the data was taken from a bio box:

> So it’s not too surprising that my original Wikipedia entry, as conceived by Joel, was — let’s be honest — more about my father (a famous New Jersey judge) than about me. Joel began the entry with my connection to my father, and immediately mentioned my father’s birthdate and the date of his death.

That's the scary part, as she alludes to in the next paragraph:

> If your name on Wikipedia is followed by a birth and death date, apparently those belong to you from that day forward, no matter whose dates they may be.

Don't know what is so scary about that. I am willing to bet less than 0.01% of "name[s] on Wikipedia..followed by a birth and death date" aren't the actual birth and death date.

Seems like just an oddmatch caused by an overzealous relative that happened due to the incredibly large amount of data Google processes.

Scary because it takes a cry to the internet to get it corrected. While the issue of being declared dead is somewhat innocuous, it could've been much worse -- what if her dad was a convicted sexual offender, and Google chose to associate that with her bio? What do you think that'd do for any job prospects, especially if she wasn't aware that it was present? It could be downright devastating.

Since it is not opt-in (nor is it opt-out), YOU have no control of it. As the author stated, she requested many times, over several months, to have it reviewed and it wasn't until she clamored about it publicly that it was addressed. What if Google never did correct it?

You need to be much more careful in wording your bets.

I think you meant to bet that the simple solution is almost always the correct way to parse the Wikipedia article, but that's not actually what you bet. What you really bet was P(correct_parse) * P(correct_data) + (1 -P(correct_parse) ) * P(wrong_parse_accidentally_giving_correct_data) > 0.9999

A 0.01% error rate is incredibly incredibly accurate. I imagine the error rate of birthdays in Wikipedia bio boxes is higher than 0.0001. Note that, for instance, Wikipedia had a wrong birthday for Jimmy Wales. http://blog.oregonlive.com/siliconforest/2007/07/on_wikipedi...

I _might_ believe that name followed by birth date and death date are the birth and death dates is the correct parsing of the Wikipedia data over 99.99% of the time. However, I doubt your proposed bet would pay off.

I would still stick by my bet as worded. For every Jimmy Wales, there are ten of thousands of people who have no odd circumstances. Combined with the much higher volume of data about recent and more verifiable people than those from a time where records were fuzzier and the fact that Wikipedia is a moving target regarding accuracy, I don't think 1 in 10,000 is unreasonable. Just the pages of every athlete competing in a major league currently would offset most anomalies.

Not sure if it was still the case but didn't google uses to list the extraction sources? I guess the work that was done with google squared also fed into these boxes? I get the impression then that they do use more than one source if possible.

It is a very hard one as the vast majority of the time the text features observed would indicate a date of birth/ death. I guess if the date is not in the metadata/ sidebar on Wikipedia they could flag it and hopefully find another source to correlate with.

It's likely that Google is querying DBPedia.org instead of trying to manually parse plain text on WikiPedia

Reply on news.ycombinator.com