Hacker News Reader

Comment by dontnotice | Hacker News Reader

Halfway down that piece:

A study this year by Stone Temple, a prominent analyst of the industry, showed Google’s search engine answered 74.3% of 5,000 questions, and on those answers it had a 97.4% accuracy rate. Both percentages are higher than services from Amazon.com Inc., Apple Inc. and Microsoft Corp.

Alternative WSJ:

Let's bury that tidbit under the fold while we nitpick and highlight edge cases in a constantly improving system.

We should also emphasising how integral google's search is to the health of society and civilization because we are mandated by management and ownership to produce anti-google PR and provide a steady supply of ammunition to its critics.

View on HN

That's absolutely the wrong way to go about looking at it. Not all facts are of equal importance. 97 correct facts about, say, gardening, and three incorrect facts about first aid is not a successful system.

If Google is presenting the tool as a canonical source of truth, it needs to be right 100% of the time. In fact, being right 97.4% of the time is worse in many ways, as it lulls people into a false sense of security about how much they can trust the system. Get 97/100 obvious answers correct, then give someone wildly inappropriate advice when they ask something more "off piste".

And let's not delude ourselves here, Google isn't doing this out of an altrustic desire to help people. They're making sure fewer people leave google.com, in the process starving the very sites they're getting information from of revenue. That should worry all of us.

> we are mandated by management and ownership to produce anti-google PR and provide a steady supply of ammunition to its critics.

This feels like an absurd criticism. Couldn't it apply to absolutely every negative thing written about Google, ever? Why should Google be immune from criticism?

And that’s a ridiculous standard because there is no such thing as 100% when talking about knowledge. Humans and experts wouldn’t agree 100% of the time so it’s impossible for computers to get there.

So why create a feature that pretends such a thing is possible, and promote it as heavily as Google does?

In many ways, "I'm feeling lucky" was a great middle ground - by clicking it you're implictly saying "I know this might not actually be what I'm looking for, but I'm willing to compromise". Google has taken that compromise and stuck it at the top of the search results page.

For profit corporations require that the senior employees they hire for impossibly high paychecks are constantly providing benefits that can be brought up in quarterly board meetings. This means taking an already functional product and making changes to it -- any change -- that can be spun as being a positive improvement worthy of a promotion.

Google Search is already where it should be. Yet Google Search employs thousands of incredibly expensive engineers. Something has to be done, and that "something" is rarely going to be good, and almost guaranteed to be something nobody actually wanted.

That seems like a question with a really obvious answer.

Because it is useful, makes users happy, and generates revenue. There really isn't another useful standard to apply here.

> There really isn't another useful standard to apply here.

...accuracy?

Good luck turning that into an objective standard for a search engine.

Is your argument "Google should not be allowed to do anything if their algorithms aren't 100% flawless and perfect"? I really am failing to understand your viewpoint.

No, my argument is "Google should not present results as if they are an absolute answer when they are not capable of knowing whether that is true".

It's pretty simple: a list of search results does not imply certainty. Injecting a single "answer" does. As I mentioned elsewhere, it's as if they have decided that the "I'm feeling lucky" button should apply to everyone.

I guess I fundamentally disagree that them providing an 'answer' is somehow unethical, unless there is some kind of verbiage on the feature that claims 100% accuracy that I am unaware of.

But is google even implying that the results are the absolute answer?

If I search for "how tall is tom cruise" and it gives me a number. It doesn't say that number is an absolute answer, it doesn't say that it's verified, it just shows the number.

I personally don't see that as any different than if it returned a few websites, all of which say the same thing when I go to them. In all cases it's "Google" giving me the answer (an evil google could just as easily return websites with false results on purpose), but the way it currently works, it gives me the answer faster and in a better format. And even if that answer isn't 100% factually correct or verified in any way, it's still the same quality I would have gotten from google in any other method.

If the answer is the same using both methods, wouldn't the only real solution to be "refuse to answer the question"?

in the case of unattributed information, i would think some edge cases risk lawsuits (libel or other things) or pr problems if the information is wrong.

imdb was sued for revealing a person's age; and while imdb won the case, a law in california was passed that dealt with the matter. what happens if google displays information about you that you feel is private (or some legal jurisdiction asserts is so)?

i noticed that "what should i do if bitten by a snake" does have attribution. however, it seems to me that the information is presented in a "this is the answer" way, ie as trustworthy and actionable... what happens if following whatever google suggests results in harm or death?

either way, i'm surprised google hasn't bothered to add couching language or some notional caveats. (even a "here's what we found:" seems reasonable distancing.)

Yes, this is just as misleading as NSA saying "we only intercept 1% of the world's traffic."

Yeah, because 99% of the traffic is not communications, and most of it is video content and torrents.

>This feels like an absurd criticism. Couldn't it apply to absolutely every negative thing written about Google, ever? Why should Google be immune from criticism?

I think this is more a direct attack on WSJ's questionable neutrality rather than a strong defense of Google's behavior.

>If Google is presenting the tool as a canonical source of truth, it needs to be right 100% of the time.

Should we burn all the dictionaries and encyclopaedias because they aren't perfectly accurate? Textbooks don't have a black-box warning on the front page saying "The accuracy of this book cannot be guaranteed. Please verify any facts stated before storing them in your long-term memory or using them for any purpose"?

"Truth" is an inherently messy concept. The very best curated sources and human experts are frequently fallible. If you assume that Google (or any other source) is infallibly accurate, then the problem lies with your education, not the source.

Google is not an encyclopedia or a dictionary, though. Encyclopedias are editorially curated products where each entry is researched and chosen very deliberately. The answers Google provides are picked by an algorithm and have nowhere near the level of oversight that dictionaries and encyclopedias do.

> "Truth" is an inherently messy concept. The very best curated sources and human experts are frequently fallible.

Here, we can agree. My objection is to Google presenting their tool as if it does provide one true answer to your question.

"The problem lies in your education" is a fantastic way of absolving responsibility. What if people's education about tech is in fact exactly the problem? Do we shrug our shoulders and say "well, they should all know better" or do we proactively try to make the situation better?

Yet Wikipedia is one of the greatest stores of knowledge on the planet, despite being full of inaccuracies.

How many Wikipedia articles are one sentence saying "this is the answer"? None. If anything they're padded out to excess, detailing the different perspectives about any particular topic, complete with referencing and footnotes.

I was going to take issue with you're "this is the answer" description but sure enough, on the featured snipped "About this result", Google interchanges the word "answer" and "result" pretty arbitrarily:

When a user asks a question in Google Search, we might show a search result in a special featured snippet block at the top of the search results page. This featured snippet block includes a summary of the answer, extracted from a webpage, plus a link to the page, the page title and URL...

Wikipedia came to mind when this topic of 100% accuracy came up, a metric people seriously think Google needs to hit in order to not cause mass hysteria, it seems. Wikipedia is constantly updated because the information is curated by humans. Humans have the ability to lie, omit facts based on beliefs, fudge numbers to paint a specific narrative, and so many more egregious examples of ways to mislead those who would use the resource. I understand having high expectations for things but just like with anything fact based there needs to be a level of skepticism and self policing of what we allow to become things we know to be true based on our own acknowledgement.

This is something I feel long time internet users have built up a tolerance to and an eye for. Of course I understand wanting to hit that 100% metric for those unfamiliar with the concept of others steering folks in the wrong direction purposefully but how can we honestly draw a defined line in the sand to gauge a systems usefulness? Especially when that system's data is based on the concept of human knowledge, an ever changing, rapidly developing, and hotly contested part of the human experience?

If an encyclopedia said the Earth is 6000 years old, and starts "Evolutionists fallaciously think that billions of years of time makes particles-to-people evolution possible", as the top hit when Googling "how old is the Earth" says, then yes, throw out that encyclopedia.

Being fallible is one thing, but Google's algorithm is misrepresenting scientific belief because an extremist group are the only ones who will pay for SEO about the subject. If Google wants to be the world's authoritative source of knowledge, they're gonna need a more advanced system where SEO spammers can't buy the truth, otherwise all we have are modern day iterations on Phillip Morris' research showing no link between lung cancer and cigarettes for decades.

> Textbooks don't have a black-box warning on the front page saying "The accuracy of this book cannot be guaranteed. Please verify any facts stated before storing them in your long-term memory or using them for any purpose"?

I just checked a 2004 medical textbok I had nearby (don't ask). YES, they do have such disclaimers, and they're worded even better than you put it.

> I just checked a medical textbok I had nearby (don't ask). YES, they do have such disclaimers

Most textbooks do not. Medical textbooks (and websites, etc.) often do because of special circumstances (legal and practical) applying to that field.

A few levels up, medical facts were the example used for needing disclaimers. The point is that Google isn't putting up disclaimers even where they should.

> ...97 correct facts about, say, gardening, and three incorrect facts about first aid is not a successful system...

General information sources that include but aren't specifically focussed on medical information don't usually have disclaimers on the medical information (e.g., encyclopedias don't have disclaimers on entries that happen to concern medical information, while medical textbooks do).

Science and medical textbooks should have a "best before" date on them.

That would be the copyright date of the book, combined with medical licensing requirements that doctors stay up-to-date in their field.

Just imagine if software engineers had the same certification requirements as, say, pilots. Or, hell, plumbers.

I want to say that would be fantastic, except that the software engineers might be pretty miserable.

Also, aren't there such certs already?

I dug up the study by Stone Temple to see what kind of questions were asked (eg: are the questions really trivial or not) and they provide a few exampled in their write up. I am actually seriously impressed by how well those "personal assistants" work compared to what I would expect. Kudos to the engineers who built those things!

(Disclosure: I never worked on any of those, nor do I work for any of the companies making them)

97% accuracy is abysmal for something you are presenting as "answers".

It’s good enough for my professor :)

Not sure what the margin of error is on these things, but generally 97% should probably be considered 100%.

There's no margin of error for the test they did, just potential sampling bias.

Given the 5000 questions they asked, the system will provide the wrong answer 2.6% of the time. Every time, until they improve it. There's a chance that they managed to ask the only 150 questions that it doesn't know the answer to, but not a very big one.

I agree -- given the sample space, there is not effective way to calculate the margin of error. Even so, I don't think that there are many examples of non-deterministic mechanisms that produce the correct answer 97% of the time.

a parser that is 97% correct is broken.

A parser (assuming you are talking about a programming language parser) has the luxury of having highly structured and deterministic inputs, and to be able to refuse giving an "answer" if they are not.

> highly structured and deterministic inputs

not all parsers deal with such inputs, and parsers that don't can and often do produce multiple interpretations of the input data.

Tell me of some parsers that do not deal with deterministic inputs and have 100% accuracy, then.

It's like Intel's floating point math used to be about 97% correct. Good enough, right?

https://en.wikipedia.org/wiki/Pentium_FDIV_bug

What you/we need is a healthy dose of taking things with a grain of salt and not just blindly trust what we read/hear.

This goes not only for results in search services today, but was also valid in the time before this, when we used encyclopedias, or asked someone.

I'm not sure why you feel the need to comment on how credulous I might be.

Anyway, the 97% is an impressive result, it just isn't so impressive as to be beyond criticism. My comment is a little over the top, in response to the other comment that sets the bar at effusive praise being the only proper analysis of the system.

This is not about "a grain of salt". Imagine you're a ten year old kid and you're looking for answers on some subject, say the Holocaust. What if all the "answers" you're getting are from far-right hate groups? How do you think that will affect that person?

Likewise, if you're looking for medical advice and you're getting nothing but anti-science woo, can we really be surprised that anti-vaxxers are becoming more numerous? These outbreaks of deadly, yet preventable diseases have serious fatal consequences for many.

"Grain of salt" means fuck all when people are dying from bad answers. You can't learn from your mistakes when you're dead.

At least Google could frame these with the idea of keeping a skeptical mind, but they should probably stop surfacing things as "the answer" and instead as "top search result".

If there was someone I knew who I could ask a question, with a 97% chance they were correct, I'd consider them to be a phenomenon, and an extremely valuable resource. For some definition of correct, I'd be happy to be 60% accurate.

Yeah but what if you got to choose between 2 people that knew the answer 50% of the time but one of the them enjoyed making up answers when they didn't know and the other just told you they didn't know?

Philosophically speaking, assuming there is always a single answer for any question and promoting a specific SINGLE answer for them is not only inaccurate and misleading, sometime it's WRONG! The world is complex, many questions will and should have multiple answers depending how you look at it.

Google used to be a place (when it was really new) where you would always find what you were looking for within the 10 first responses because of the power of it's "Pigeon Ranking". I'm not claiming 97.4% in all cases but really often "I Feel Lucky" would be the right answer. Slowly this got gamed by people that wanted to have their sites there and this magic disappeared.

I imagine these selected answers will either come for a dwindling amount of sources of "truth", turning Google in a sort of Yahoo! with lots of handpicked results or again these algorithms will be gamed and we get the same mess as Google turned into where it was constantly fighting new waves of spam.

Reply on news.ycombinator.com