Google indexing DDG search results | Hacker News Reader

Google indexing DDG search results

google.com

66 pointsvincent_s16 comments

And?

Their robots.txt allows it [1] -- it only disallows URL's with a question mark in them (i.e. GET parameters) and none of the indexed pages have that:

  Disallow: /*?

And Google is indexing pages like:

  https://duckduckgo.com/Boeing

So not sure what the point of this submission is?

[1] https://duckduckgo.com/robots.txt

Yeah, the https://en.m.wikipedia.org/wiki/Robots_exclusion_standard dates from a more civilized time, when URLs were more, erm, uniformly structured.

Interesting, They block the Wayback Machine's Scraper

Someone found it mildly interesting. And it shows how you can mess up your robots.txt.

Google has a policy of not indexing search results so these will probably be removed although technically there's nothing wrong here.

It would probably be more conspicuous if they didn't.

If you look at the cached Google page, you will see that Google is only indexing the knowledge panel, not the results themselves. E.g https://webcache.googleusercontent.com/search?q=cache:t8Z5T9...

Looks like the query results come from a url that includes a ? and therefore they are blocked by the robots.txt "/*?" rule. https://duckduckgo.com/d.js?q=%C5%8Ctsuchi&t=A&l=wt-wt&s=0&c...

I'd love to see a feedback loop of Google and DDG both looking at each other for factual information to deliver alongside search results.

Have you ever thought, maybe that's how DDG intended it? small time search engines like DDG have no luxury to block any potential routes of people discovering them, especially through Google. I would even bet they are doing "SEO" to optimize their search results for Google.

Is their robots.txt valid? https://duckduckgo.com/robots.txt # No search result pages Disallow: /*?

It is valid, and so are Google's results as the URLs don't contain a question mark.

It almost looks staged.

It's weird that there some subtle defensiveness here. To heck with robots.txt and what not, when a site touts it's privacy, even this, however irrelevant, is slightly eyebrow-raising.

How does this in any way reflect on either the privacy of DuckDuckGo (I don't think they ever claimed or even vaguely implied that if I search for, say, Banana, no one else will be able to see the results page for Banana, as that would be absurd), or Google, who are simply indexing publicly accessible pages, which is what they've always done since coming into existence.

What violation of privacy or hint thereof is happening here?

None. Thus my question of why the Curt responses and passive aggression. DDG has positioned itself as the anti Google and lo and behold Google still finds out what's being searched there, though not necessarily who searched it. If you're a bit cynical, it's vaguely interesting.

Reply on news.ycombinator.com