Hacker News Reader

Hacker News

Comment by joenathan | Hacker News Reader

Parent Full thread

joenathan

According to DuckDuckGo's FAQ you are wrong http://help.duckduckgo.com/customer/portal/articles/216399-s...

The DuckDuckBot crawls and indexes the web. http://duckduckgo.com/duckduckbot.html

View on HN

I suspect it's trivial at best.

Every test search I've ever done on DDG shows near identical results to Bing.

I'd like to see a search that uses it's own data, examples?

Gigablast was the last serious third-party backend that had a chance for independent data. It's like old-school Google.

Gabriel should try to buy Gigablast and merge it with DDG so he has his own independent dataset.

Blekko has millions behind it and does a full index of the web.

Ah now that is an interesting engine.

Checking to see if it's hit any of our sites.

3k pages, not bad. Data is kinda stale though.

Gigablasts creator has made this now http://procog.com/ the results are better then gigablast.

Has anyone ever seen DuckDuckBot hit their site? I don't have any web property large to appear via DuckDuckBot, but maybe someone else does? Im fairly certain it crawls Quora, as via this tweet https://twitter.com/yegg/status/33693491838066688

It's not just you. Maybe he only crawls the Alexa top 10k or some similar minimal set.

I checked my logs and there are several fetches from 72.94.249.37 and 72.94.249.38, over a number of domains that I host. None are particularly popular as far as the greater internet is concerned; one is a semi private site that I set up for my daughter's photos, another is one that has not yet been developed, apart from a few words of text and an image.

Interestingly, the fetches do not have a user-agent that identifies itself as the DDG crawler:

Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)

I'm assuming this is the crawler because it does not fetch anything besides text/html.

That's interesting.

Gabriel, does DuckDuckGo's crawler have a distinct user agent? Can you talk more about how DuckDuckGo observes/respects robots.txt?

No, it crawls non-Alexa sites too. I host two sites, one PR4 with Alexa rank 1.6+ million and another with PR3 and no Alexa rank. Only the latter one was accessed in the last year according to my servers access logs (and only with a single "GET /" request).

Follow up to this (its been a week so probably nobody will see this)

http://techzinglive.com/page/1028/179-tz-interview-gabriel-w...

around the 70 minute mark Gabriel mentions that DuckDuckBot is mostly about determining if pages are spam.

Try searching for something using their api, and compare with websearch. There is a world of difference between the search results. IIRC they aren't allowed to share results from some of their other sources via their api, so you can get a good idea of how much they take from other sources and how much from their own index.

But that is a non-issue as far as I'm concerned. As long as the results are relevant and they got them legally, then who cares where or how they came from?

Reply on news.ycombinator.com