Hacker News Reader

Comment by AznHisoka | Hacker News Reader

DDG and even Bing/Yahoo both need to index much more of the web if they want to be competitive. You just can't beat anyone in quality when you have fewer results to work with - it's just simple math. I hope DDG uses some of its investment money on solving this problem (as well as speed, but speed is secondary). The bang codes, privacy, etc are all great.

View on HN

isn't ddg a meta search engine (if I remember correctly they scraped google and used yahoo search api)? Or has this changed?

It uses Yahoo, Bing, and a number of other sources (but not google). See http://help.duckduckgo.com/customer/portal/articles/216399-s... :)

Then they are limited to what's in Bing's/Yahoo's index, and their index is vastly inferior to Google's... it's not even close. This is partially why Google has better results - they simply crawl more of the web (plus their algorithm too, of course).

Why can't DDG just cut their losses and invest in building their own index? It's much more worth it in the long run. All DDG is doing is adding cool, nice to have UI features on top of Bing/Yahoo, but what about the core?

DDG's main schtick is that they don't track users. It would be nearly impossible to match Google or Bing's ranking relevance without using signals derived from logged user traffic. Their core principles essentially ensure they won't ever be a serious competitor to the leading search engines, unless a lot of people decide that they're willing to sacrifice a lot of relevance in their search results in exchange for stricter privacy.

I'm quite sure they used to use google in the past

> Why can't DDG just cut their losses and invest in building their own index?

That's far from trivial task.. Scraping and using other peoples results is easy, building and scaling a huge database like search index is one of the hardest problems you can find.

Ok that's fair. But in that case, I don't understand what compelling way DDG is differentiating themselves from other search engines besides a few cool, nice, thin value features.

.. that's almost exactly what I'm thinking too ..

Because right now there are only 2 companies in the world with sufficient technical chops to build a competent end-to-end modern search engine. It's a HARD problem. No startup can do it.

Well for one, results come from far more sources than /just/ bing and yahoo, and they are also put through a custom algorithm. Then there is the 0-click info, official sites, etc. As for building an index, I'm not sure what the plans are there, however there has been some discussion of a distributed spider. Obviously, spidering would mean a LOT of resources directed away from search (don't forget the limited human resources too).

> it's just simple math

More results doesn't always give better results. Generally, they do, but depending on the problem it's often very marginal.

The actual ranking algorithm is more important, especially (I suspect) pre-processing. Google's main advantage (I'd guess) is its expertise in algorithms and processing.

Reply on news.ycombinator.com