Hacker News Reader

Comment by finnjohnsen2 | Hacker News Reader

I should use DuckDuckGo, but when I need to search I'm too much in some mindset and context I can't allow it to get broken by the poor search results DDG gives me. So my life has come to the point where I'm aware that I give all my search data to someone I know spies on me. 24/7/365.

View on HN

Just search more specifically, this should resolve 80% or more of the 'poor search results'. Every once in a while, I'll go to Google but that's for maybe 2 searches a month! You'll be surprised at how decent the results for DDG are.

That works when you know what it is you're looking for.

Google still wins when you don't know the word/term/name etc for something and are describing it. The reason for that is mostly that every time someone searches on Google, they help make Google smarter. Effectively every time you search, the results are largely based on what someone searched for historically and what they clicked (or even subsequent searches and clicks).

e.g. "movie about kidnapped daughter" (I'm looking for the 2008 movie Taken, but have "forgotten" the name).

Google: 3rd result. Bing: 8th result. DDG: 7th result.

Why is Google 3rd and DDG/Bing 7th and 8th respectively? They likely have all indexed the Wikipedia and IMDB pages for the movie Taken. However Google likely ranks those links closer to the top because historically that's what users want when they searched for that phase.

Serious question though, why can't DDG do the same thing without compromising privacy? It would seem like it'd be possible to keep a log of things like "A user searched for 'movie about kidnapped daughter' and clicked on 'Taken'. Maybe weight that one higher next time" -- without keeping user details at all.

As an example: One really useful data source that I'm sure Google uses heavily is query reformulations. If a user does a search for Q, and then later does different queries Q' and Q'', and finally clicks on a search result, that's evidence that the result was actually relevant to the original query Q -- the search engine just wasn't smart enough to return it the first time around.

By itself, one data point like that is extremely weak evidence; the later queries might actually have been completely unrelated. But that's a source of error that tends to disappear when averaged across a large number of different users. In the aggregate, the data can be extremely valuable. But doing that kind of analysis requires correlating multiple searches from the same user, and storing the resulting profile for a long enough period of time to do useful aggregation.

Can't DDG use session-only cookies to link Q, Q' and Q"? Or would that be considered a breach of their privacy rules?

You can't count on a browser session to be particularly short-lived. Even if you don't tie query logs directly to a username, the mere act of correlating different queries from a user inherently compromises privacy, as demonstrated by the AOL leak.

I think this kind of issue involves DDG's AI. Google's is far superior in this respect. Heck, G has gotten me to things that I faintly remember but I know, without a shadow of doubt that I'd still be uncertain of 'that thing' I was trying to remember.

This is a search problem. I honestly don't see why DDG can't develop an AI that understands cultural and language contexts. Currently it doesn't but innovating this will take a lot of brain power, something DDG is lacking (when comparing the sheer #s of personnel focused on search).

This sometimes works, but I've found quite a lot of cases where it doesn't. The two most frequent culprits for me:

1) Smaller index. Some stuff just seems not to be in DDG at all. Even after finding a page I'm looking for via !g, I'll go back and try to see whether I could've gotten DDG to return it with some more specific search term (even verbatim phrases from the page) with no luck.

2) Languages other than English. The Google/DDG search quality gap is much, much larger in Greek than in English, for example.

What fringe items are you looking for? Ah, non-english, that sounds like a solid point. I only search english.

I tried a few days of DDG fully aware that DDG can't read my mind. So I did my best giving it context but I gave up in anoyance over how much time I wasted trying to give it the correct context, while a !g just hit spot on.

I should probably have tried more but I guess I'm too pragmatic for my own good.

On the contrary it usually gives me what I'm looking for even if the results are not exactly the same as google's. Sometimes I find that the customized search results and the insistence on prioritizing things I've searched for in the past can interfere with my ability to find what I'm looking for.

Reply on news.ycombinator.com