DuckDuckGo announces partnership with Yahoo | Hacker News Reader

DuckDuckGo announces partnership with Yahoo

duck.co

412 pointsalpb115 comments

I think a better link to explain is here: https://duck.co/help/results/yahoo-technical-implementation

It looks like they are just querying yahoo for results and ads as a proxy and it sounds like, through some contractual/technical limitations, they have to make that request from a yahoo controlled domain name (duckduckgo-owned-server.yahoo.net). However, they say they fully control the server and yahoo does not get to touch it, as yahoo delegated DNS to them.

The change does not seem malicious and if what they say is true, then the implementation might not be too bad. The volume of traffic they get probably means you cannot profile an individual from Yahoo's side. However, this potentially means that your search history is leaked to Yahoo, albeit in an aggregate manner with other DDG users, which may have attacks that I'm not aware of.

What I can see, however, is a lot of noscript users get startled by this change as they will see DDG use a script from yahoo.net (actually duckduckgo-owned-server.yahoo.net) as by default noscript does not show the full domain.

So they data mine this traffic at the nameserver level, just to reasonably say that DDG doesn't share info? Making it technically your computer sharing the info, by way of their forced implementation? Seems like the ad blockers are making the right move here.

This comment is puzzling, since authoritative nameservers only see actual end-user traffic on an occasional basis given the cornucopia of resolver caches in near-unanimous use (as well as the fat TTL on the yahoo.net side of the delegation). This denies Yahoo! the reliable data that you seem to think they've successfully pulled over on everyone, at least if I understand your comment correctly which I'll admit I'm having a hard time doing.

It's also an odd concern given that most everybody these days uses Google or their ISP's resolvers which would be far more interesting "nameserver level data mining." (I have a hard time believing Google successfully logs that traffic in detail, however.)

Just being devils advocate here, but wouldn't the cache rate be more or less uniform and able to be interpolated out into real traffic numbers?

If yahoo observes that every 1 in 3 yahoo.com requests in NS lookups then they could still figure out the traffic?

In practice.... i doubt this is valuable data at all.

I can't grok what you're asking me but I'm almost certain the answer is no, as you've already inferred. Extrapolating DNS analytics to actionable intel is incredibly misleading in nearly every case on the service side, particularly since there are a number of caches that consider TTL a guideline and since large shared caches are commonplace.

Understanding what your company is doing if you run the resolver is where it makes more sense. But on the service side? You might as well hire a cat to report metrics by pawing a ouija board, and then you at least get cuddles with your random numbers.

Think of it this way: DNS is basically a different form of ARP, in a certain manner of speaking. I wouldn't try to quantify performance data from either except to troubleshoot operations.

So they data mine this traffic at the nameserver level, just to reasonably say that DDG doesn't share info?

What does this mean? If you mean nameservers might get to see your IP, that isn't true. At most they'll see your primary DNS resolver's IP, but if you use Google DNS or Open DNS they will only see that (I've implemented CDNs, and this is a very big problem for them if they operate at the DNS level - which most do).

> if you use Google DNS

> Afraid to give Google my search queries

>> I'll give them all of my DNS lookups instead

So... don't? Just use your default ISP resolvers?

Depending on what threat you are trying to avoid there are options here. The OPs threat assessment was wrong, though.

The cynic in me says that it all starts like this. With multiple, gradual ToS changes, eventually, it will become like any other company out there. And what if Yahoo acquires DDG?

Then one of us can try to break into the search market!

privacy first search market

That would be very foolish of duckduckgo, their whole niche is privacy.

Isn't that basically the MO of every startup?

* Offer something for free, with VC money. * Gain market share since you don't need to worry about profit. * Realise you need to make money, so start turning into the companies you stole market share from.

Just want to point out that although that may be a model for many startups, DuckDuckGo is already profitable through (non-tracking) advertising and affiliate revenue. More info: https://duck.co/help/company/advertising-and-affiliates

Disclaimer: DDG staff

That's great news, and probably needs to be more widely known.

youdontknowtho is shadowbanned.

No he isn't?

Might have used the wrong term, but the above comment was dead earlier, and several of his other comments are dead despite not seeming to violate any rules.

I don't want to be rude, but I don't think DuckDuckGo has actually taken much market share to worry about yet, so maybe it's not following the usual startup route.

Does it matter if it's following the "usual startup route"? If it's profitable, and it hasn't compromised on it's values, then it's winning.

Maybe. It's not difficult to be profitable though. Servers are cheap. I don't think it's going to trouble Google though, which is a shame. We could do with some serious search competition.

I think it's very, very reasonable to be skeptical of any claims about DDGs actual concern about privacy. Actions speak far louder than words.

Gabe made his first few million by making a classmates.com clone that was the opposite of respectful about privacy. Those actions made clear that he has no ethical or moral stance regarding privacy, and that he will happily violate privacy for profit.

Add to that the fact that he took venture funding. Venture funding aims for a maximized exit price, not a moral stance. The moral stance will disappear the moment your site gains meaningful traction. It's what Ayn Rand (one of your founder's favorte authors) would have wanted.

DDG users who care about privacy are, to be blunt, laughably gullible. Your current business is pro-privacy because:

1) the search context creates sufficient value even without the personal target; and

2) the tech is a dumb layer on top of other tech, and it can't provide personalizaton.

As such, yegg's pretending he cares about privacy... but he's just selling the limitations of his lousy tech as benefits. It's a PT Barnum move. As soon as the tech is better, the privacy will disappear.

That link is pretty good at explaining the DNS delegation, TLS and NoScript. However, as DDG runs on AWS, they don't have sole control of their servers anyway.

Edit: output of a DNS/Ping tool I wrote for a book:

    Enter a hostname or IP address:
    duckduckgo.com
    
    Performing DNS lookup of duckduckgo.com
    Complete, duckduckgo.com = 54.229.105.92
    Performing reverse DNS lookup of 54.229.105.92
    Complete, 54.229.105.92 = ec2-54-229-105-92.eu-west-1.compute.amazonaws.com
    
    Complete, duckduckgo.com = 54.229.105.203
    Performing reverse DNS lookup of 54.229.105.203
    Complete, 54.229.105.203 = ec2-54-229-105-203.eu-west-1.compute.amazonaws.com
    
    Complete, duckduckgo.com = 46.51.197.89
    Performing reverse DNS lookup of 46.51.197.89
    Complete, 46.51.197.89 = ec2-46-51-197-89.eu-west-1.compute.amazonaws.com
    
    Complete, duckduckgo.com = 176.34.135.167
    Performing reverse DNS lookup of 176.34.135.167
    Complete, 176.34.135.167 = ec2-176-34-135-167.eu-west-1.compute.amazonaws.com
    
    Complete, duckduckgo.com = 176.34.155.20
    Performing reverse DNS lookup of 176.34.155.20
    Complete, 176.34.155.20 = ec2-176-34-155-20.eu-west-1.compute.amazonaws.com
    
    Complete, duckduckgo.com = 176.34.131.233
    Performing reverse DNS lookup of 176.34.131.233
    Complete, 176.34.131.233 = ec2-176-34-131-233.eu-west-1.compute.amazonaws.com
    
    Pinging duckduckgo.com 4 times
    Ping attempt #1 of 4
    Success
    30 ms
    Ping attempt #2 of 4
    Success
    30 ms
    Ping attempt #3 of 4
    Success
    30 ms
    Ping attempt #4 of 4
    Success
    29 ms
    
    Press any key to exit...

Source code (C#): https://github.com/PacktPublishing/ASP.NET-Core-1.0-High-Per...

TIL! I didn't realize they ran an AWS stack.

I've been using DDG for years (use !g about 1/3 of the time I think), and I just like having the alternative.

I wish we had more search engines, but I also realize the barrier to entry is really high. Using AWS does make a lot of sense in their market.

I thought their primary index was Yandex though, and they purchased supplemental data from Yahoo and Google?

Supertip: Use !sp instead of !g to get googles results but without the tracking.

Looks like https://startpage.com uses http://www.routit.nl

Interesting. Source? Maybe they're hosting? Both are based in the Netherlands. http://www.routit.nl/oplossingen/cloud/

Same source as before. Yes, I assume that's their hosting company.

    Enter a hostname or IP address:
    startpage.com
    
    Performing DNS lookup of startpage.com
    Complete, startpage.com = 37.0.87.7
    Performing reverse DNS lookup of 37.0.87.7
    Complete, 37.0.87.7 = rt87bb0-37-7.routit.net
    
    Complete, startpage.com = 89.146.4.146
    Performing reverse DNS lookup of 89.146.4.146
    Complete, 89.146.4.146 = rt4bb146-89-146.routit.net
    
    Complete, startpage.com = 145.131.132.81
    Performing reverse DNS lookup of 145.131.132.81
    Complete, 145.131.132.81 = rt132bb131-145-81.routit.net
    
    Pinging startpage.com 4 times
    Ping attempt #1 of 4
    Success
    26 ms
    Ping attempt #2 of 4
    Success
    27 ms
    Ping attempt #3 of 4
    Success
    26 ms
    Ping attempt #4 of 4
    Success
    28 ms
    
    Press any key to exit...

https://duckduckgo.com/?q=whois+routit.net&ia=whois

    Registered to  J.H. de Baat
    Email          info@routit.nl

Protip for the supertip: I just use !s for startpage. One keystroke lesser. :) It's the one I use most of the time.

Although my default search engine is DDG, for my use (technical and otherwise, with date based searches) DDG is still more like a toy that doesn't provide me the relevant results more than half the time. So I stick to startpage most of the time and sometimes use Google directly (with tracking related precautions/protections taken).

This looks to me like Route 53 going straight to EC2, with no ELBs. Amazon own a lot of the 54.x.x.x IPv4 addresses, which is often a hint.

I like (and trust) DDG, and use it all the time. But it is interesting to know who else has access to their infrastructure.

I was surprised by the AWS result, as I expected them to co-locate their own dedicated hardware.

I've been astounded at how rare it is for a search API to offer date information. I talked to Gabriel about the problem many moons ago, and even with their significantly greater clout (than a random dude working on a tiny project that uses search engine APIs), and despite asking for it, they couldn't get that info from their partners. So, it's good to see they finally made it happen with Yahoo.

But, I thought the Yahoo BOSS search API was gone, and the only way to use Yahoo search was basically just a black box sort of deal (i.e., you pass them the query and they hand you results, which you display as yahoo sent them). I'm not sure I understand what DDG is doing, and when you'd hit those Yahoo results. It's especially convoluted since Yahoo gets its results from Bing, right?

Always loved DuckDuckgo - kinda came across as a search engine that didn't seek to takeover the world, and only sought to fill a (necessary) niche. That seems more genuine than "don't be evil".

Google originally won everyone over with superior results, the rest of it came pretty organically.

The underdog is always everyone's favorite until they somehow wind up on top.

There's some truth to the pattern, though; Once you're no longer the underdog, those seeking power will seek you out, and it's hard to prevent infiltration by the power-hungry.

Good things dont scale. Sergey Brin & Larry Page could have been the old Japanese man in the subway station sushi joint making the the best bites of a lifetime for a few hundred humans a day. They decided to go a different route.

I don't think I follow. A good web search engine cannot be a niche product with a few users. The bigger the resources, the better the results. That's an impossible problem to overcome by merely being really good at something. DDG overcomes it, reasonably effectively, by getting results from a huge list of partners.

If there weren't huge companies offering search APIs, DDG simply couldn't compete with Google (without massive scale of their own). As it is, DDG still provides somewhat weaker results in some cases; it is still my primary search engine on all of my devices, and I like it, but when I'm frustrated by the results, I make use of !g and usually find what I'm looking for.

I think the implication is for Google to just do search. Nothing else. Not to be niche product with a few users.

Then Bing could have won.

Or you can use !s for StartPage and have results from Google.

Seems the key is to continually use the underdog.

> The underdog is always everyone's favorite until they somehow wind up on top.

Google was everyone's favorite for a while even after becoming the market leader. And it seems the reasons to dislike google are pretty diverse (although around here I mostly hear "because they forced g+ login on youtube" or "because they killed reader". FTR my biggest annoyances with them was 1: when they killed off Google Desktop Search, 2: the years when they insisted on fuzzing any search until it returned millions of results )

>> although around here I mostly hear "because they forced g+ login on youtube" or "because they killed reader"

You forgot {{insert all possible privacy concerns}}, and on that wave, total disrespect for user choices. As a very tech savvy human I still can't dodge all their malicious attempts to collect my data, imagine the non savvy human fellow.

Ah, Vic Gundotra.

The basic problem with Google is the age old software engineer problem, not knowing when enough is enough.

They kept tweaking results and trying to be helpful, and over time building a "bubble" around the individual user rather than trying to produce the objectively best results based on the query.

As I recall, when I started using Google in the late 90's, it was fast, amazingly fast, and that was in the days of a clunky internet.

Please help me understand why DuckDuckGo decided for the query to go in the URL, as opposed to being encrypted on the client before being sent to the server as a get or a post.

Most users probably have their browsing history switched on, and synced with their friendly browsing software provider anyway, totally reversing any privacy advantage a user might potentialy gain by using DuckDuckGo.

They do have an option to turn on POST instead of GET.

This is from their page on privacy ( https://duckduckgo.com/privacy ):

> Another way to prevent search leakage is by using something called a POST request, which has the effect of not showing your search in your browser, and, as a consequence, does not send it to other sites. You can turn on POST requests on our settings page, but it has its own issues. POST requests usually break browser back buttons, and they make it impossible for you to easily share your search by copying and pasting it out of your Web browser's address bar.

I want to use DDG. I've made it my default search engine. But the result page is just so slow. Both in slow to load (v Google), and actual responsiveness. It's just slightly laggy, or at least it feels like that. Google has neither problem. I'm willing to put up with not-excellent relevancy, but the slowness is killing me.

I'm using FF 49 64-bit on Win8. Am I anomalous or does anyone else feel this at all?

I generally have no problems with it. I've tried FF/Chrome on Win8/10 and Arch. The load time might be marginally slower than google but the UI is fine.

No slowness here.

The only downside is the quality of results.

Most of the time when searaching technical questions I get better results on google compared to ddg.

It looks like ddg doesn't like stackoverflow.

Agreed: underranking SO is my biggest issue with DDG.

Which of course depending on their ranking implementation seems kinda odd, as you could expect a big part of the users of SO also to be the ones using DDG. But again if it categorically undervalues results from there, then those users will of course find alternatives.

This is what I've noticed as well. Searching for Rust programming questions is very hard with DDG, whereas Google gets relevant results at the top. Otherwise, DDG and Google are pretty much on par.

Might be the query then, I usually get the proper SO result as one of the first 3 matches. Anyway if you're getting results that are really off, hit the 'Send feedback' button. A DDG employee here on HN claimed they use this actively to make results better - hard to teel if that is so, of course.

I haven't noticed but I wonder if the complexity of making the infinite results pop into the page as you scroll is causing the difference. For me this feature alone is worth a mild slowdown in responsiveness. On Google you gave to click to get to results >10.

You're not anomalous. I'm not usually using DDG, but checked it out on chrome 51 64-bit on Win10, compared with google and there is definitely noticeable lag on DDG in many areas, though NOT to the extent of making page unusable.

I've noticed that DDG is maybe twice as slow as Google (still damn fast), and much faster than the links returned in the results. That said, I have recently noticed that occasionally, no results load at all, and I F5-it.

I do sure hope that any agreement DDG has made with Yahoo doesn't get cancelled by their eventual new owner.

But if Dan Gilbert ends up being the buyer my guess is that he'd actually like Gabriel Weinberg enough to continue. Not so much with Verizon or the private equity firms bidding.

DDG is a meta search engine that uses mainly Yahoo and/or Yandex (depending on the region).

Yahoo had turned their Yahoo BOSSS API (acccess to Bing search engine) from free to pay-for-each-1000-queries and completely discontinued the API earlier this year. So now we learn DDG gets access, but no one else of the smaller fishes (other meta search engines, etc).

For the layman like me: if I do a search for "cute cats", does the results comes from Bing after it has been through Yahoo before DDG shows it? Who does the actual searching?

Yeah been seeing the Yahoo logo on the bottom right of my DDG results, for a couple weeks now.

Not going to be moving away because of this, we'll see if it gets even better.

I'm still getting Yandex, guess Gabe's algo suspects I'm not much of an ad clicker :-)

Which is funny since I have changed my hosts file and use uBlock Origin and extensions like that for anything that might have missed.

Same here. How new is this 'news'?

I don't have any technical concerns about this, because I trust DDG based on their actions to date.

But TBH it does tarnish their brand a bit, in my eyes, and I don't fully understand why this is a big enough improvement to dilute the brand at all.

Is this actual Yahoo tech or Bing result enrichment by way of Yahoo?

I heard Yahoo has a search team but I don't know exactly if they were just enhancing on top of Bing really.

For those who'd been missing date-restricted searches, DDG now offers these, thanks to the Yahoo partnership.

It's not as flexible as Google's date-ranged searches, in which specific start and end dates can be specified, but you now can restrict searches to the past day, week, month, or year. I'd noticed this (and submitted to HN) about a month back.

https://news.ycombinator.com/item?id=11819435

(The control's moved to the Hamburger Menu since.)

(I'm hoping more controls come into effect.)

Great! Missing date filtering was a major downside.

I switched to using StartPage[1] a while back, because I got sick of waiting for DDG to add date-filtering. I've kind of got used to StartPage now, so not sure if I'll switch back... although some of DDG's 'bangs' were handy.

[1]https://startpage.com

DDG has now date filter support.

DuckDuckGo should wait a little longer and then buy Yahoo.

Why do they say "duckduckgo-owned"?

It appears they are using AWS. Who really owns the server?

   echo 50.18.192.251 duckduckgo.com >> /etc/hosts

Avoids needless DNS lookups; saves DNS logs from your footprints.

... and will break as soon as that instance is shut down.

That's why I'd prefer a search engine that was not using AWS, or Yahoo. Too many needless dependencies.

Out of curiousity I'm starting a counter today. Will post something when this IP addr fails. Feel free to take a guess how long it will be used by DDG.

And when it does change, if ever, I have a one line shell script that uses ed to remove entries from files, so all it takes to add a new IP addr to HOSTS is typing

    ed-script-name domain-name /etc/hosts
    echo aa.bb.cc.dd domain-name >> /etc/hosts

In my opinion, the usefulness of a website is inversely proportional to the frequency with which it changes IP addresses.

For example, the IP address for HN rarely changes.

But feel free to keep looking it up in DNS every day. Just in case.

If there are problems with DNS, most users will probably not remember the IP address for HN and will not be able to read the stories.

Meanwhile, the user that stores IP addresses as a backup will have no such problems.

"We've worked closely with Yahoo to implement these new features in accordance with contractual terms and our privacy policy. To be as transparent as possible, we've written up details on the technical implementation."

As a search user, I couldn't care less about this stuff. I don't think it's even possible to build a good search engine "in accordance with contractual terms".

Yet another Privacy Oriented Search Engine: Oscobo https://oscobo.co.uk

Not so bad results, although a bit slow.

Oscobo is a Yahoo partner. It is Motherpipe.com, so similar setup as DDG, but without the "owned" server in between.

i noticed a slow down sometime ago... did Duckduckgo slow down when/because yahoo got involved?

If you look at their search statistics, they've been growing slower this year.

https://apps.axibase.com/chartlab/e8635882/7/

Does this mean that Yahoo results appear in DDG?

Will DDG power Yahoo search in any way?

From the "technical implementation" article they link to

>When you do a search on DuckDuckGo, depending on the particular search, we will make a call to Yahoo for links, ads and other content.

What would they power Yahoo with? DDG has always just shown Yahoo results (which in turn comes from Bing).

>DDG has always just shown Yahoo results //

Pretty sure that's not true. DDG used to do its own indexing IIRC. Early on in the project we used to get regular updates here.

Motto of DDG was "Still better than Yahoo" ; )

That's great. Anything that sees more competition in the search space is good for the Internet.

In most cases I'd worry, but I think Gabriel is trustworthy and long-term focused.

Interesting. Probably won't do a ton though for their bottom line.

And so it ends. The mere act of partnering with Yahoo indicates desperation and signals poor decision making.

I'm not sure I understand HN's fascination with this site but this is not a good sign for those who claim to use it in these never-ending threads.

How? its all proxied and the 'yahoo server' is duckduckgo owned.

Privacy Statements are the tech equivalent of calling food organic.

You need a reputable third party to audit your practices. Or your software needs to be open source so anyone can see what you're doing.

I'll be switching from using DDG because of this. I don't trust Yahoo, and I barely trust DDG.

Ironically, your name is organic_tech34. For the record, the term "organic" is regulated by the USDA.

thatsthejoke.jpg

> the term "organic" is regulated by the USDA.

I think the point the previous poster was making is that even though it's regulated, there are so many black boxes that it's rather easy to sneak in a violation and no one notices (whether the company is doing it or it's accidental). Just look at how many organic product recalls happen in any given year due to various causes.

Ok, that makes sense.

You seem to have very high standards for privacy. So I'm curious: If DDG doesn't pass muster, what search engine will you be switching to?

What are you switching to? As far as I know ddg and yahoo are basically just bing already. So what is an alternative you are comfortable with?

Searx is pretty cool:

> A privacy-respecting, hackable metasearch engine

[0]https://searx.me/

[1]https://github.com/asciimoo/searx/wiki/Searx-instances

[2]https://github.com/asciimoo/searx

Sorry could not abstain

Two turkeys do not make an eagle

For the record, this isn't really news.

A lot of details can be found at https://duck.co/help/results/yahoo-technical-implementation

https://duck.co/help/results/sources also states that they partner with Yahoo!, and in some regions and scenarios, Yandex and Bing.

Different topic:

Why do they use www.duck.co as their development url and www.duckduckgo.com as their search engine name?

The latter is just way too long to type. Especially for casual users who has google set at their main search engine. Compare URL lengths:

duckduckgo.com 14 (10+4)

google.com 10 (6+4)

yahoo.com 9 (5+4)

yandex.ru 9 (6+3)

bing.com 8 (4+4)

baidu.cn 8 (5+3)

I really want to like duckduckgo, but the name is really not well chosen. Even google (with a relatively long name) at least have two repeating characters.

There's ddg.gg, but I don't know if that's an official domain and it's not necessarily easier to remember.

DuckDuckGo is to long and ddg.gg url is kinda makes it disservice. It instantly gets labeled as "nerd-stuff" if you recommend it to some non-techy friend, you just know they never going to even try it.

ddg.gg 6 (3+3)

https://duck.co/help/company/short-domain

Thanks, that was what I was looking for. It should be the default url.

If I had to guess, because they don't own duck.com and developers have a better grip on unusual TLDs than most users.

Addendum: Whoever owns duck.com forwards it to google.com, so duck.co would be a really bad place to be as a search engine

> Whoever owns duck.com forwards

https://whois.icann.org/en/lookup?name=duck.com

And Google owns it... how can this be legal?

I can't remember the last time I typed in a search engine URL. I type my queries into my browser URL field and let it do the right thing. When I want Google, I use "!g query" (because DDG is my default, and that's how you tell it to redirect the query to Google).

Under what circumstances do you type in the name of your preferred search engine?

When I want to use the non-default search engine (non-default is usually ddg). When trying a search engine for the first time. When using a public computer.

Biggest problem is on mobile. On Chrome (for iPhone), I have the following options:

1. Google

2. Yahoo!

3. Bing

I don't see any way to add a search engine not on the list.

This doesn't change the fact, but you really shouldn't expect anything else from a browser made by a search engine provider. They probably wouldn't even allow the other two options, if they could get away without monopolization lawsuit.

> I don't see any way to add a search engine not on the list

Then that's a problem with Chrome. Using Safari on iPhone you can set DDG as your default search engine.

Wow, what are our options now?! Also, Yahoo now has information about general searches made by DDG users even if they can't get info on a particular individual.

Yahoo, the company that saves your unsent and deleted email drafts forever and will turn them over to law enforcement even though they say that should be impossible?

http://motherboard.vice.com/read/how-deleted-yahoo-emails-le...

Maybe Yahoo is sincere in their privacy pledge, maybe they're not, but one thing has already been proven: they don't have a good enough grasp of how their own servers work to make that promise.

Very disappointed in DuckDuckGo. I expect they won't be around for too much longer, much like Yahoo.

Reply on news.ycombinator.com