DuckDuckGo faces controversy over tracking agreement with Microsoft
metro.co.uk
metro.co.uk
When most other browsers on the market talk about tracking protection they are referring to 3rd-party cookie protection e.g., blocking them) and fingerprinting protection (i.e., restricting APIS scripts can use), and our browsers impose these restrictions on all third-party tracking scripts, including those from Microsoft. We also have a lot of other web protections that also apply to Microsoft scripts (and everyone else) that most browsers don't do, including Global Privacy Control, 1st-party cookie expiration, referrer header trimming, new cookie consent handling (in our Mac beta), fire button (one-click) data clearing, and more.
What this article is talking about specifically is one web protection that the major browsers don't even attempt to do — stopping third-party tracking scripts from even loading on third-party websites. See for yourself the bottom 'tracker content blocking' section of this audit site: https://privacytests.org/ios.html (scroll to the very bottom).
This web protection is a particular challenge to get right because websites can easily break when scripts that they depend on don't load, but because it makes for better privacy and faster page loads, we've taken it on while still trying to not break sites. As a result, it is far from perfect, as we can't block all scripts do to persistent breakage issues (which we try to workaround), and domains change all the time (and we continuously crawl sites looking for new tracking domains).
Though because we're doing this protection where we can, and also offer many other unique protections (e.g., Google AMP protection, smarter HTTPS upgrading, tracking protection for other apps in Android, email protection to block trackers for emails sent to your regular inbox, etc.), users get way more privacy protection with our app than they would using other browsers. Our goal has always been and remains to provide the most privacy we can in one download. And we have a lot more planned.
The issue at hand is all of our web protections apply to Microsoft scripts on 3rd-party sites (again, this is off of DuckDuckGo.com, i.e., not related to search) except this one around full content blocking, as we are currently contractually restricted by Microsoft there. However, this limited contractual restriction (about this one web protection) is the only one we have, and we have been and are working with them as we speak to reduce or remove it.
I understand this is all rather confusing because it is a search syndication contract that is preventing us from doing a non-search thing. That's because our product is a bundle of multiple privacy protections, and this is a distribution requirement imposed on us as part of the search syndication agreement that helps us privately use Bing results to provide you with better private search results overall. While a lot of what you see on our results page privately incorporates content from other sources (including our own indexes), e.g., Wikipedia, local listings, sports, stocks, lyrics, weather, quick answers, etc., etc., we source most of our traditional links and images privately from Bing (though because of other search technology our link and image results still may look different). Really only two companies (Google and Microsoft) have a high-quality global web link index (because I believe it costs upwards of a billion dollars a year to do), and so literally every other global search engine that wants to offer a search product competitive with Google for mainstream users, needs to bootstrap with one or both of them. The same is true for maps btw -- only the biggest companies can similarly afford to accurately map every neighborhood. And we are still a very small company relative to these companies -- literally on the order of 1,000 times smaller.
Anyway, I hope this provides some helpful context. I understand why people are upset with us on this, and we will do better. We're working on updates to our app store descriptions and other materials.
(Also FYI -- this was discussed extensively at https://news.ycombinator.com/item?id=31490515). This is also a somewhat misleading title since it's not about our search engine, which people would assume, and also as noted above Microsoft scripts are actually restricted in our browser a lot, so I would suggest changing it.)
“[W]hen you search, you expect unbiased results, but that’s not what you get on Google,” @matthewde_silva quotes @yegg"
Seems you're backtracked on being unbiased with the recent announcement of DuckDuckGo censoring sources on the back of the Ukrainian War. Any comment on that - seems like the bigger DDG gets, the further you stray from your initial values
Unlike other search engines (like Google), we don’t alter search results based on someone’s previous search history. In fact, since we don’t track our users we don’t have access to search histories at all. Those other search engines show you results based on a data profile about you and your online activity (including your search history), and so can be slanted towards what they think you will click on the most based on this profiling. This effect is commonly known as the search filter bubble, but using DuckDuckGo can help you escape it.
This does not mean our search results are generally “unfiltered” because, for every search you make online, a search engine’s job is to filter millions of possible results down to a ranked order of just a handful. In other words, a search engine has to use algorithms programmed by people to determine what shows up first in the list of results, what shows up second, and so on. Otherwise, for every search you’d just get a completely random set of results, which of course wouldn’t be very useful.
However, we do this ranking in a strictly non-partisan manner, and not based on my politics (or anyone's for that matter). I left another comment here on how that works: https://news.ycombinator.com/item?id=31557837
"Like so many others I am sickened by Russia’s invasion of Ukraine and the gigantic humanitarian crisis it continues to create. #StandWithUkraine
At DuckDuckGo, we've been rolling out search updates that down-rank sites associated with Russian disinformation."
We get it - you are making these moves because of financial reasons or political pressure....but don't try and play to both sides with the corporate speak.
** We actually do not intentionally censor any news results, meaning media outlets are not being removed or their stories displayed so far down in the results they are effectively removed. That is, unless legally prohibited, you should find all media outlets in our results, and they should generally show on top if you search for them by name or domain name. If you are seeing otherwise, please let me know and we will investigate.
A search engine's primary job is to rank results, trying to put results that most quickly and accurately answer the query on top. We do this ranking in a strictly non-partisan manner. Ranking for news-related searches is particularly difficult because for most news stories there are often hundreds of media outlets covering the same story, many with similar relevancy in terms of keyword matching and popularity. As such, we look to another ranking factor to ensure just the top of the results aren't taken by obviously very low-quality news results so that users have more sources of relevant, high-quality news results to compare and choose between.
The non-partisan factor we've found to help accomplish this is a rare, but well-documented history of a site's complete lack of news reporting standards, such as routinely using spam or clickbait to artificially inflate traffic, consistently publishing stories without citing sources, censoring stories due to operating with very limited press freedom, or misleading readers about who owns, funds, and authors stories for the site. And since we do not censor sites, even state-sponsored media in countries with very limited press freedom, these sites will still show up in results, and even on top like when you search for them directly. **
With that context, RT is a media outlet with "very limited press freedom" where journalists must censor their articles or else face jail time, or worse. And RT still shows up in our results, and on top if you search for it directly, e.g., https://duckduckgo.com/?q=rt+ukraine&ia=web
The fact you started censoring "Russian media outlets" right after the start of the war shows the political nature of your decision, those "limited press freedoms" weren't a concern until Mar 22.
Do you publish the decision making process for how the "non-partisan" decisions on censorship are made - or is this another blackbox? I'm sure Dailymail, New York Post and Al Jazeera are all down-ranked too, considering they meet your "non-partisan factors" right?
Strange, but RT does not show up in DDG for me. Not by using your link, not even by searching "Russia Today" on it. The first results I get are Al Jazeera or Fox News. No Russia Today at all. How's that?
So, for me, DDG is hiding some results while other search engines that I can use in the EU, do not.
Brave Search, just to give one example.
To me, this is really simple. You entered a contract that allowed a company to circumvent tracking protections. the extent and which product this happened on are only semantic details. It fundamentally undermines your trustworthiness as a supposedly privacy-oriented company.
I do think the details matter though because based on the comments across the web, most people seeing this news cycle are coming to false conclusions, e.g., that this is about our search engine (it's not), that we give 3rd-party Microsoft scripts a free pass on 3rd-party websites (we don't), that we don't block 3rd-party cookies in our browsers (we do), or that any of the major browsers offer better web protection (they don't, and in fact they don't even attempt this particular type of protection).
Unfortunately this takes a few paragraphs to unpack fully. Probably someone else can explain it better and more succinctly, but I'm trying my best here to be succint.
I don't want to read too far into the specific wording, but you mentioned "bootstrap" and this feels like a good opportunity to ask the question. Is DDG planning or thinking about how it might move towards it's own index for general results? I feel like most of the criticism DDG receives these days is because it's backed by another, less privacy concerned, mega-corp. While it may not be something that's feasible today, and not in it's entirety, are there moves being made towards building an independent global web link index?
I understand the controversy (or at least I think I do), and I also respect what you're trying to explain.
But if DuckDuckGo has to have a browser (instead of simply recommending to its users to change their default search engine on their current browser), then that browser needs to live up to its promises, and also to very high standards.
On the current page for the DDG browser [1] it says at the top:
> Seamlessly take control of your personal information, no matter where the Internet takes you.
If this means anything, it means the DDG browser blocks all trackers, always, in all places, no?
Sure. But this is usually addressed by filter lists.
> see https://privacytests.org/
Very interesting link, thanks. Didn't know that localStorage and sessionStorage could be read across domains!
But the comparison should include (browser)+uBlock Origin. I wonder if there are many users who care about privacy and don't install uBlock Origin by default.
Edit: added Origin to avoid confusion.
You've created a completely unrealistic standard. Nothing meets the standard of blocking all trackers everywhere as that isn't possible today or in the foreseeable future.
Also, I don't think that's true. And it's part of the problem with the current DDG defense. They have an agreement with MS to not block 3rd party trackers, and they say "in any case it's very difficult to do and nobody else even tries to do it".
It's a misdirection.
The best tools like uBlock, Umatrix, Privacy Badger, etc. all miss plenty of trackers, but not for a lack of trying. A complete solution doesn't exist today, nor is there anything promising on the horizon that I'm aware of. I am very curious to know the source of your optimism (I don't have much at the moment).
> They have an agreement with MS to not block 3rd party trackers, and they say "in any case it's very difficult to do and nobody else even tries to do it". It's a misdirection.
They shouldn't have done that to be sure, they've said as much, and are paying the price. They also said that future agreements won't include that carve out. They screwed up and will hopefully learn a valuable lesson from all this.
I will say, you should have used more precise language and said something like, "they have an agreement to not block 3rd party trackers in their browser" (emphasis mine). Your statement suggests that they don't block those sorts of things across their entire line of products, which isn't true. You're adding to the confusion by using overly broad language that is not precise and not 100% truly accurate.
Finally, I'm not aware of anything that is widely available that is as good privacy-wise as DDG and is actually usable by the most people. Until something exists that's just as good privacy-wise and is somewhat useful, I see very little choice but to use DDG. We'll see how DDG behaves in the coming months and years. Long term, they really should figure out something besides ads as a business model, since the incentives don't align with privacy.
Yandex and Baidu also have pretty good index, and perhaps would be even cheaper than Bing. I'd say they only lack in terms of the data (search queries, search result links clicked etc). It's just US protectionist policy (on behalf of the BigTech) at play here to avoid these services. Both Yandex and Baidu outperform Google when it comes to Russian and Chinese search queries, clearly indicating they are at par (or even better) than Google in terms of search algorithms. The only thing they lack is better data, and that's why they struggle with foreign search queries. (Though I find Yandex has improved a lot here, enough to make it my regular search engine).
I don't know many who use their browser, personally I moved over to Brave recently and so far so good.
Sad to hear that DDG lost it's integrity with this move.
Brave’s business model is based on ads which is very controversial, and that model always ends up compensating with the cost of user privacy.
I say this because NPR was running the same angle yesterday, and several "more reputable" sources are reporting the same thing as well.
Frankly, for DDG to spend all this money blasting "we actually care about privacy" advertisements in radio and online, and then to play around like this, it's certainly dangerous. That advertising campaign is probably why news outlets are even discussing this. "You defined your brand standard, why shouldn't we hold you to it".
I'm sure their marketing folks are upset! All that money on a message, undone in a flash.
If RMS took a job with Adobe, do you think people would question his commitment to open source?
In short though, you're right it isn't about our search engine. It also isn't about 3rd-party cookies (another point of confusion I've seen), which our browsers block (including for 3rd-party Microsoft scripts). 3rd-party Microsoft scripts are also further restricted in our browsers with additional web protections, e.g., fingerprinting protection, Global Privacy Control, referrer header trimming, etc.
What this article is talking about specifically is one web protection that the major browsers don't even attempt to do — stopping third-party tracking scripts from even loading on third-party websites. You can see that for yourself at the bottom 'tracker content blocking' section of this audit site: https://privacytests.org/ios.html (scroll to the very bottom).
So far they are being transparent. If you were a DDG user, you should keep supporting them for now and see where it takes us. There aren’t many tech companies that are privacy focused. And so far I think it’s in our best interest that privacy first becomes mainstream.
What this article is talking about specifically is one web protection that the major browsers don't even attempt to do — stopping third-party tracking scripts from even loading on third-party websites. You can see that for yourself at the bottom 'tracker content blocking' section of this audit site: https://privacytests.org/ios.html (scroll to the very bottom).
(For anyone wanting full context and a more detailed explanation, please see linked comment mentioned above -- https://news.ycombinator.com/item?id=31557587.
I don't believe you actually understand why people are upset with this -- it shows in the replies. You also don't seem to understand your users. It appears that you're entirely convinced that others are just misunderstanding you and that's why they don't see the genius/cleverness/importance/validity --whatever, in the actions DDG has carried out over the last months.
In all honesty, it feels like you're moving the company towards acquisition and looking for an exit.
[1] https://twitter.com/yegg/status/1501716484761997318 — I think it's sad that I have to clarify that I am personally also sickened by Russia's invasion of Ukraine, but that I don't want DDG to be an arbiter of truth despite that
I also did some smell tests by searching controversial topics, and Kagi does seem to present a more mixed bag of sources than DDG and Google on the first page of results.
Theres no winning as half the people want to be lied to and it has so obviously been weaponized.
If you think disinformation comes from one side you have been brainwashed good
https://misinforeview.hks.harvard.edu/article/right-and-left...
1: Source, Gabriel Weinberg's book Traction.
The people that use DDG overwhelmingly are not using it as a search engine to find Russian disinformation. This is literally a case of a business announcing that it's working to do its stated business activity, which is to provide relevant search results.
You may note what Microsoft had stated earlier [0] "We are removing RT news apps from our Windows app store and further de-ranking these sites’ search results on Bing so that it will only return RT and Sputnik links when a user clearly intends to navigate to those pages."
[0] https://blogs.microsoft.com/on-the-issues/2022/02/28/ukraine...
It's like complaining about water making you wet.
and rating stuff as disinformation is censorship. i want my search engine to be like the internet, i.e. a conduit for bits, not a self righteous mind that controls what goes thru.
There isn't really a neutral option (no "conduit for bits") – you either let low quality, high resource sites flood your search results, or you try to filter for quality. Here, that choice is also aligned with a political choice.
country with low tier GDP has supposedly more resources to move SEO than the first or second tech powerhouse in the world? Hum. I smell bullshit here.
If I take your argument as valid I should worry about Spain next.
Downranking sites that disrespect user privacy using trackers and similar is different, because it is not ideologically motivated. I guess you could call it censorship, but no-one except the site owners are going to take issue with it.
A simple way to tell if something is political censorship is to ask: who will take issue with the absence of this piece of information that is being removed or hidden? If the answer contains groups of people that aren't just corporations, I think it's pretty clear what is going on.
And I guess I have to repeat myself here: no, I don't personally belong to such a group in this context. I just happen to not want anyone telling me what I'm supposed to believe, even if I'm "on their side".
Just because it may be "political" doesn't give it special protection.
It is completely reasonable to question their algorithms and processes. I don't see why Russian disinformation needs to be singled out given what is allowed in the corporate press every time the US starts a fight somewhere (can they please downrank all sociopath war mongers?). But while it may be political activism it isn't censorship. If you use DDG you ask for them to make judgement calls.
Every time I tried to switch to DDG I ended up reverting to using the !g bang on every other query. While Kagi supports the same functionality, I haven’t felt the need to use it once in the past two months that I’ve been trialing the private beta. It’s genuinely better than Google for me, which I never thought would happen.
I understand that their business model is "you pay, you are our user, we won't sell your data"[1] But there is a simple privacy-friendly business model to be made with free anonymous search engine as well, when you look for "horses" we show you ads of horses. This is what Duckduckgo Search and Google Search both do. (Well google then runs ads on many website which are then linked to your search query, that's the real issue)
Now with Kagi, I have a company which knows all my darkest secrets whether I use a different computer behind a VPN or not.
under the hood they track the shit out of you, even more than DDG with their machine learning tools for user based scoring system for links
remember everyone, duckduckgo refused every independent audits, proprietary stuff like kagi is no exception, history repeats itself
also i find it weird that product placement comments like this one are always ranked super high on this website
they are done with DDG, they shifted focus to kagi instead? suspicious, it's not even out yet, only work on macOS, and is a team of "benevolent living in San Fransisco", living on donations in the most expensive state of the US, sure :thinking_face:
> kagi is just like duckduckgo
There is almost nothing similar between Kagi and DuckDuckGo, starting with the fact that DuckDuckGo is a global brand that has ~100M users and sells ads to make money, where we have few thousand and are 100% user supported with a subscription business model.
> under the hood they track the shit out of you, even more than DDG with their machine learning tools for user based scoring system for links
We literally don't. Kagi is not only zero telemetry but we do not log or associate searches with an account. The reason is we both don't want and don't need to. Our business is selling subscriptions not user data.
> also i find it weird that product placement comments like this one are always ranked super high on this website
Perhaps people simply genuinly love the product?
> only work on macOS
I think you refer to our Orion browser, which does work on Apple devices only for now. One needs to start somewhere.
Kagi search works in any browser on any OS.
> and is a team of "benevolent living in San Fransisco", living on donations in the most expensive state of the US
I literally have no idea what you tried to imply here.
You can find my email in bio and reach out if you need any additional clarification, I am happy to provide it.
We actually do not intentionally censor any news results, meaning media outlets are not being removed or their stories displayed so far down in the results they are effectively removed. That is, unless legally prohibited, you should find all media outlets in our results, and they should generally show on top if you search for them by name or domain name. If you are seeing otherwise, please let me know and we will investigate.
A search engine's primary job is to rank results, trying to put results that most quickly and accurately answer the query on top. We do this ranking in a strictly non-partisan manner. Ranking for news-related searches is particularly difficult because for most news stories there are often hundreds of media outlets covering the same story, many with similar relevancy in terms of keyword matching and popularity. As such, we look to another ranking factor to ensure just the top of the results aren't taken by obviously very low-quality news results so that users have more sources of relevant, high-quality news results to compare and choose between.
The non-partisan factor we've found to help accomplish this is a rare, but well-documented history of a site's complete lack of news reporting standards, such as routinely using spam or clickbait to artificially inflate traffic, consistently publishing stories without citing sources, censoring stories due to operating with very limited press freedom, or misleading readers about who owns, funds, and authors stories for the site. And since we do not censor sites, even state-sponsored media in countries with very limited press freedom, these sites will still show up in results, and even on top like when you search for them directly.
[1]: https://twitter.com/thezedwards/status/1528808759027331072What this article is talking about specifically is one web protection that the major browsers don't even attempt to do — stopping third-party tracking scripts from even loading on third-party websites. You can see that for yourself at the bottom 'tracker content blocking' section of this audit site: https://privacytests.org/ios.html
Do you publish these scorecards publicly?
If there is no transparency, we cannot assume your assessment is non-partisan.
A wise man from Texas once said: Fool me once, shame on you, fool me twice... can’t be fooled again.
While I don't believe a company should decide if an opinion about gun controle should be rankt or not. but if someone is claiming that these are just crisis actors and nothing happend I don't see a problem to block it. After all we are used to blocking spam these things are spam. I don't see a reason to treat this as any different just because they are selling political lies instate of product lies (and a lot of times these people are selling products in the end)
these kind of lies are eroding democracy and should be treaded the same way as some one yelling fire in a cinema
It took DuckDuckGo years to built up their positive privacy reputation and it can all fall apart in a few days.
They actively censor content based on political views and they now selectively enable Microsoft tracking. All in the span of like a month. And in return for all of this you still get a half-baked browsing experience.
Someone remind me what value DDG actually provides at this point? It feels like we're witnessing a self-implosion.
This is news to me - could you elaborate? Does it go beyond just passively duplicating Bing's censorship due to using their search results?
People are angry that a search engine (something dedicated to giving you relevant and true information based on what you searched for) is less likely to give you Russian state propaganda because the right wing have associated their political beliefs with that of pure free speech without realizing that such a thing just doesn't make sense in most modern settings.
But since you're trying to muddy the waters, I'm not "right wing" nor do I support the Russian invasion into Ukraine. I would have a similar reaction if CNN, Fox News or BBC were censored as well. I also wholly reject that a search engine's purpose is for "giving you [...] true information" -- I don't need that nor have I asked for it.
Now as Google search is the entry page to Amazon, Bing sees some increase in serious traffic and MS is getting more serious about their business strategy.
We don't need a Web3, we need a decentralized spider.
And when we deal with "trust" perception is just as relevant as reality.
This is a fork in the road for DDG, they can come out of this stronger than ever, or be tarnished and doomed.
Even bing is better than duckduckgo's results
>traditional links in the search results, which we also source from multiple partners, though most commonly from Bing (and none from Google).
https://help.duckduckgo.com/duckduckgo-help-pages/results/so...
They are not the same, duckduckgo use to be decent. Now I can't find anything programming related.
I even removes it as my default on my work computer its that bad.
Sounds like you learned that the profile Google has on you has benefits. The more of the general population that uses DDG, the more the results will skew towards the general population who are not programmers.