DuckDuckGo browser seemingly sends domains a user visits to DDG servers
github.com
github.com
I’m new to this issue and happy to commit us to move to doing this locally in the browser and will have us move on that ASAP.
That said, I want to be clear that we did not and have not collected any personal information here. As other staff have referenced, our services are encrypted and throw away PII like IP addresses by design. However, I take the point that it is nevertheless safer to do it locally and so we will do that.
And if this gets fixed in a reasonable timeframe, this is just one of those "everyone makes a mistake one in a while"-things, no big deal.
Can you imagine Google doing something similar? Heck, they're just about to throw the Android rooting community under a hardware-attestation DRM-filled bus.
It was a forgivable-but-negligent decision to write/approve that code in the first place. It was a sign of a bad process that a reported security vulnerability was not escalated to people security-conscious enough to immediately identify this as a major problem.
I don't agree with the outrage. Anyone who has followed DDG knows they're legit. They just need to do a bit better. They probably will.
Their main feature is privacy. They should be at least as sensitive to privacy vulnerabilities as their most aware users.
DDG should announce that they now pay out privacy-related vulnerabilities like this and send the reporter $5k. It would be good honest PR and well worth the expense.
The former. This is a quick initial fix that will fail on many sites.
At what point did we stop taking people's word and commitment as valid? Sure, I too want to see proof that they are doing the right thing here (because I don't understand the design decisions that led to the creation of that service in the first place), but because these changes are not immediate, this statement does at least answer some of the questions I personally had (Like "will this be fixed"? "when"?).
At the point that they where caught violating privacy while claiming to respect it, and standing to make a profit off of violating it.
If I trusted people as blindly as people seem to trust DuckDuckGo, I would have trusted Google to not be evil and never have switched to DDG in the first place. This breach of trust destroys the whole point of using it, so I switched my default search to Searx between reading this submission and writing this (had been procrastinating the switch for a while and this was exactly the motivation I needed).
"...as has happened so many times before."
Clearly it's not just the response.
It's not immediately obvious whether it is more privacy preserving if the client automatically makes a request to each site in the search results while scrolling through the results, especially since you're already trusting DDG when performing the search.
Maybe this should be an opt-in rather than an opt-out feature?
All in all its really not as big of an issue as people here make it out to be.
Anyway, great decision by Gabriel.
Scenario #1 - "We need to show favicons in our browser tabs. Lets develop an API that requires every domain be sent to us!"
Scenario #2 - "We need to show favicons in our browser tabs. Hey look, we've already got a service that provides this. We know it collects no PII and our users trust it already."
Obviously the second scenario is flawed thinking, because (of course) it's better to not send that info at all. However, I can easily see how their developer(s) may have arrived at the conclusion that this is still compliant with their privacy ethos.
The fact that the favicon service already existed (and was trusted by users) before this was implemented, makes it much easier to understand how this could have been a legitimate mistake and thus, they deserve the benefit of the doubt.
Please refrain from speaking for others without being asked to.
By way of analogy, when you earlier said "This is ludicrous. Can we switch to the metric system already?" no one thinks that you're speaking for everyone, but rather are advancing your own belief in the "This is ludicrous." sentence.
‘we never used this data, other than showing favicons’
In fact, we didn't (and don't in general) collect any user-level data in the first place, per our strict privacy policy: https://duckduckgo.com/privacy
In this case, the way it works is you hit our favicon service and it returns the favicon, not using any PII in the process, and our web servers are configured not to log any PII. In other words, our system is technically designed and we are legally bound by our privacy policy to not use this data for anything other than showing favicons.
Contrary to this framing, it’s not possible to not incidentally become aware of every single browser users’ usage timing and user IP addresses if the browsers are phoning home this way — a colloquial understanding of ‘collect’, not the James Clapper NSA dodge definition of ‘collect’. Most normals think of collect as become known not as permanently store. You knowing it means others can know it if you break trust or are required to comply with authorities.
And regardless of end-to-end encryption, that this user is phoning home to your fave icon endpoint, when, and from what IP, is revealed to every ISP in the chain. You’re leaking browser usage telemetry to every single party to that traffic — the source IP address PII you mention is in unencrypted metadata.
The fact this browser connects to that endpoint reveals demographics (choice of privacy browser) and behaviors (when and how much web surfing) to e.g. ISP or nation state firewall operators who are certainly not bound by your ‘just trust us’ privacy policy.
Privacy policies are a patch for insufficient privacy engineering.
To be a strong privacy browser you could consider what it would take to be “NSL proof” such that if handed a national security letter with gag order, you cannot comply. That is not the case with this faveicon telemetry endpoint.
Getting the icon from each site means surveillance would have to be at origin or every site, while telemetry going to DDG gives a single surveillance point.
(I always knew there was a business model for privacy and I'm glad someone is working to figure it out)
So, you do collect information just not the kind you would classify as 'personal information'. I wonder if my personal domain with my full name qualifies?
There is no way this feature would be created in a company built on privacy considerations.
Like, wow!
Didn't that server endpoint need an upkeep? Didn't they wonder why it's getting all the traffic? Maybe they were DDOSed or something?
Multiple people in that company knew exactly what they were doing.
Considering they have operations in the EU, I would imagine that would fall under what the GDPR considers personal information (https://gdpr.eu/eu-gdpr-personal-data/), or risk being in breach of it.
However, this problem demonstrates gross incompotence for a browser team supposedly concerned with privacy. Will you please do a post-mortem on how this code made it through your code review process in the first place, as well as how it managed to stay in place for a full year after it was pointed out that it represented a privacy problem?
"Sends every URL you visit to the vendor's servers" is the single worst thing DuckDuckGo could have done for privacy in this web browser, and that needs to be accounted for. There was a major failure in the code review process, ticket review process, and in how you treat your community. A standard marketroid "by design" response with washy promises that "we'll take very good care of this highly sensitive personal data, just trust us" is not something I want to see in the future from this team.
[reposted from GitHub]
I agree that for a company built around privacy even the appearance of impropriety needs to be avoided. DDG holds themselves to a higher standard and their users hold them to a higher standard.
This was a design flaw and a process flaw. DDG prioritized speed and efficiency over privacy (or in this case, perceived privacy) and I suspect there isn’t a soul on HN who hasn’t made that trade off at some point. They assessed the cost/benefit and risk/reward and it turned out their assessment was wrong. Now they’re fixing it. It happens. But to call this gross incompetence is really blowing it completely out of proportion.
The first rule of privacy is never handle the private data in the first place. An accidental leak is one thing, but deliberately designing a feature whose side effect is exfiltrating heaps of private data, then doubling down on it for a year after it's pointed out to you, then doubling down again when it's raised on HN - this is gross incompetence.
My browser syncs URL history between my devices, and that’s a feature that I value about it. Your comments on this topic seem to suggest that all users are making the same decisions about what is acceptable usage of their data, and that’s pretty obviously not true.
Like our search results, the favicon service adheres to our strict privacy policy[1] in that the requests are anonymous and we do not collect or share any personal information.
The service is private as we do not collect any personal information (e.g. IP addresses) on any requests for this or any service and the requests are all end-to-end encrypted.
Potentially saving a few requests here and there is certainly not worth phoning home with that kind of data regardless of what records you keep how much you do to anonymize it. This is especially true for a company that has built its brand on promises of privacy!
Besides, favicon requests are small potatoes compared to the kind of tracking, ads, metrics, and other often-unnecessary page resources that bog down most of the modern web. And a well-designed website can mitigate the issue pretty easily.
It's a really bad look and you should ditch it.
This is troubling.
If you say the service is anonymous and does not leak data, prove it.
This sentence is 100% meaningless. I understand you have good intentions, but these things must rely on proof, never on trust. Either you get this information or you don't; whether you say you "collect" it is inconsequential.
If it's not present, then you have options, and yes, using your weird API is an option (which I still don't like, but ok). But sending private information to your servers even when sites follow the standard show either that you're probably not trustworthy, or that your product team is so painfully incompetent that I'd be afraid to use their browser at all.
Which presumably means you've already created the logic for determining favicons. I'm not sure why this couldn't be implemented in the browser.
I don't think here's a need for adjectives here. Why stress that it's anonymous (when that's hard to verify) or that the search engine is private, when that too is starting to come into question? Repeating these things won't will them into the reader's perception.
> In addition, doing it this way avoids another request (and potentially multiple) to the end site.
This isn't true, unless I'm missing something here? When I access a website, the HTML response I get from that website includes all the information my browser needs to, on its own, get and display the favicon. Can you clarify why you think/say this avoids one or more requests? What mechanism is this service a substitute for?
Sidenote: The more I use pi-hole the more I realise how essential it is!
Those simpler popover-ads which can be closed clicking an X in the upper right corner still are blocked tho...
Spilling my secret tho (and YouTube execs hate me for it!): i block YouTube in my mind and only rarely go to it if i really need to watch a video (which, for me, is rarer than i ever thought it'd be).
Or mpv (for single videos, or local playlists), or mps-youtube, or youtube-dl.
Uncheck the very last option "Site Icons"
The issue, as I understand it, is that the Android app loads the favicon service for search results you actually open in the app.
Sure they can. Doesn’t mean you have to believe them.
Seems like time to get SearX a try now: https://searx.me
I have read your explanations in good faith and they don't cut it. This behavior cannot continue. Good privacy promises are not based on trust - they're based on not ever handling private data in the first place. If you don't quickly admit your mistake and roll this back, it will jepoardize your entire brand - and rightfully so. If you believe this behavior is okay, then it demonstrates incompetence; if you don't believe this behavior is okay but do it anyway, it demonstrates malice.
This is the one thing you Should Not Have Done.
Do Chrome, Firefox, or Safari do this? I would assume they do it on-device.
CREATE TABLE meta(key LONGVARCHAR NOT NULL UNIQUE PRIMARY KEY, value LONGVARCHAR);
CREATE TABLE icon_mapping(id INTEGER PRIMARY KEY,page_url LONGVARCHAR NOT NULL,icon_id INTEGER);
CREATE TABLE favicons(id INTEGER PRIMARY KEY,url LONGVARCHAR NOT NULL,icon_type INTEGER DEFAULT 1);
CREATE TABLE favicon_bitmaps(id INTEGER PRIMARY KEY,icon_id INTEGER NOT NULL,last_updated INTEGER DEFAULT 0,image_data BLOB,width INTEGER DEFAULT 0,height INTEGER DEFAULT 0,last_requested INTEGER DEFAULT 0);
CREATE INDEX icon_mapping_page_url_idx ON icon_mapping(page_url);
CREATE INDEX icon_mapping_icon_id_idx ON icon_mapping(icon_id);
CREATE INDEX favicons_url ON favicons(url);
CREATE INDEX favicon_bitmaps_icon_id ON favicon_bitmaps(icon_id);
I haven't looked at Firefox and Safari but I assume they do something similar.
Also, in SQLite, note that LONGVARCHAR is the same as TEXT, and that you don't need to specify both UNIQUE and PRIMARY KEY (it is redundant), and that if it is not a INTEGER PRIMARY KEY and not WITHOUT ROWID, then it isn't the real primary key but just an index (same as UNIQUE); add WITHOUT ROWID if you want to make it a real primary key, but note that the way the data is stored differs then, and WITHOUT ROWID is inefficient with tables storing large blobs.
Not that I'm against making money. But there's a tipping point associated with some height value in a pile of cash, and once you cross that point then the pile controls you. DDG probably hasn't crossed that point yet, but self-justification is one of the steps on that path.
Firefox and Google Chrome probably have the equivalent of many small high quality libraries embedded in them, implementing 'business' logic or protocols, that could be reused in more places.
I guess a large scale study on github could be done, with a graph analysis to show potential "cut off" points in codebase.
It’s a bit telling that they linked to the GitHub repositories rather than specific lines of code they were talking about.
To me it looks to be trying to uphold your anonimity until you commit (click) through to the site/link. But certainly other ways they can approach this if it really bothers people.. I'd prefer DDG doing the lookup.. or having no fav icons.. over my computer going and downloading all my bookmark or other source icons
Maybe this should be an opt-in rather than an opt-out feature?
Edit: as pointed out by warpspin in another comment, this is about the DDG Browser, not search results.
This is mostly a UX issue IMO.
https://duckduckgo.com/settings#appearance
Uncheck the very last option "Site Icons"
How if you type a url into the browser how do you stop the browser from sending that url to ddg to get the favicon?
I’ve been an avid DDG user for years and it worries me that DDG staff don’t see why this is an issue. We shouldn’t have to trust your privacy policy if you minimize exposure.
Can you tell how many visited site A and also site B?
Generally speaking. Mine is shielded with lead.
I just switched to DDG browser a week ago and will now be looking for a new browser now. I hope you know this is not an appropriate response to the situation. Especially because all you guys do is preach about how much you protect your users’ privacy. Now you’re here asking us to trust you not to abuse our data and just linking us your privacy policy. I’m sad to say that my faith in the DuckDuckGo company and team is now lost.
I'ts not like we couldn't have predicted this disaster.
One more reason to never trust a companies "word". Show me the code.
If that's true, then I am so glad they ghosted me when I applied there.
Sounds like you'd prefer him to have run a message past management/public relations first?
Not as worse as publicly denouncing an honest engineer while referencing his paygrade. I hope there is no affiliation you have with DDG to be honest, because this is much, much worse.
I was once an honest engineer too, publicly. Being honest in private is enough for me now. It’s a lesson worth learning.
That said, it’s not like a single HN comment will make or break a company, so if they’re really just a rank-and-file engineer, I hope the company won’t come down on them too hard. A simple “don’t do that” would suffice.
I use DDG and the possibility of getting a statement directly from an engineer conveys much more trust than a carefully crafted PR statement ever could. I would think again about using it if the company does indeed come down on employees that live the values the company writes on its flags to have honest and transparent business practices.
That said, I am careful too when I state things about my company, even if I believe there is nothing to hide. Still, people that think it isn't the place for others with knowledge to comment are often not too impressive and would have difficulties in convincing me that privacy and transparency are real goals instead of just looking decent enough.
Furthermore the naming of management of DDG creates a stark contrast to the suggestion for more professional distance. I don't like PR very much as you might have guessed, but like a good design it needs some congruence.
If people find out that you just shut up for your company, it might give people the wrong impression about their business.
By commenting on an ongoing PR crisis without consulting management, you are both undermining their ability to respond in an effective way — imagine how strange it would look to see a “Hey, X from <company> here” after an existing one was already posted — and you’re acting on your own rather than in a team. You’re a part of a team; how could you think it’s a good idea to act alone?
Of course, I am talking to my former self with this comment, since that’s exactly what I did at S2 when working on HoN. It was a mistake, and I gave the community the wrong impression about the company’s priorities.
You have to understand, when you’re given money to do a job, you’re not given authority to become that job. Just because your job is getting beat up on social media doesn’t mean you should just jump in and go “Hey, that’s not true!” It doesn’t matter whether it’s true. Here, let me pretend to be DDG:
“Hi, Shawn from DDG here. You’re right; this was an oversight on our part. Obviously we dropped the ball on this. To clarify, we were unintentionally gathering the data as a side effect of our favicon service. <some technical details here>. We’ll be acting immediately to reverse this, and we’ll be enacting policy changes to ensure that user privacy — our core mission — is maintained going forward.”
But that’s not what they said. And if you’re gonna tell the community the opposite of what they want to hear, you’d better be in charge of the company’s Telling The Community Things division.
All you have to do is to read this comment thread to see the kind of damage that a single statement by someone affiliated with the company can do.
Agree, didn't mean to imply that.
The repeated handwaving that no one in your company is ever going to do something bad or stupid when the browser phones home for what amounts to a cute sticker is extremely suspicious.
Nobody in the company at any point thought that it could be a problem?
Your strict privacy policy mean nothing by the way, because you are a USA company and you must respect your local laws such as the patriot act, which are not very privacy friendly.
Curious to know why this is an issue.
An online favicon generator will create these variations
<link rel="apple-touch-icon" sizes="57x57" href="/ico/apple-icon-57x57.png">
<link rel="apple-touch-icon" sizes="60x60" href="/ico/apple-icon-60x60.png">
<link rel="apple-touch-icon" sizes="72x72" href="/ico/apple-icon-72x72.png">
<link rel="apple-touch-icon" sizes="76x76" href="/ico/apple-icon-76x76.png">
<link rel="apple-touch-icon" sizes="114x114" href="/ico/apple-icon-114x114.png">
<link rel="apple-touch-icon" sizes="120x120" href="/ico/apple-icon-120x120.png">
<link rel="apple-touch-icon" sizes="144x144" href="/ico/apple-icon-144x144.png">
<link rel="apple-touch-icon" sizes="152x152" href="/ico/apple-icon-152x152.png">
<link rel="apple-touch-icon" sizes="180x180" href="/ico/apple-icon-180x180.png">
<link rel="icon" type="image/png" sizes="192x192" href="/ico/android-icon-192x192.png">
<link rel="icon" type="image/png" sizes="32x32" href="/ico/favicon-32x32.png">
<link rel="icon" type="image/png" sizes="96x96" href="/ico/favicon-96x96.png">
<link rel="icon" type="image/png" sizes="16x16" href="/ico/favicon-16x16.png">
<link rel="manifest" href="/ico/manifest.json">
<meta name="msapplication-TileColor" content="#ffffff">
<meta name="msapplication-TileImage" content="/ico/ms-icon-144x144.png">
Nonetheless, the browser can see this when parsing the page and choose the appropriate path.
DDG is unneccessaryly producing (aggregating), transmitting (and collecting?) very sensitive user data here, which is just the opposite of data protection. I can't even understand why they try to justify their actions. It's like omitting the seat-belt in a car, then telling customers that this was required to make the in-car entertainment system more usable.
[1] https://de.wikipedia.org/wiki/Datenvermeidung_und_Datenspars...
The transmission of ip address alone, which is necessary for the TCP request to happen, deanonymizes the request enough to not be considered anonymous within the GDPR framework.
GDPR Article 5 (1) c: "Personal data shall be ... adequate, relevant and limited to what is necessary in relation to the purposes for which they are processed (‘data minimisation’);" - this is the "Datensparsamkeit" you mentioned.
Exceptions from Article 7 do not apply: The user has to give wilfully give informed consent, which he cannot do as the privacy policy of the browser omits the information that all visited domains are transmitted to DDG servers.
GDPR Recital 30 "Natural persons may be associated with online identifiers provided by their devices, applications, tools and protocols, such as internet protocol addresses, cookie identifiers or other identifiers such as radio frequency identification tags."
Oh, and the fact I'm downvoted for a purely informational comment additionally does not shine a good light on DDG.
"When assessing whether consent is freely given, utmost account shall be taken of whether, [..] the performance of a contract[..] is conditional on consent to the processing of personal data that is not necessary for the performance of that contract."
As DDG's favicon-hack is not strictly neccessary for operating the DDG-browser, DDG would need to give users the option to opt-out of the favicon-retrieval, otherwise they may have "forced" the users to consent to the data processing, thereby voiding that consent as far as the GDPR is concerned.
Their appstore pages link to the generic privacy policy of the company instead of for the browser specifically. This alone will usually already NOT be accepted by the supervisory authorities at least in Germany - the Landesdatenschutzaufsichten usually require a product specific privacy policy being linked for it being valid (personal experience), or at least making clear in the general policy what applies to the product, what not. And as mentioned, the fact visited domains are sent to the DDG servers (also outside the European Union) was not mentioned at all in that policy when I looked some minutes ago.
Wouldn't they need to give users the option to opt-in, under GDPR?
Any human readable ways of dealing with that?
IP-adddresses are considered personally identifying information. TCP requests transmit IP addresses.
Under the strict interpretation of the GDPR, a lot of things which are common outside the EU might be illegal, like e.g. embedding Google Fonts. To be on the safe side, people usually at least list these external dependencies in their privacy policies to construct some kind of "consent", but till we have more actual court rulings, this is a huge problem area.
For the problem at hand, it is pretty clearly illegal, as it's not only an ip address transmitted, it is a combination of ip address plus visited unrelated domain. This allows the creation of profiles. It does not matter for the GDPR, if the profile is ACTUALLY created, the pure possibility of creating it any time is enough to be a problem.
They might have to prove that their privacy policy is indeed GDPR conformant and that their service works as advertised, but in practice this is likely more about public trust that legality.
Art. 4 GDPR (1) clearly makes the (ip-address, visited domain) tuple personal data Art. 4 GDPR (2) defines "processing" data, and the pure "collecting" of data, even if immediately thrown away, is usually already considered "processing", therefore the GDPR applies.
If you are doubting this, just for a moment imagine, instead of the visited domain they would have sent all form data, including for example credit card data, you entered somewhere on a third party webpage to their central server and did not mention the fact in their privacy policy.
Do you really think then there is "nothing of interest for the GDPR" just because they do not actually permanently record that information? It would clearly be a violation. But to the GDPR, the importance of that data is equal. In fact, the domainnames might actually be more important to the law, as article 9 establishes event stricter rules for "sensitive" data about e.g. health or sex life of a person, and the domainnames might just leak that information.
If the TCP request carries personal data like the name of a visited website plus the user's IP address, then it "breaks the GDPR rules" in so far as you now have to fullfil your GDPR transparency/consent etc. duties /before/ sending that request.
Maybe not all website names look like sensitive data to you, but some website visits you surely want to be treated as sensitive, personal data (like names of hospitals, doctors, political parties, religion etc.).
Or put another way, a TCP request sent by your app from my computer can not be considered anonymous.
When that happens, you have to do insanely stupid seeming stuff like explaining in your privacy policy even, why your application, which is clearly for accessing a webservice, actually needs the Android permission to access the internet at all, true story. So I am pretty sure, an app sending visited domains to a central server outside the EU without even a mention of that fact in the privacy policy will cause problems with them, should they ever check the app. Of course, other countries might handle this differently, as Ireland shows.
So whoever thinks my interpretation is overly broad should first have the decency to step forward and actually explain why, instead of hammering a button and second, talk to me again after he had a meeting with the responsible authority and listen to THEIR interpretation of the GDPR ;-)
People should not mistake my interpretation with endorsement of the overly broad text of the GDPR itself.
Edit: I'm speculating here. But specifically because of the way you've replied here and on Github, my actual level of trust in DDG team went down.
Besides, how do you handle Intranet, VPN sites, and auth-only sites where DDG's god-tier favicon parser in the cloud couldn't fetch the URL anyway?
- Would you be ok to use a third party for this with same privacy policy?
That must be the worst justification for this possible. Favicons. Complicated to locate? Who are you trying to fool, 5 year olds?
Dear DDG, you are getting complaints on GitHub and Hacker News. This is not the general public, it’s people who understand the issue. You should definitely reconsider whether you’re doing something wrong.
why?
If you think the next time I hit the shitter I'm not going to be looking for a new browser, you're dead wrong.
Just do the basic checks and then fall back to a DDG logo, no one cares that much about the favicon.
... is completely irrelevant. Even if they were trying to save babies from a fire (which they really aren't) it wouldn't excuse the fact that they're doing something orthogonal to their stated policy and sole reason for existing.
Everyone makes mistakes, that's not the point. The point is to correct them when they're found, instead of digging one's heels in the ground and pretending it's nothing.
I accept DDG's statement that this is about a favicon and that they "do not collect or share any personal information", and despite that, I also agree with others that DDG should be on the safe side and just stop doing this small thing. It's just the safer and more moral thing to do (So DDG, as many are suggesting, plz stop doing it. Today is good).
But... the reaction here is "they made a mistake, let's pile on like kids in a playground" ignoring the genuinely huger issue of the amount of info and mining that google et al. do. There's no measure of proportion in the responses, someone is making a mistake then there's a wolfish, pack-like desire to get stuck in and hurt someone.
Which is why politicians rarely admit mistakes, because it's taken as a sign of weakness, not strength, to admit you were wrong. DDG isn't the big evil on the web but from reading some of these you'd think it was the 2nd google.
This isn't about DDG, just the proportionality of responses in public errors and what society you'd like to have.
(no affiliation to DDG)
The reaction would have been actually a lot different if someone from the company admitted the mistake and promised it will be changed.
Update: Gabriel Weinberg has promised to change it, linking it here so it does not get buried in the pile of comments: https://news.ycombinator.com/item?id=23711597
If they are confident that this feature has no privacy implications they are right to defend the point, despite what people say or think
People don't run DDG, the company
People can use another search engine if they want
I'll keep using DDG anyway
Where's my pitchfork?
You are faulting someone for defending thier own argument. You suggest that people who do not cow and apologize to the mob deserve the anger and retribution the mob has to offer.
People have a right to think differently and express themselves without threats, bullying, or shaming.
The mob does not deserve apologies. The comment above is spot on - we've lost all sense of proportionality.
It is an indication of the modern online mob sickness that they always demand others beg for forgiveness.
What emotional void are mob participants trying to fill with the apologies of others?
you obviously should be allowed to make a mistake and be forgiven for it. that does not mean that i personally would ever forgive any `company` that markets itself as pro-privacy after its been caught gathering data on its users.
i could forgive the people working at the company and would definitely expect future employers not to hold that against them, however.
but if a `company` does something while claiming to stand morally opposed to exactly that.... proves that it doesn't actually care about the topic. it just wants the publicity for marketability, discrediting them entirely for all future communication.
in this particular case, i wouldn't go that far however. they weren't gathering any data on their users if i understood it correctly. it was just a badly implemented feature, which will get changed
Once people have gone down the avenue of earnestly reporting private information leakage, the correct answer is to investigate. DDG decided not to do this and instead dismissed the problem completely out of hand and ignored it for almost a year without taking any action.
No one asked for an apology, they asked DDG to admit fault and then to fix the problem. ie "we shouldn't have done that" not "we're sorry we did that."
You keep saying "mob" but these people didn't collect to harass, if you actually read all of the comments the vast majority are people who are (rightly!) concerned about their data privacy advocate built software leaking every visited URL.
Well hold on now. If there's a valid technical argument, and it's not a violation of privacy, why doesn't that make sense?
If people are so distrustful of DDG that they don't believe that argument, why use their browser under any circumstance?
Let's assume DDG is a great and honest company and will collect all the info, but never use it for anything bad. Guess what, they can get still hacked and all the info leaks out.
This is a horrendous breach of trust that they WERE collecting it however, and I'm glad they got caught. It will not be tolerated. They'll have to change this or face a revolt.
By using DuckDuckGo you're doing the opposite of ignoring privacy issues in Google products.
And hence the reaction. Why use DDG at all, if they're not safely protecting your private data 100%?
Why use a condom which doesn't protect you 100%?
I mean, I agree with the principle of storing as little data as possible but this isn't a huge deal in my eyes.
Anyways, what’s this got to do with Google? “Privacy browser violates basic privacy to do something useless” is actually ridiculous. And the response is even more ridiculous! How does literally every single other browser do it?
I’m disappointed because I put my reputation on the line to recommend DDG to users based on...privacy. But here we see they actually do not hold true to their stated values. And they don’t even seem to care.
Of course they do. They sell users' privacy for advertising dollars.
I am not defending DDG here, as they are clearly in the wrong - but let's not pretend that their error is even close to what Google does.
Because I don’t think they do.
What does DDG 'sell'?
Which is to say, how much have you paid them?
If you're not paying for it, then you are the product, that's the inevitable trade. There is no free lunch.
Didn't see anyone claim that this was on a google-level of bad, more like pointing out that google started out as a small company wanting to "do no evil", but slowly turned into what it is today.
Is it really that weird that people are worried that this might be the first of many small steps down the slippery slope?
The point is that people are reacting as if it was.
> But instead to react professionally and contritely you made it worse to stamp on the shards to make sure no useful piece of trust will survive.
> I've just de-installed the Duckduckgo app and also won't be using their search engine anymore. Trust is lost. Their CEO can put his statement where the sun doesn't shine.
Sadly people simply derive satisfaction from piling on like this. The pop psychology explanation is that it's because people are dissatisfied with their own lives and are lashing out at anything that allows them to vent that underlying frustration. It sounds plausible, but it also sounds like it might be an over-generalisation.
I think it's certainly fair to say - as depressing as it is - that it may be somewhat in our nature to behave in this way, that most or all of us may possess this characteristic to a greater or lesser extent, and that the current political, cultural, economic, and media climate is only serving to amplify that tendency.
Thus, dark patterns emerge. Point is, once you get used to 'that' online-aura, you unwillingly carry it with you wherever you go. Human brain does not have buttons to switch between modes. It's comically easy to get into this mindset and hard to shun it.
It's not one mistake, but several. Other then the initial mistake there is also the sloopy reaction and the fact they just closed the issue without bothering to fix it. And this was 1 year(!) ago. Nothing changed in the meanwhile. Now someone pushed it to public and after just some hours they reopen the issue and promise to fix it.
This is the reason why people react loud, because it works. And often it's even the only way that works.
The mistake is rather an area of improvement where they can change something that respect privacy by policy to something that respect privacy by design.
It has zero implications if you trust DDG and good privacy is not based on blind trust. Keep in mind that you also need to trust the government under which DDG acts to not require them to disclose this data, trust the government to not put black boxes in the DDG data center, trust DDG's security apparatus against external state actors, trust any rogue DDG employee to not use this data and so on.
My opinion is that DDG should have never made this choice in the first place, but as far as I am concerned this is at the level of an implementation detail that can be improved, not as if DDG was intentionally using user data in some non-private way.
That just shows how much also the small things matter.
If you only care about "the biggest" or "the worst" you'll never get anywhere...
Man. You just described 2020.
Anyway, I’m now using the DDG browser, which until today, I didn’t know existed. I think DDG will do the right thing, ultimately.
It's really quite amazing that when a company that's hitched it's brand entirely to privacy first commits a big privacy faux pas, hides it for a year, and then doubles down on it not being a problem, you have somehow managed to turn the top voted thread to a discussion on the failings of other companies instead. Bravo.
Huh? It's an open source project. Maybe you consider them closing the issue on GH to be "hiding it" but I don't. And the plenty of people talking about it in that thread still apparently don't, either.
IMHO their response made sense. You're already trusting DDG with your search history which for most people might as well be a list of 99% of the domains they visit. If you trust their privacy policy for that, I don't see the big deal here.
I do agree that it should be changed, but only because as is it seems imperfect and I seek perfection in most things. Again, it's an OSS project. AFAIK, any one of these people comprising the screaming masses are free to fix it themselves. Personally, I don't think it's worth my time.
Most complaining or being heavily critical about DDG are probably already upset to the point of abandonment with other services and they don't want the same trend to happen to this competitor (DDG). This sort of reaction is, IMHO, due to poor diversity of viable competitors.
In our societal structure, competitive options are the only things that keep power in check. I'm personally not entirely convinced you can have a reasonable amount of diverse competition in our economic system and there are some inherent equilibria that we tend to converge on over and over again in a market space (without corresponding massive social equalibria shifts).
If you do have any sort of faith left in our economic system, then you certainly want competitors like DDG to be different and be successful. Even if you don't have much faith, outside of say stringent regulation, supporting these sort of competitors is really the only practical option we have in the current state of affairs.
And for the record, collecting your browser history just to display a stupid favicon is the most ridiculous excuse I've heard in a long while. And I am not going to blindly believe them because they said that's what they use it for.
As a privacy-first company, for them to make any decision that is for performance but adds additional privacy risk shows they make the wrong decisions.
They also left this for over a year after being told about it and part of the only reason it was caught was that their browser app is open-source. What about all their closed-source services, like their favicon service their browser apparently relies on.
We shouldn't blindly trust any company and a privacy-first company should be willing to assume we don't trust them and therefore it is up to them to prove everything they do.
I get a few of the responses on their Github page are now non-helpful, but the big point here is that by DuckDuckGo taking this series of actions (implementing the browser like this, ignoring a valid bug report about it, and only reopening the issue after massive push from users) it shows that DuckDuckGo isn't as privacy-focused as I thought, nor many others.
I've been using DuckDuckGo as my sole search provider for several years now, on all my devices, but I'm concerned about what else they may have done in the sake of "performance" over privacy.
Though I agree the the implementation could be better, they should just check the head and the root for the icon and if not found that's that. But the possibility of something be used for malicious ends does not entail confirmation of it being used that way. Just because we have knifes in our kitchen does not make us automatically guilty of stabbing people. I find it easier to believe that this was actually, if misguided, an attempt to solve a problem rather than a nefarious plots to track users across the web.
What about genocide too? Please people, stop with these "but the other bigger unrelated issue should get more visibility".
> Which is why politicians rarely admit mistakes, because it's taken as a sign of weakness, not strength, to admit you were wrong.
Can you point me where DDG admitted they were wrong doing this? They didn't... they just explained why they did it but completely ignore the greater issue because they consider themselves "good". Just like that politician you may talk about, or Google, or whatever.
This is about DDG.
What about genocide too? Please people, stop with these "but the other bigger unrelated issue should get more visibility".
> Which is why politicians rarely admit mistakes, because it's taken as a sign of weakness, not strength, to admit you were wrong.
Can you point me where DDG admitted they were wrong doing this in their first response? They didn't... they just explained why they did it but completely ignore the greater issue because they consider themselves "good". Just like that politician you may talk about, or Google, or whatever. They are part of that bigger issue you mentions.
This is about DDG.
Luckily that pile of kids in a playground made them realize that mistake, they would have ignored otherwise (like they did on their first respond).
Source: https://www.theregister.com/2019/11/07/ubiquiti_networks_pho... https://palant.info/2019/08/19/kaspersky-in-the-middle-what-...
Time to check what my ubiquity router is up to once I get home from work.
A direct correlation exists between the revenue Google receives for selling data and the quality of its search. Google focuses completely on tracking and search, with privacy behind a far far away afterthought (if it's a thought at all).
The argument is "Trust DDG". That argument is being attacked as "DDG's founder has done bad stuff in the past, it's likely DDG will do bad stuff, so I won't trust it". That seems to be attacking the argument to me, thus not an ad hominem.
[My opinion on trusting DDG] - [Reason for my opinion on trusting DDG].[More information about the reason]
In this case, because I don't use the "personal attack" to reach my conclusion that the person was wrong, I don't think it would be a case of argument ad hominem.
But if someone read it like this:
[One reason for my opinion on trusting DDG] - [Another reason for my opinion on trusting DDG].[More information about the second reason]
Then I am using the "personal attack" to reach my conclusion that the person was wrong, so I think it would be a case of argument ad hominem.
My comment wasn't written very well, and I'll try to write better comments in the future. Not that adding an ad hominem to a valid argument makes it invalid, I guess?. But it's still good to avoid fallacies, and to write one's comments so they're likely to be understood the way one meant them.
If you're willing to sacrifice search quality for privacy, as in switching from Google to DuckDuckGo, then you might as well take a step further and switch from Google to Searx/Ask.Moe.
If you want to market yourself as a champion of privacy, then the absolute minimum criteria should in my opinion be that your codebase is open source.
Because they have a good privacy policy. They would face legal consequences if they were lying. Nation state actors can presumably override privacy polices but it's better than nothing.
I couldn't figure out which searx instance to use and found no good way of knowing who to trust, most of the engines i used were broken and telling me to find another searx engine.
It looks like it's pulling most of its info from duckduckgo anyway.
Personally I'd rather trust a known entity than an unknown entity anyday, especially when the unknown entity is slow, complicated, buggy and broken in many places.
If you hosted your own instance then it would be a lot more reliable since the IP wouldn’t send a suspiciously high amount of requests.
As for your trust argument, I couldn’t disagree more. You choose to trust DuckDuckGo, who happens to be closed source, because of their branding. The same way people trust/trusted Google/Apple/etc. because of theirs. This thread is a perfect example why being open source is the most important thing for any privacy service (because otherwise this privacy leak likely wouldn’t have been discovered, and people wouldn’t have known that the company so carelessly violate people’s privacy and fail to correct it when people point it out.. it should really make you wonder what’s happening in the search engines codebase).
Don't really understand why they do extra work to get worse results... This feels to me slightly worse than just a privacy concern, it's a misunderstanding of their domain which leads me to the question of what else do they not fully understand.
The good news is that you can have the DDG search engine as a default in other browsers.
(I understand that the DDG browser is probably not their main focus and any lack of knowledge can potentially be just on their mobile browser.)
Also different expectations
Note also that these are favicons for results that DDG has already given you. This isn't tracking your clicks. The list of sites that appear on the search result page is not new information to the search engine that just gave them to you.
EDIT: Like other commenters, I was not previously aware that DDG had a browser, and my comments were about this behaviour for the search engine results page.
The service is private as we do not collect any personal information (e.g. IP addresses) on any requests for this or any service and the requests are all end-to-end encrypted.
There are so many options.
Even if you don't like the reply it's good that we're getting replies.
I understand utilizing a single service/feature to accommodate multiple platforms is a big win from a programming standpoint. But if it carries the risk of losing the trust of your user-base while at the same time risks breaking the core mission statement of the business (privacy), it's probably not worth it.
Why use that term when it clearly can't be true or even seems applicable/relevant?
https://github.com/duckduckgo/Android/blob/b2131d7d2f47fb09d...
"Tired of being tracked online? We can help."
And then they track you.
Yes, that might not be intentional and is used "just" for the favicon, yes they might not use the info on the domains you visit for tracking you today, but the data is there.
Why not use that data tomorow "just" to see what kinds of pages their customers (browser users) are visiting so they can better place their ads.. and then maybe some other idea.. this is a path that many such companies went ("don't be evil").
You either respect the user privacy or you don't - there is no middle "just for this little feature" ground
My advice:
Install ungoogled-chromium: https://github.com/Eloston/ungoogled-chromium
Install these extensions: https://github.com/gorhill/uBlock https://github.com/ilGur1132/Smart-HTTPS
There is also a Chromium extension that lets you install from Chrome Web Store: https://github.com/NeverDecaf/chromium-web-store
Set duckduckgo.com as your default search engine with a blank home page. But you could also use @pkrumins home pages of https://techurls.com or https://finurls.com as nice home pages.
Use Mullvad VPN: https://mullvad.net/ (They are EVEN available on F-Droid now, which is AMAZING)
Security harden your Android device: https://niftylettuce.com/posts/google-free-android-setup/
Security harden your Mac: https://gist.github.com/niftylettuce/39597a7b3bc0660ffe1e09d...
P.S. If you need email forwarding for your domain name, you can use something I made. https://forwardemail.net - it is 100% open source.
Follow me @niftylettuce on GitHub and Twitter for more
Disc: Googler
>Faster, less bloat
Stock Firefox doesn't actually have that much bloat, and it's not noticeably slower than Chrome on reasonably modern hardware (i.e.: most page loads are near-instant, same as in Chrome on a good connection).
>Extension support for Chromium is way better too
Until Google decides that your extension is unworthy of being in their store. This notably happened with Pushbullet quite recently: https://news.ycombinator.com/item?id=23168874
... Until Google inexplicably restored it a few days later, but not before lots of accusations were thrown around.
---
IMO it's also worth noting that ungoogled-chromium is (obviously) an unofficial fork of Chromium. Google may at any time change Chromium so substantially as to either require Google integration at some fundamental level for even the most basic functionality, causing too much work for such a low-profile effort to continue, or just make Chromium closed-source. With Firefox, that risk doesn't exist because of the business motivations of the company that develops it.
I haven't used Chrome regularly in the last year or so, but it would get noticeably slow by the time I had 3-4 windows with 10-20 tabs each. Firefox hasn't really had that issue yet.
Years ago though, Chrome was noticeably faster than Firefox. That changed (IMO at least) at some point in the last few years.
I got used to Firefox's dev tools and can't find my way around in Chrome's.
People here are:
- early developers from majir browsers
- extension creators
- creators of major web properties
- people who spend their work days on the web
- etc
seriously. Don't underestimate HN.
Edit: and as someone who's been very into customizing and extending browsers since early Firefox: your ideas about Firefox vs Chrome tells me that you are either
- deeply biased
- use a subset of browser features that is too small to realize the problems with Chromium (in which case I suspect your 100x superuser is wild hyperbole)
You can come back when Tree Style Tabs including related sub-extensions works as well on Chrome as on Firefox.
https://en.wikipedia.org/wiki/Impossible_Is_Nothing_(video_r...
Brave is the only browser that includes strong ad and tracker blocking by default, which is the best way to ensure your privacy on the internet.
In fact, every decision I've seen from them makes me think they care the most about user privacy, period.
Previous discussion: https://news.ycombinator.com/item?id=23474842
You can install uBlock Origin in 10 seconds from the GitHub release page (see previous link shared). This is vastly better than anything I've seen for ad-blocking.
Also that just makes me trust them more – they are the only browser vendor that are trying to find business models that don't involve selling my data or selling out to someone who does.
Edit: I feel it is slow because it has extra bloat added. Like I said, take my comments here with a grain of salt because I'm a very biased power user. I respect the efforts of these developers regardless of what project it is, the focus on privacy and building something different is truly awesome.
Ditto for developer / power-user experience – it's still chromium. Just with an ad-blocker written in Rust built-in and some other features.
Data or stop spreading FUD. There is nothing "slow" about Brave.
A Brave fork meant to address this issue. Time will tell.
- All new load balanced infrastructure - Browser extension + API wrapper - Support for pixel tracking blocker (opt-in checkbox or TXT setting) - Smart alerting - Globby/regex support
If you follow my Twitter or the GitHub releases you will get updates.
Advertising your Twitter for the advice of "switch to somewhat well-known browser X, install these very common extensions and use a VPN" is also a bit ... odd.
I haven't checked all links but some things need to be updated, for example Skimmer Scanner is gone from the Play Store and Yalp Store is abandoned and doesn't work anymore, you should be using Aurora Store. I'd also recommend Aegis or FreeOTP+ over FreeOTP for 2FA. NewPipe is better installed from this[1] repository until this[2] issue is solved.
(But this instance, the favicon service, is not a good privacy-functionality trade-off)
By using bangs you're sending your search history to DDG even when using search engines that aren't DDG.
https://support.mozilla.org/en-US/kb/how-search-from-address...
I maintain a set of such keywords in my Firefox, that I use on iOS and Android too.
The easiest would be to just use Google btw, as it does a reasonable job to give you the website you're thinking of just by mentioning it in the search query, e.g "Frozen imdb" (though it does a perfect job in this case with just "Frozen").
If you use DDG, because I'm assuming you value your privacy, sending your search history to DDG when you could avoid it doesn't make much sense.
I did try firefox search keyword anyway once a while ago but it isn't that easy to configure and the setting were lost not too long after I started using it (I don't remember the details but I think my profile was corrupted by ecryptfs and the keywords didn't import to a new profile or something like that). Not importing them isn't a horrible choice (if that is what actually happened) considering there doesn't seem to be an easy way to even list them, but there should at least be a distinct import/export for keywords. The search page of preferences has a keyword field next to search engines but it doesn't seem to actually list them nor can you add a keyword from that page. All in all, it doesn't seem like a feature Mozilla cares about at all, although it would be a nice feature with a bit more work. Use from right click search on text selection would be another thing that would make it much more useful.
I tend to use ! in two circumstances, one as others mentioned when a DDG search didn't return what I wanted (and I am fine with letting DDG know that since it is the exact same search I just searched DDG) and secondly when I am looking for something on a particular site. Some of the second case would be best done via Firefox keyword search but the advantage of DDG is that they have a lot of obvious names going to the obvious place so rare specific searches are much more convenient than doing anything to set up a list. Since I very rarely use such searches, there isn't anything DDG could learn about me from avoiding just those searches going to DDG. OTOH, using site searches more frequently would be a win for privacy.
I was astonished by this at first, but I think you must mean "you're sending all searches performed with bangs to DDG". I worried that you meant somehow the browser search history was being sent to DDG, but that seems impossible.
If I don't want ddg to know what I google, ofc I won't do it by typing in !g in ddg...
I actually use bang functions because I want to help ddg out by informing them when I'm unhappy with the results I got from them.
That's what they promise to not do and the whole point behind many people's decision to use them.
And it might be obvious for some, but it still makes no sense :-) given the browser is capable of doing it.
But yes, when I go to ddg and type in "!g <thing>" after having typed in "<thing>", I'm on some level hoping that this is somehow informing ddg that their search results for <thing> could be better.
Now, if they have somewhere stated that they are not going to do this, then yes, it would be bad.
Except that you do, exactly in the way that the reporter of the issue explained to you.
But you choose to patronize them and ignore the issue.
I really like DDG - it works good, it is fast and it does not use my personal search for giving me "better ads" or "better search results" that put me in a filter bubble.
But there is a different issue here at play; because of errors like this the whole DDG brand gets a bad rap - and thats not only bad because of the risk of people losing a google alternative but because it is real easy to exploit situations like this for google-like companies to give an impression that "all this privacy thing is bs, all companies work in a same way". There are a lot of people that are not really sure is this "privacy thing" is worth the inconvinience of swiching to some other search engine/browser/app and situations like this one are not helpful in that regard.
Lot of folks are aware of this and are displeased for risking brand confidence of such a visible privacy-concerned company for miniscule gains like performance gains for fetching a favicon for the first time - just fetch the favicon after you display the rest of the page and cache it, maybe dont even try to fech it if the connection is poor - who cares really
DDG has repeatedly said that they have "not collected any personal information".
For example,
1. Does the service store the fact that it got a request for a domain?
2. Does it store any ID along with that information and if so, how unique is that ID? How is it generated and what is it linked to?
3. What other information is stored along with the request?
4. How does DDG process this information?
5. Who has or can get access to this information?
I'm pretty sure 90% of websites provide one in a standard way. If not, just draw a letter there, or anything.
But I don't know. I think that either there is more to this story, or DDG team completely lost common sense.
It's much easier if I don't even have to trust you. Please change this.
1. Check for <link rel="icon" ...> tag(s)
2. Check for /favicon.ico
3. ...give up?
Someone correct me if I'm wrong!
Really shouldn't be complicated enough to need a special service to handle "edge cases".
Another comment mentions web manifest - I guess try those first before meta tags, or whatever order the standard says to use. I mean, we're talking a web browser here, it's designed to do these kind of tasks.
On a technical level of course its doable but in reality it's a complete waste of data and processing, not to mention it could take a long time to show up. I imagine they have these favicons all cached on their side so they can quickly send the right file down and/or do this processing if needed.
That being said maybe they should just not use a favicon if it's that big of a deal.
In that case, yeah, I don't think the icons are necessary to show at all...
I think that distinction needs to be made. I think DDG should treat this app as a web browser which means phoning home to this endpoint is unacceptable.
I never got how people trust companies like ddg or Brave. If you don't trust Google and Apple why would you trust a smaller company in the same jurisdiction. They will be forced to hand out all data as well regardless what they say.
[0]: https://github.com/mozilla-mobile/android-components/tree/ma...
The problem with that is that it requires users to "trust" DDG, which is not how the world works today. If you are a company that collects info, and you expect users to trust that the info will remain safe, secure, and never get misuse that is downright foolish for anyone to believe a word of that.
We all know that DDG cannot claim it's impossible for them to get hacked and have all that data leak out. Hacks happen all the time and so the solution for DDG is to simply NOT collect the data, rather than collect and claim it's all secure.
And we all also know DDG has (or will) get a NSL (National Security Letter) from the NSA to secretly turn over the data anyway, and when that does happen the DDG employees are not even allowed to admit it ever happened.
this would never happen with a consumer-facing product from apple or google; someone would have to MITM their whole OS to discover phone-home
Personally, no.
Since DDG owns the icons.duckduckgo.com service and the domain you were interested in is in the request to icons.duckduckgo.com, you've sent the domain to DDG's servers.
This makes sense to me and is not alarming. Getting favicons actually is difficult to do robustly; many applications and websites use Google's service to do so, which then leaks the request to Google: https://www.google.com/s2/favicons?domain=ycombinator.com
Putting this logic in the client is not feasible. You want to send requests directly to every shady site that shows up in your search results, load their pages in the background, work through network delays and HTTP errors, and parse out the location/format of the favicon files?
DuckDuckGo hosting this functionality themselves is also a positive. They have previously been burned when the Web of Trust service they were originally using was found to be farming data, and turned it off immediately once discovered. Processing, hosting, and serving the icon themselves prevents that from happening again.
This is not to say that DDG is perfect: links you click do seem to be redirected through a /l/ page on their domain, which can cause problems: https://lapcatsoftware.com/articles/duckduckgo.html
So if you're a privacy browser. Don't. Favicons are not essential. Or, at the very least make them "opt in" and explain it.
DuckDuckGo can't do that because they don't have access to your browser's history and favicon cache, precisely to protect your privacy.
EDIT: This reply is wrong if DDG is the browser.
As for complexity, I'm sure users of privacy oriented software would prefer to not have favicon in small percentage of corner cases rather than leaking browsing history.
Wait, what? Why?
And I have no clue how search results come into this? Of course the browser isn't fetching favicons on search result pages, that's the servers job, but that's not what this is about.
It's really odd that this is the top comment right now, given that even the headline makes it clear its about the browser, not the search results.
I don't see why not. Browsers are constantly dealing with these very issues. It's one of their core competencies.
I am surprised the user is not complaining about this instead of the favicons. Their privacy policy goes on about the privacy implications of Referer headers and instead of calling out browsers for sending Referer by default, they instead give themselves power to record all the user's clicked results themselves. The Referer problem is something that can be solved by the user at the browser level through, e.g., using a client that does not send Referer, browser extensions/plug-ins that can control headers sent, or perhaps with a local proxy to remove the Referer header.
Unless DDG has changed, these prefixed result URLs are the default. It is possible to get unprefixed result URLs using the "lite" version of DDG however that is not the default. "Privacy-focused" search engine chooses less private default. News at 11.
I recently noticed that DDG has started redirecting queries submitted via POST to /lite/. The redirect is to the same domain. No explanation. I have a custom client that does not follow redirects and I now have to submit two sets of HTTP headers instead of one.
These guys are trying to make money from advertising just like everyone else. They have to be very particular in the methods they use to do it -- check the exceptions in their privacy policy -- but it is the same game. Ads and affiliate links. That sort of business and privacy are always going to be at odds with each other.
If I recall correctly there is a way to turn off the prefixed result URLs without having to turn on Javascript, by adding/changing a URL parameter, e.g., kh. However that is not the default setting. Not very friendly toward the privacy-conscious user who has Javascript disabled and does not read HN to find out about otherwise undocumented URL parameter usage.
For certain queries, I can actually get a different first result based on the HTTP method I use.
curl -Huser-agent= https://duckduckgo.com/lite/?q=xyz|grep result-link
curl -Huser-agent= -dq=xyz https://duckduckgo.com/lite/|grep result-linkSee here: https://help.duckduckgo.com/results/rduckduckgocom/
DDG's privacy policy also goes on about the privacy implications of User-Agent headers combined with IP addresses.
So let's say I take what they have put in their privacy policy to heart and I stop sending a User-Agent header. In response DDG sends prefixed result URLs? WTF?
Using haproxy, for example, I can use a "modern browser" and send no User-Agent header. I still get prefixed result URLs.
You're free to disagree, but in that case you can make use of the option to change the default that they provide for you.
I would not call haproxy "referer masking software". In any event, a proxy is not even needed.
Modern browsers are open source, right? Users can edit the source and remove the code that sends Referer header.
Even easier, I wrote my own http client. I can send any header I want, or none at all. According to DDG's privacy policy this is a good thing.
> You want to send requests directly to every shady site that shows up in your search results, load their pages in the background, work through network delays and HTTP errors, and parse out the location/format of the favicon files?
Looking at DuckDuckGo search results and visiting a page that you navigate to are two different things.
1. DuckDuckGo search results:
DDG already returns the search results so there's no privacy violation to return the favicon or the URL for the favicon in the list.
2. Any page that isn't a DDG search results page:
Use client side logic to locate the favicon. This means worse performance but better privacy - which aligns with DDG's goals.
If you want to optimise this then DDG could send the client some precalculated Bloom filters with info about known sites. The client could use these to try certain methods of favicon retrieval first.
What does Google have to gain from it? Google has pretty aggressive anti-scraping protections to protect against this exact behavior, so why would they allow Startpage to get away with it?
What does Startpage have to gain from it? Unlike DDG, they don't seem to have any core product, so they fully depend on Google's goodwill which is very shaky grounds when it comes to a long-term business.
"You can’t beat Google when it comes to online search. So we’re paying them to use their brilliant search results in order to remove all trackers and logs."
Isn't that exactly like DDG but Google instead of Bing?
https://help.duckduckgo.com/duckduckgo-help-pages/results/so...
> We also of course have more traditional links in the search results, which we also source from multiple partners, though most commonly from Bing (and none from Google).
So basically it's all Bing and I see no effort to reduce that dependency as you claim.
Mojeek [1] is also a good privacy option that doesn't get talked about as much as DDG. They distinguish themselves by having their own search index.
[0] https://www.runnaroo.com/search?term=python+unit+testing
Go here and turn your "web and app activity", your "location history" and "ad personalisation" off, if they aren't already:
https://myaccount.google.com/data-and-personalization
If you do that, there's no evidence that Google is more privacy invading than DuckDuckGo, since you're left with taking their word for it. And frankly I trust a big, bureaucratic company more than I trust a startup. The ideal would be to trust technology (e.g. end to end encryption, open source) but that's not the case here.
Note that I also use DuckDuckGo in my Private Mode, as Firefox allows setting different search engine for Private Mode. I do that because it's better to compartmentalize your online personas, plus I keep my "web and app activity" in Google on, with deletion after 3 months.
They should probably change the behavior to how it’s suggested in the thread, but I’m still going to use DDG over alternatives for the bang feature.
The favicon is acquired from DDG servers for the result you've just retrieved from DDG servers.
How is this leaking anything? What additional privacy would you gain from getting the favicons from the domains directly of search results delivered by DDG?
1. End user does a DDG search for "food" 2. The "food" query returns a list of search results, these results have each have a link, DDG wants to display the favicon for each link. 3. To be clear, DDG does not store or log the IP address of the user doing the query. They do, however, know what was queried, so they know "somebody" somewhere searched for "food". They have to know this, they are a search engine after all. 4. Since DDG wants to show the favicon "privately", and they dont want to put that logic/work on the client side (which could leak your IP), so instead DDG finds the favicon internally. 5. A DDG server, completely separate from anything search-related is then tasked with finding the favicon for your "food" query results, lets say the #1 result is www.allrecipes.com, so a DDG server goes to www.allrecipes.com and finds the exact favicon location. 6. The "found" favicons are then stored in a cache, and displayed from the cache like this: https://external-content.duckduckgo.com/ip3/www.allrecipes.c... (and if no favicon is found in the local cache, you get a grey arrow by default) 7. I'd like to note, even with all this action, DDG doesn't know if you actually "visited" www.allrecipes.com, they simply know that some anonymous user did a search for "food", www.allrecipes.com was a search result, and a favicon was displayed. They dont know who searched for it because the users IP is not stored anywhere, they dont know if you visited www.allrecipes.com, they prevented you from leaking your IP to allrecipes.com since they didn't force the end user to load the favicon.
So whats the issue? What am I missing here?
PS: You know this works because after doing all these searches for food and seeing allrecipes.com (and even clicking allrecipes.com result in the DDG Mobile App or browser extension), guess what? allrecipes.com doesn't follow you around with re-targeting ads! Why? Because DDG prevented that from happening!