DuckDuckGo Removes Pirate Sites and YouTube-DL from Its Search Results | Hacker News Reader

DuckDuckGo Removes Pirate Sites and YouTube-DL from Its Search Results

torrentfreak.com

864 pointsTangerineDream446 comments

Question: does anyone actually know what DDG does with user data? Like they market themselves as a "privacy respecting" search engine, but how much of this is truth?

I'd imagine there's good money in convincing people they have privacy because then they'll provide more interesting data.

Has the company ever been audited? Why should they be trusted to not compromise user privacy? Imo at least Google is honest: you know when you use their products as intended you have no privacy, and they don't try to hide this

Edit: since DDG isn't open source like searx, how do we know there is ANY truth to their marketing claims?

Edit: Just for accuracy, the browser extensions are open source. But as far as I know, the actual search engine isn't

Edit: They made over 100 million in 2020. They clearly can (and should) get an independent audit. It's shocking that they haven't had a single audit. Even startpage has

If Duck really collects user data, the moment this is found out, they’re dead, so for that reason alone, they probably don’t do it. The alternative is that they’re betting everything on nobody ever finding out which sounds crazy.

A certain philosopher named Occam would suggest that the simplest explanation for nobody finding out so far is that it is three-letter agencies who might be the ultimate funders and data buyers.

Surely the simplest explanation would be that no one has found out because they don't actually collect the data.

Is that really the simplest explanation?

Or is the simplest explanation that they're not lying and they do what they say they do?

That’s a pretty wild misapplication of Mr. Occams blade, I’d think. There are much simpler explanations than that.

A certain philosopher named Occam would suggest that appeals to conspiracy are much harder to substantiate than appeals to incompetence or irresponsibility.

That's Hanlon. He has a "razor" that says that anyone who maintains plausible deniability should be assumed to be innocently blundering, no matter how much that blunder benefited them.

"For nothing ought to be posited without a reason given, unless it is self-evident (literally, known through itself) or known by experience or proved by the authority of Sacred Scripture."

So I guess you're going with interpreting the scripture as "render unto Caesar that [data] which belongs to Caesar"?

I'm interested in the thought process that made you conclude that that is the simplest explanation

There are so many issues with DDG, but nobody cares [1]. Idea of having privacy focused search engine is strong. Then use kagi or searX.

[1] https://lemmy.ml/post/31321

I think we care, but sometimes DDG marketing is pretty strong.

That link you provided makes a good case for why DDG is not the ally we need in the privacy war.

Can you provide a TL;DR of the article? I'm a DDG user but open to considering alternatives, never heard of kagi and tried searX a long time ago.

Currently, whenever DDG doesn't cut it, I add the !sp bang and get Google results through StartPage. I know that SP is owned by an advertising company but I still prefer them to straight up Google. I tried using searX to get proxied Google results but it required a few extra steps so I don't usually go for it.

Checkout https://you.com - it's privacy focussed, ad-free and also supports bangs

and, unfortunately, it's zero-HTML and requires modern JS to function. :(

I use kagi as it still uses all of the ddg !bangs, and at some point they will ask me to pay for them (they are in free beta ATM) rather than show me ads which I'm excited for.

How is kagi more private? It requires an account to use, and it’s going to be a paid service. So it always has a strong user ID associated with every query. Whether it is not stored and not processed cannot be verified. It’s the same story about trust.

(Kagi dev here)

Being a paid service means Kagi's incentives are different. Instead of selling your data or profiting from feeding into ad-tech food chain, we are interested in selling you a subscription.

We do not associate queries with an account, as a matter of fact we do not save queries at all. You are right that this can not be verified, the question you need to ask yourself is what kind of incentive we have for misusing user data in any way.

Unless users want us to do start using their search history to improve quality of searches (something we may introduce as an option in the future as we've been asked to), we have no reason to do so.

> Being a paid service means Kagi's incentives are different. Instead of selling your data or profiting from feeding into ad-tech food chain, we are interested in selling you a subscription.

Until your growth slows down and you realize you can get away with doing both.

There are so multiple industries (TVs, Games, Streaming) where paid users are still being monetized to the maximum extend possible by showing them ads, selling their data, etc. that this argument is just dishonest now.

Personally I find that you did a great job with Kagi. I agree that the incentives are important, and I hope that Kagi will be successful and may prove that a different business model is possible. I'm also inclined to trust Kagi, but that's just that, trust and not much to lose.

When I read about the founder and their privacy policy, I get the impression that this is something they care about.

At the same time, as far as I know there has been no independent audit. Considering they made over 100 million in 2020, they clearly have the finances to fund an independent audit. It would also improve their reputation and clear up some of the uncertainty about their collection of user data in practice

Even better (but more unlikely): they could open source the search engine so we all can audit them.

Open sourcing still won't solve the issue right . What they open souce needn't be exactly the same they run on their servers

Back when DDG first started, Gabe was asking for ideas/opinions on features they had and might add, and I talked to him on the phone for a few minutes. He seemed a pleasant, sincere and honest person, and I've seen nothing since to contradict that. Unlikely he's got that kind of time any more ... here's a (2018?) interview (with Vivaldi, they'd just made DDB their default search), listen for yourself. [https://www.youtube.com/watch?v=OU9U26IWSYE]

Suspicious of most of the rotten-meat smells on the net as I am, I use DDG for everything. Compare the person(s) who said "Don't be evil" (then...) to picking the name DuckDuckGo.

> If Duck really collects user data, the moment this is found out, they’re dead

It'd take a whistleblower for anyone to ever find out. What are the odds of that happening? I don't think we can count on someone who is being paid by a company to tell us about their actions when their livelihood/gravy train depends on it and they may be opening themselves up to legal problems for coming forward.

Not may people have the sort of integrity that folks like Snowden, Klein, or Tice demonstrated and even those that do can be pressured into keeping silent.

>It'd take a whistleblower for anyone to ever find out.

No it won't. If you have a hypothesis you can just test it out. Go submit some fake personal data and then see if it shows up anywhere else. This trick is as old as using a honeypot name or email address to test if a service signs you up for junk mail.

> Go submit some fake personal data and then see if it shows up anywhere else.

this is not a valid test, for a whole lot of reasons. especially not in terms of internet search engines.

One reason for this, is that the specific things you search aren't what's actually valuable. Companies use that data to make guesses about you. It's those guesses, not your specific search terms, that are used to profile you.

So why not just try to get yourself profiled a certain way? Search a bunch of stuff about a specific disease and click a bunch of the links and see if you start getting targeted ads about those diseases? Well, who is to say who sold that data? Was it the search engine, or the websites you clicked on?

Not all the data collected on you is used for targeted ads, and of course even when it is used for ads not all of those targeted ads demonstrate a proven link back to your search teams. If you see an ad for cheap airline tickets maybe it's because you've been searching for stuff about another country, or maybe some algorithm thinks your bipolar and entering a manic phase since they are more likely to buy tickets. Maybe it's just an ad for cheap airline tickets because the airline wants more people to fly right while weather warms. You can't know.

Better ways to test what DDG is doing behind the scenes might be things like get hired there in a central role. Have the government or an independent auditor review their data collection practices.

>this is not a valid test, for a whole lot of reasons. especially not in terms of internet search engines.

Then come up with a better test. I'm sure you can think of something that answers those questions to a reasonable level of confidence and can be targeted at any data broker. This is literally what a lot of independent auditing actually is. The reality is, you can't go and work on the inside at every data broker and that defeats the purpose of an independent audit anyways. So you have to come up with something else.

As I said: "Better ways to test what DDG is doing behind the scenes might be things like get hired there in a central role. Have the government or an independent auditor review their data collection practices."

> The reality is, you can't go and work on the inside at every data broker and that defeats the purpose of an independent audit anyways. So you have to come up with something else.

It's true that it's unreasonable for people get a job at every company they have to work with. The real solution is regulation and oversight to protect our data from being collected, sold, and used against us. Experts all agree we need it and they've been saying so for a very very long time, but so far there's not been much effort from congress.

Lately, almost every conspiracy theory has been proven true. Or a more outrageous conspiracy theory is based on somewhat less aggravating true facts, that are then misconstrued as a more far fetched conspiracy:-)

https://en.wikipedia.org/wiki/List_of_conspiracy_theories

I am very interested in seeing you explain which ones have been proven true and your citing of sources.

Example A - Outrageous But False Claim's: Pizzagate

-> "Debunked conspiracy theory that after Hillary Clinton's campaign email hack falsely claimed emails contained coded messages that connected several high-ranking Democratic Party officials and alleged human trafficking child sex ring"

Minor But Still Serious and True Facts:

Anthony Weiner was elected for New York congressional terms as Democrat politician 7 times. In September 2016, claims were published that Weiner had engaged in sexting with a 15-year-old girl from North Carolina, and devices owned by Weiner were seized as part of an investigation into this incident. The report prompted a criminal investigation and Weiner's laptop was seized. Emails that were pertinent to the Hillary Clinton email controversy were discovered on the laptop; The Wall Street Journal reported that federal prosecutors were weighing whether or not to bring child pornography charges against Weiner over the incident On May 19, 2017, Weiner pleaded guilty to another, unrelated sexting charge of transferring obscene material to a minor, and was sentenced to 21 months in prison, ordered to pay a $10,000 fine, and was required to permanently register as a sex offender.

Example B - Crazy Conspiracy Theory: Government Was Stealing Dead Bodies to do Radioactive Testing.

True: “Project Sunshine" https://en.wikipedia.org/wiki/Project_SUNSHINE

Example C - Conspiracy:

The Dalai Lama is a CIA agent.

Minor But Still Serious and True Facts:

...According to declassified intelligence documents, The Dalai Lama earned $180,000 in connection with the CIA’s funding of the Tibetan Resistance to the tune of $1.7 million per year...

https://en.wikipedia.org/wiki/CIA_Tibetan_program

Example D - Conspiracy: The government is using it's vast resources to track citizens.

True: https://en.wikipedia.org/wiki/Edward_Snowden

"Lately, almost every conspiracy theory has been proven true."

Example A: You showed it isn't true. Example B: not present on list(1) Example C: not present on list(1) Example D: not present on list(1)

(1)https://en.wikipedia.org/wiki/List_of_conspiracy_theories

Well done.

I am not the NSA. And the comment was not meant to be understood as talking about every crazy nutter conspiracy theory aggregated into a Wikipedia page.

So what are you stating with the comment? All conspiracy theories are false or did not originate on some less outrageous event?

I'm simply pointing out your comment, "Lately, almost every conspiracy theory has been proven true." is incorrect. Almost every conspiracy theory is a very encompassing term. If you had said something more along the lines of some conspiracy theories have had some truth to them I would be in agreement.

Because there is a tiny bit of truth in something doesn’t proof the whole sentiment to be true. It’s a fallacy for everyone like you who wants to cry gotcha and run away with their hands on their ears.

Perhaps they just keep the data in a vault just in case it might become useful someday. Not many people need to know about that. It can be just some box sitting at the point where data enters the datacenter. That's the darkest theory I can come up with.

Anyway, I'm a happy DDG user (on devices where typing "!g" isn't a pain).

> The alternative [..] sounds crazy

I also think that it is unlikely but, let's not forget that some unlikely things are regularly found to be true.

In 2019 many thought that a global pandemic was unlikely, in 2007 they thought that a housing market crash was unlikely, and so on.

Personally, I think an independent audit would go a long way in clearing up my doubts.

I mean Okta’s core business is authentication so we’d expect them to do the bare minimum in disclosing security breaches and yet we saw how that went recently.

Thats the philosophy of every criminal entity including govts, police, security services and everyone else.

>you know when you use their products as intended you have no privacy

This isn't true. Google's privacy policy is not lax as you suggest it is.

Could you clarify a bit? If you are logged in they store almost everything, as is evidenced by Google takeout.

If you give data to Google it's not like that data is now public.

Unfortunately, Google takeout doesn't seem to give all the data. Last time viber stored "secret data" in google drive. I tried to download everything via takeout. It doesn't seem to include those data.

I read about a Github issue [1] where someone reports that all websites a user clicks on to DDG servers. Reading the employee's response was eye opening.

They literally do not care if it has a bad look, they just say "we don't collect your personal information." What??? They are literally admitting to collecting domains in the feed of the Github issue but then just copy and paste their manifesto and expect us to think it's fine. I seriously do not understand this.

[1] https://github.com/duckduckgo/Android/issues/527

This needs more visibility

Seems understandable to me. The explanation isn’t just “copy pasting” their privacy policy, either. You are misrepresenting that thread and discussion.

They’re not a perfectly secure E2E encrypted zero-trust system. They do require some measure of trust to use. This has always been true. Don’t use them if you don’t trust that they won’t misuse your data.

>Seems understandable to me.

The explanation seems superficially plausible, until you realize many (all?) other browser seem to work fine without a remote server to fetch favicons.

They're probably not collecting IP addresses or user cookies, but they will undoubtedly have a log of what their users search for, what results were turned, and possibly what clickthroughs happened from their search page. You can do all of that without retaining identifying information for the person who clicked through.

Worst case, if you go back and forth between google search and DDG, a comparative analysis might be able to identify you or people like you from those logs, but it would require some work.

I think most DDG fans would be thoroughly surprised DDG ever popped the hood or tried to verify their privacy claims, which is why I don't think DDG ever will.

Better to elude to a nebulous definition of privacy rather than give specifics. Even as a privately traded, for-profit company based in the U.S. with zero accountability for its claims, DDG doesn't even have that much IP to lose (as a Bing reskin) should it ever fold under a class action lawsuit.

I always wondered the specifics of their deal with Microsoft regarding the use of bing index. Guess we will never know.

What specifics? Wouldn’t they just pay MS for access to the API (though with some special volume discount that won’t be public)?

Yes, but what data do they pass onto MS and what do they do with that data? https://news.ycombinator.com/item?id=31051050

I asked Kagi, and they said they aren’t forced to send such data and aren’t.

And unless Ecosia (where the quote seems to come from) is extremely shady, they are only sending two octets and not three.

Did you keep reading? This issue was subsequently fixed.

> Hi all, CTO of DuckDuckGo here.

[... mucho explanation...]

> So, we went ahead today and implemented the change for both Android (#878) and iOS (duckduckgo/iOS#667) that will move this logic onto the client, and we will no longer be using the favicon service in our apps. These changes are currently in the release phase and are rolling out live now.

https://github.com/duckduckgo/Android/issues/527#issuecommen...

Here's my take:

Duckduckgo could easily be fully owned and operated by some three letter agency. The NSA is already able to go onsite and tap into the data that passes through corporations and they've been doing exactly that for decades (see Room 641A) and they can force corporations to keep silent about it using national security letters. You should already assume that every US based company is sending every scrap of data you give them to the state.

With no way to avoid your data from going to the state, what are you left with? Worries over companies collecting, selling, and using your data against you. That's a very real and perfectly valid concern.

We know that other search engines are doing those things, so it's best not to use them if we can avoid it. Duckduckgo might be doing those things, which at least gives us a chance, and even if they are it'd be better to hand your data over to several different companies than to give them all to one source (like Google for example) because the more data points any one company has on you the more control they have over you.

The worst case scenario would be that Duckduckgo is actually secretly run by Google and the data being collecting from the service is being used to help fill your dossier at Google but if that's the case we're never going to know about it until a whistleblower comes forward.

As defeatist as this all sounds, I do believe in taking steps to try to protect your privacy where you can, and I take many steps that go far beyond what most people are willing to, but we also have to accept the reality of the situation we have where our laws and regulations do not protect us, and there is very little we can do to protect ourselves but depend on others to do what they say. That's why I use duckduckgo right now. not because it's trustworthy (we can't know that), but because they might be and that's (sadly) the best option we have at the moment.

> The worst case scenario would be that Duckduckgo is actually secretly run by Google and the data being collecting from the service is being used to help fill your dossier at Google but if that's the case we're never going to know about it until a whistleblower comes forward.

The problem with this (and most other conspiratorial thinking) is that of course a whistleblower is going to come forward.

Are you thinking that every employee of DuckDuckGo that knows this (and many would have to), is paid so highly as to just be quiet? That not one of them thinks, "hey, it would be fun to be famous? It would be fun to expose this thing that is suddenly going to make me a hero to millions? I could write a book about it afterwards and make a ton of money..."

And of course, Google would consider this in the first place. Like "maybe this isn't a great idea, because secrets like that are hard to keep? And maybe this could destroy our company and that kind of risk isn't a great idea?"

This depends on the strange theory that every intelligence agency front or partnership has been discovered by the general public while it was operating, and that the agencies are themselves irrational for ever even trying it.

edit: sometimes the arguments against "conspiracy theories" become arguments against the current existence and operation of intelligence agencies. As if all of those employees did nothing all day, and all of those billions are flushed down the toilet.

> Are you thinking that every employee of DuckDuckGo that knows this (and many would have to), is paid so highly as to just be quiet?

I'm not thinking that DDG is secretly being run by Google at all, but I'm forced to admit that there's nothing stopping Google from doing it, that they have the money and resources to do it, and that it would benefit them to do it, so however remote, it is a possibility that should be considered right along side the possibility that duckduckgo is exactly what they say they are and nothing more. We have no evidence of either being true after all and can only speculate. If we're asking ourselves that the risks of using DDG are, we have to at least consider what the worst case scenarios might be.

So while I consider the possibility that DDG is being run by google to be possible, but highly unlikely, just as a thought experiment, why would anyone but a very small number of Google employees working at DDG need to know it was run by google and sending data back home? Any other DDG employees don't need to be told. If google employees ran the servers, DDG employees would be free to work on the front end and day to day stuff entirely unaware.

> The problem with this (and most other conspiratorial thinking) is that of course a whistleblower is going to come forward.

This is not a guarantee at all. Edward Snowden came forward, but only after many many years and not one of his co-workers in the same role did. DuPont knowingly poisoned people for decades and not a single whistleblower ever came forward. Johnson & Johnson knew that their product (which we were covering infants with) caused cancer but there was no whistleblower to break the story to the public.

Over and over and over again we only learn about horrific acts committed against the public by corporations after decades of gathering scientific data and court actions specifically because people were "paid so highly as to just be quiet". It's a fact that whistleblowers are the rare exception and not the rule.

The idea that companies are so afraid of whistleblowers or their reputation that they wouldn't do evil things is demonstrably false as we've seen example after example of companies who knowingly put profit over human life time and time again and even when there were whistleblowers the companies often faced zero meaningful consequences beyond paying back a small percentage of the profits they made while building a body count.

Whistleblowers are awesome, and they remain a very import check against bad actors, but do not count on them always being there to save you and don't expect that fear of them will prevent a company from committing acts far far worse than logging your internet searches.

DuckDuckGo was never meant to defeat spying by three-letter-agencies.

It's meant to protect us from tracking by advertisers.

I don't think it makes any sense for a three-letter-agency to run it. They can just NSL duckduckgo. They obey the law, as this news item shows clearly.

I agree. Can't dismiss the possibility since anything advertising itself as a service to protect your privacy could make a very nice honeypot for the state, but there's a market for pro-privacy services that private companies are happy to serve and nothing stopping the state from just taking whatever that want from them anyway.

or, How I Learned to Stop Worrying and Love the Panopticon

How I stopped loving panoticon and ended up marrying it.

Love it or hate it, the Panopticon is here and we're stuck under it's watchful eye until laws and regulations catch up. In the meantime, just do your best to protect your privacy. DDG has been my search engine for many many years. I recommend it. I still can't fault anyone who questions how much we can trust them. Simply put, right now we can't. I still need to search the internet though.

I love conspiracy theories!

They can be fun, but I wish we had actual oversight and regulations to protect us so that we didn't have to think about what kinds of shadowy things companies might be doing with our data.

If someone is worried about DDG, I can't blame them. They have every reason to worry, because we have no protections. Personally, while I sadly can't discount the possibility, I'm pretty sure DDG isn't run by google or the NSA, and they've been my search engine of choice for a very long time.

I used them nearly exclusively and recommended them to all my friends. Once they started censoring content for political reasons (Ukraine), that ended instantly.

I've always been a bit suspicious that despite having seemingly no way of making revenue they manage to plaster every corner of the internet with their paid ads. Like, aren't they supposed to be a damn nonprofit or something? Makes absolutely no sense.

They serve ads on DuckDuckGo, though.

I’m not aware of them being non-profit. Not that any particular tax status precludes ad spend.

DDG makes revenue from serving ads and participating in affiliates networks.

https://help.duckduckgo.com/duckduckgo-help-pages/company/ad...

> censoring content for political reasons (Ukraine)

what type of content did they censor? Do you have a link with more info?

Here's one: https://mashable.com/article/duckduckgo-search-engine-russia...

People complain about search engines like Google being full of garbage and fake info. DDG takes actions against that and people cry that it’s not fair.

Are they supposed to just let governments astroturf their way to the top with propaganda?

They certainly shouldn’t be singling out one specific country’s astroturfing for down ranking.

I agree, but a response to a topical issue is not necessarily evidence of that.

One of the reasons I left google is precisely because they started downranking content for political reasons, just as DDG started doing.

Which search engine are you using now? I bet we will find them downranking for political reasons as well but I bet you are fine with their reasoning…

We are talking in circles. If I'm gonna use an engine that downranks for political reasons then I might as well not use a shit one like DDG.

No you are avoiding a simple question with a nothing burger answer.

Google. It provides better results than DDG in many cases. The one case where DDG provided better results than Google was unfiltered news and opinion. That is no longer the case, so I'll use the shitty search engine which works a bit better for things like technical searches.

I think some people are under the impression that it’s possible to build a useful search engine that is completely algorithmic and unbiased. It’s just not possible, though.

Any algorithm for which there is incentive to game, people will game. And legitimate sources often have no incentive to game the algorithms. There’s no one algorithm that will do everything perfectly. Eventually you’ll have some phishing scam, life-threatening suggestions, or illegal content popping up as a top result, and you’ll have to add manual exceptions.

I think it all comes down to the type of curation being done and if the choices are made out of objectivity of subjectivity.

Like prioritizing a legitimate website over a scam website isn't punishing the scam website because of a controversial opinion or the search engine operator didn't like the content, it's because the website is objectively a scam, it's easy to objectively identify it as such, and everyone agrees it's one.

When it comes to Ukraine vs. Russia and propaganda, it is entirely impossible to have objectivity. As angry as this will make some people, opinions on the war in Ukraine are subjective. Russian outlets shouldn't be ranked lower than Ukrainian ones solely because a lot of people are on the side of Ukraine.

I'm sure some Russian outlets are spreading objective falsehoods, and some Ukrainian outlets are spreading objective falsehoods. These individual outlets should be punished in search rankings, but to classify all Russian outlets as spreading misinformation and all Ukrainian outlets as objective truth completely demolishes the objectivity a lot of people want their search engines to have.

It's a difficult topic, I feel like I talked in circles writing this comment.

I think people are running with the idea that DDG is only punishing Russians. Russian media also loves pushing this idea.

I'm not convinced that DDG isn't downranking all junk. Russian disinfo spreaders and consumers just scream the hardest when their garbage is rightfully pushed aside. Same as the Q-worshipping crowd.

> it's easy to objectively identify it as such [scams], and everyone agrees it's one.

Is it actually easy to identify all scam content? Then why is it possible to find scams in search results?

> When it comes to Ukraine vs. Russia and propaganda, it is entirely impossible to have objectivity.

So should they treat every news outlet as equally ranked? Because I'm struggling to think of any news reporting that could be said to be free of bias or could be said to be completely objective.

> ...to classify all Russian outlets as spreading misinformation and all Ukrainian outlets as objective truth

Is that what they did? Because what I read was that they would "down-rank sites associated with Russian disinformation." So you have to dig a few pages deeper to get to 'Russian disinformation', and I don't see any anything indicating that 'Russian disinformation' means everything published by a Russian News outlet (re: the invasion).

I don't think something being a top search results implies that it is an objective truth either, especially with ongoing news reports.

>Is it actually easy to identify all scam content? Then why is it possible to find scams in search results?

I think to a knowledgable observer and moderator, sure. Algorithmically it's probably more difficult, which is why you still see it in some search results. These scams are less prone to subjectivity, which is why you rarely see Twitter uproars about a bitcoin pyramid scheme being booted off a search engine.

>So should they treat every news outlet as equally ranked?

Yes....

>Because I'm struggling to think of any news reporting that could be said to be free of bias or could be said to be completely objective.

....and that is the reason for why they should be equally ranked. You can see on the initial page load the full spectrum of reporting from MSNBC to Fox News. Of course one of them has to be ranked above the other in the UI (it's a list of items) but that should relate to objective things like what words from the search query appear, etc.

>Is that what they did?

I think what people take issue with is that we're trusting these sites to determine what disinformation is. How is DDG deciding what is and isn't misinformation? Without a clearly defined process for misinformation, this becomes subjective.

>> "Weinberg didn’t elaborate on the decision, or how the down-ranking will work."

But other sites are simply doing a blanket ban:

>> "Since then, the internet industry has responded by blocking access to Russian state-sponsored media outlets such as RT and Sputnik News for users in the EU. In addition, Twitter has placed warning labels on tweets linking to Russian state media. (Google News decided to de-rank RT and Sputnik News back in 2017 for allegedly circulating propaganda.)"

>I don't think something being a top search results implies that it is an objective truth either, especially with ongoing news reports.

Not right now, no, and that's the issue. Most people assume it to be though.

>>So should they treat every news outlet as equally ranked?

>Yes....

>>Because I'm struggling to think of any news reporting that could be said to be free of bias or could be said to be completely objective.

>....and that is the reason for why they should be equally ranked.

Just because you can find a fault with a source does not mean that all faults are equal. For instance, the AP should probably be ranked above the National Enquirer.

> I think what people take issue with is that we're trusting these sites to determine what disinformation is. How is DDG deciding what is and isn't misinformation? Without a clearly defined process for misinformation, this becomes subjective.

It is an inherently subjective process and always will be. There are organizations that have attempted to make the process objective (i.e. fact checking organizations) but they can all be criticized as subjective as well. Pure objectivity simply isn’t possible.

This kind of thinking is exactly why the Russian misinformation campaigns have been extremely successful (especially in Russia). Russia doesn't care about the individual outlets as long as they help spread lies, no matter how absurd or obvious. This doesn't have to be done as carefully or subtly as the western world seems to think.

Russia is discrediting truth as a concept entirely, the narrative being "we are obviously lying, but so is anyone else". That others might not be lying as much seems to be harder and harder to understand by the targets of these campaigns (based on firsthand witnessing). With objective facts / reporting discredited what's left is opinions. Opinions generated by a carefully managed army of influencers promoting whatever cause is deemed useful (from bloggers to friendly/financed foreign government officials). And this _is_ done with the secrecy and subtlety you'd expect.

It might not be possible to have complete objectivity regarding the invasion of Ukraine, but imho a blanket ban of any media with Russian ties would be a heck of a lot better than to let Russia continue to destroy objective reporting, one of the foundations of our modern society.

"Manufacturing consent" was written in 1988. Did Russia's campaign against the truth was already happening?

When I was growing up, certainly before even RT existed, it was common knowledge that journalist will cover the events according to their own (or bosses) interests. IME (not russian, but also not american), the idea of objective reporting is the anomaly.

I think we should try to enforce transparency and make sure that false information can be disputed.

If Russia sends armies of trolls to clandestinely influence social media consumers as part of their information warfare operation then we should try to stop that.

But if Russia officially publishes its own position under the name of its own news agency or in their government media then we should not censor it and we should not keep it off western platforms where it can be clearly marked and disputed.

I want to know how Russia justifies its brutal war of aggression against a neighbour. I fully expect that their justification is full of lies and distortions. But it's truth in the sense that it is in fact how Russia justifies the war. I want to know that truth.

For example removing tass from their index

https://duckduckgo.com/?q=tass

https://duckduckgo.com/?q=site%3Atass.com&t=ffab&df=m&ia=web

https://www.google.com/search?q=site%3Atass.com&tbs=qdr%3Am

Not sure what is going on with those URL params, but I see results just fine at https://duckduckgo.com/?q=site%3Atass.com and in any case an easier way to do things is to not use a site search but just put the domain or name in the query like 'ukraine tass' and something from that site will usually come up on top.

I don't. I can't get a single hit on duckduckgo from tass.com. I tried "site:tass.com bucha" and variations there of.

GP's links include a filter for "past month".

Which enlightened search engine did you switch to that doesn't bias results?

If I'm gonna use a biased engine why use a shit one like DDG?

Which search engines are not "shit" in your estimation?

I'm not OP, but I'll weigh in:

Google - it's a really good search engine, but obviously biased and doesn't care about your privacy. I use it on my work machine.

Brave Search - I'm not sure if another service (like Bing) powers their results, but it works pretty well for my personal use cases.

I switched to Brave Search and generally am pleased. I did notice current events is somewhat lacking. I was searching one of the wildfires in NM yesterday and no dice. Otherwise for general information or more "static" type info I've been pleased.

Code / stackoverflow is also somewhat lacking on Brave Search, but I'm a big fan of the ability to do !g to redirect the request to Google (I don't care if they see that I'm trying to figure out why my build is failing c: )

I switched to Brave Search and browser recently. It's based on chromium/Blink and feels a lot snappier than Firefox/Gecko.

I've tried Firefox a few times over the years and always switched back for the same reason - it didn't feel "snappy" and some things didn't render properly. Something has changed recently though. Both these issues appear to be completely gone. And now that I've got the Containers extension installed, it's superior to chromium in all material ways, except maybe the dev tools.

I compile my own fork of FF for myself and my family for years, and i've switched us all to brave search for the same reasons as ` colordrops`.

`yegg` can do whatever he want's, I just want no part of it.

Maybe it's time to use those search bookmarklets again. When I think about it, many times I already know where the answer could be.

I guess I have thousands of bookmarks. I always search my bookmarks first, known sites second, DDG next and Google last. In both Firefox and chrome there is exclusive bookmark search within the addressbar. In FF it is * and then space.

Can you explain how an unbiased search engine would work? Sounds like a billion dollar idea tbh.

Compose an index by randomly spidering IP addresses. Return search results by randomly selecting an exact string match from the index. Rats, probably not a billion dollar idea after all!

One could reproduce a similar effect by surrounding your query with double-quotes on Google.

> Return search results by randomly selecting an exact string match from the index.

It could potentially surface results that you could not have found otherwise. A random dump of links, each one's relevance you must determine on your own. Or another way to put it: you do the ranking. Not a very fun idea to most.

The whole point of a search engine is curation, and excluding garbage or in the case of Russia/Ukraine, making willful misinformation less prominent is an act of curation.

A search engine is largely two things:

(1) crawling and discovering all available resources (2) parsing a user’s request for a resource of some kind and providing the user with a means of accessing “best matches”

This process is inherently, unequivocally and fundamentally biased. It HAS TO be. There is no abject, absolute, canonical result for a search engine query. Ddg/google “censor” () all kinds of results constantly. Malware sites, sites that just mirror content from stackoverflow, sites that may cause public harm, etc. That’s the whole point.

Another note: they may be downranking those results because users are unsatisfied with their quality. DDGs job is to help the user find what they are looking for, and if the user finds a bunch of Russian propaganda that is blatantly not what they wanted when they search for something (perhaps because Russia is exploiting SEO or using other techniques to inflate their position), why wouldn’t they tweak their product?

> The whole point of a search engine is curation, and excluding garbage or in the case of Russia/Ukraine, making willful misinformation less prominent is an act of curation.

Unfortunately it seems that in 90%+ of cases "willful misinformation" is just anything that contradicts the political views of the democrat-leaning Google management.

Please provide proof.

Hunter Biden laptop story is a good example of "right wing disinformation"... oops I meant reality!

Not sure what that has to do with anything or what point you’re trying to make.

The hunter biden laptop story was true and yet it was forcefully removed from many media outlets, more than any story I've ever seen. Glenn Greenwald was working the story and got pushed out of the organization he founded, The Intercept, because of it. There were very powerful forces aligned to blacklist the story to protect Biden's candidacy.

You're right to be skeptical. They are essentially a client state of Microsoft. Their results come from Bing and they are hosted at Azure. Their privacy policy is just vague enough to not rule out the possibility that Microsoft collects all the stuff that DDG says they don't collect.

They sure sound like a Microsoft shell company. Because nobody that's fully conscious will ever deliberately use Bing they had to get creative and rebrand it with the usual privacy and safety buzzword slogans that VPNs have perfected in the last years.

The most obvious problem with this conspiracy theory is that Bing became available in 2009, but DuckDuckGo was founded one year earlier.

Bing is (a) sound of having an idea. Perhaps that idea was DDG. bing.

DDG used Yahoo to start with; Yahoo had their own index at that time. DDG switched to Bing later, and since then Yahoo abandoned their index and started using Bing.

>since DDG isn't open source like searx, how do we know there is ANY truth to their marketing claims?

You wouldn't know this even if it was open source. Open source does nothing here. Looking at the source code will not tell you their data retention policies or what is actually stored in their databases. It will also not guarantee the source that you see matches what is on their servers.

It will tell you if their client doesn't send data to a server.

I use DDG not for any concern over privacy, but because the developer tools and results are so nice. !bangs are also excellent and a huge timesaver

`! mdn window.postMessage` .. so easy!

You can do this natively in every major browser too. You'd have to configure the triggerwords yourself, though.

Open source doesn't mean anything. They could just have a closed source service read DB/logs and do stuff with the data.

They have a privacy policy, and if they don't follow it then they could be liable for it.

Their use of cookies is fairly easy to inspect, although that doesn't prove they aren't doing fingerprinting or ip tracking.

> Question: does anyone actually know what DDG does with user data? Like they market themselves as a "privacy respecting" search engine, but how much of this is truth?

It doesn't matter.

Why? Because when going through the exercise of identifying risks in the system one can't assume the actors are benevolent and won't ever use the access+data they have for evil.

That's not at all to say all actors are evil and will always do the most harm possible. Many risks are never exploited in practice. But that doesn't mean the risk doesn't exist. It still does! And it might be exploited in the future (with companies, all it takes is a reorg that puts someone less ethical in charge).

Thus, when doing your threat modeling exercise, for the purposes of identifying risk, assume the various actors could do as much damage as they possibly could with the access they have.

So concretely, when I evaluate risk on google vs. DDG: I won't take into consideration any "privacy respecting" marketing, that's not important. What matters is how much damage can each party do, which one is less risky?

Both get my search queries which is inevitable for a search engine. So there's that risk but it's a wash.

But google has its tendrils woven into far more points from which they can and will correlate data. Google analytics, AMP, gmail/gsuite, chrome (for people using that), also most people have an active login session with google most of the time, etc.

DDG has a much smaller footprint on the internet from which to correlate data.

Therefore, even assuming both parties are equally evil, DDG presents a smaller risk.

Question: does anyone actually know what Microsoft do with the data that they get from Duck? Other syndication partners of Bing I have spoken to told me they are expected to pass on first three octets of IP. So with other data, for example useragent, location this could likely be used for fingerprinting.

Another syndication partner of Microsoft is far more transparent in their privacy policy than Duck: "Bing: IP address (obfuscated), user agent string, search term, and some settings like your country and language setting. We never communicate IP addresses along with search queries. We only send IP addresses to Microsoft in obfuscated form, meaning we remove parts of the IP address when we sent it."

Even ignoring the indirect data correlate risk; it is also poor risk management to use one company for your entire digital life. It means having to keep actively abreast of what you lose access to in the event of an account ban & do contingency planning because correlated failures in systems are much more painful than random ones.

Better to just have 1 Google service you use and find someone else for the others. Their search index is a little better than DDG's I think, but the difference is pretty small compared to the risks you outline and the risk of Google going rogue and deciding I'm a Russian or committed some other similar crime & need to be booted from their services.

> > Question: does anyone actually know what DDG does with user data? Like they market themselves as a "privacy respecting" search engine, but how much of this is truth?

> It doesn't matter.

> Why? Because when going through the exercise of identifying risks in the system one can't assume the actors are benevolent and won't ever use the access+data they have for evil.

Storing any data in database is just asking someone to either steal it or abuse it. So only solution is to not store it, if it's not critical for operation. And if it's critical, store privacy data in encrypted form(and keep decryption keys away from database, so database breach won't jeopardize keys, like in different business unit in corporation). One such example is logs, store some of the data encrypted, and if you need it(with a really good reason) ask it to be decrypted. Also you can encrypt various forms of data with different keys, and make accessing one type easier while more privacy critical data will be harder to get access to.

Something feels off about DDG, especially once I found out that they funnel you into downloading their iOS app in order to sign up for their new browser’s waitlist.

It’s like a dark pattern that an advertiser would use, not a privacy-focused search engine.

It's amazing how much collateral damage is caused by our horrible copyright laws. Mostly just so the MPA/RIAA can protect their roles as gatekeepers of what we're allowed to see and hear.

They can put enormous pressure on even the wealthiest and most powerful companies to act as copyright police on their behalf. Even Google is afraid of them. ISPs are forced to spend huge amounts of time and money working for them. Now duckduckgo is being strong armed into doing a bunch of free work for them too? Maintaining lists of websites and domains to block and removing links to even non-infringing material like youtube-dl just to keep from being sued into the ground.

I don't know what it'll take to rein in these guys, but I doubt the courts will be the ones to do it. So far courts seem fine with the idea that ISPs must permanently ban users from their service over nothing but repeated unsubstantiated claims of infringements which is an insane amount of power to give any industry.

Has any US politician ever run on a platform that includes copyright reform?

Lawrence Lessig

I forgot about that! What a missed opportunity that was for this country. He's exactly the sort of person we need to see more of in politics.

His campaign was much more about electoral reform than copyright reform.

https://en.wikipedia.org/wiki/Lawrence_Lessig_2016_president...

Which is broadly more important than copyright reform…and broadly Americans don’t seem to particularly care about either of them all that much.

Either would have been a win as far as I’m concerned.

The case for copyright reform is much more clear-cut, though. There's no plausible argument for the current extent of copyright.

Both are pretty clear cut.

Lessig, in particular, doesn't push for throwing out the electoral college all together (to avoid system shock) but rather to move towards proportional representation the state level.

That is, if your state has 10 electors and votes 60:40 R:D, then you should allocate 6 electors for R and 40 electors for D (rather than the current "winner take all" method we do). This kills off gerrymandering as a tool and makes it so that rather than appealing to swing states, presidential candidate would actually have to convince every state that they are the right choice (because you don't just automatically get California. The difference between a 55:45 cali and a 90:10 cali would completely change how you campaign).

Right now, Presidential candidates campaign to swing states which is why you get weird things like presidents talking solely about auto manufacturing or natural gas extraction.

It wasn’t a missed opportunity. It was intentional. When he became popular enough to get into the main debates the Democrat party changed the rules to exclude him.

It has led many to believe the party leaders don’t want his message out there

There is no political will in the US to change the copyright system. Aside from some technologists, I haven't seen anybody who actually wants that. You will anger basically everyone else in the information business. Writers, researchers, artists, architects, musicians, composers, filmmakers, actors, podcasters, even a lot of software developers, you name it. These people all depend on copyright enforcement to get paid.

That to me is the worst thing about repeated phrasing of this as some kind of evil special interest groups against everyone else. The "special interests" here are the people who produce the copyright material you want to access. You have to play ball with them or they will simply not be able to produce those things anymore.

Fully agree.

"collateral damage" just means...why can't I just consume everything for free, as if it's some human right?

Not true in the case of Yt-dl.

The point is not "consuming things for free", the point is neutering "the Biz".

I for one wouldn't mind to lose Hollywood as an industry, or Broadway, or Silicon Valley...

All of these industries are way too big and over-influential in all aspects of our lives, and leave us without any chance of a more human-scale alternative to counteract.

I mean, how many TV networks are owned by Disney? Do you really think that all that lobbying is just to keep people away from watching copies of Mickey's Fantasia?

This makes no sense. Disney is not stopping you from making your own home movies. You can even put them on YouTube or any of the other alternatives.

There's actually lots of good criticisms of Disney, but you didn't mention any of them.

It's not about my own movies that don't get made.

It's about all the other movies that don't get to be financed because of the 19th re-hash of Star Wars (or Marvel movie, or live-action remake of Lion King/Aladdin/Pocahontas/Dumbo/Bambi) that is going to be released and it's "guaranteed" to make another billion dollars in the box office.

It's about having two small kids who get bombarded with products from their franchises on every media outlet.

It's about them using their movies and works to promote whatever ideological propaganda that gets them in favor with government.

Film funding is not a zero sum game. There are lots of other places to secure funding. I've seen about 20 new indie films already this year.

The rest of your comment is unrelated and somewhat unhinged, and appears to be going off into a general complaint about advertising, it has nothing to do with copyright.

> I've seen about 20 new indie films already this year.

Good for you! Now go ask the other 99% of the population what they watched this year?

> The rest of your comment (...) has nothing to do with copyright.

It has to do with concentration of power, of which copyright (and any kind of "intellectual property") is one of the favorite instruments from big corporations.

I also used to fall for this "people are free to choose" techno-libertarian bullshit. The problem is that it is not in the interest of all the corporations for you to choose responsibly. They make the game so complex and so energy consuming that eventually people break in one way or another.

"Copyright enforcement" only benefits the big players and the corporate world.

But that's not true, if you didn't have copyright the biggest one would just copy everything and nobody could do anything about it. With copyright you have a way to sue them for ripping you off.

Or maybe copyright and "intellectual property" is precisely what leads to existing a "bigger one" in the first place?

> Disney is not stopping you from making your own home movies.

Excuse me? What if I want to adapt some Hans Christian Andersen story over 150 years old and get a cease and desist from Disney? Even if their adaptations are older than many people alive today.

Why do these corporations get to have functionally infinte state granted monopolies on trivially infinitely copyable data as if it's some basic right?

It means people being forced off the internet because of nothing but unproven accusations because they only have one ISP in their area.

It means artists who never publish their works because they're afraid of being sued for being a little too close to what someone else did once a century ago.

It means creative projects that never get started because while the will is there, the rights needed are impossible to obtain.

It means non-infringing works like youtube-dl get silenced and hidden from the public because they are inconvenient to media giants

It means accepting internet censorship without oversight.

It means DRM crippled hardware that you but are never allowed to own.

It means DRM crippled software that can never be preserved and archived in a usable format.

It means your culture and your history being paywalled off or locked away

If you this this is an issue of "I just want free stuff" you clearly haven't been paying attention.

No, it means almost none of that. Pretty much nothing you said has anything to do with copyright law. This is why I have a problem with this rhetoric, it very frequently goes off the rails towards something that doesn't help the goal at all.

> It means people being forced off the internet because of nothing but unproven accusations because they only have one ISP in their area.

This is not related to copyright law at all. I agree ISP monopolies are a problem, for any number of reasons.

>It means artists who never publish their works because they're afraid of being sued for being a little too close to what someone else did once a century ago.

Anyone can sue you anytime for any reason, that doesn't mean they'll win the lawsuit. You could make a good argument here about how copyright law is too long, but that point is lost in the rest of the sentence.

>It means creative projects that never get started because while the will is there, the rights needed are impossible to obtain.

No, that doesn't make any sense. This is a blatant misunderstanding of copyright that I see so often. Copyright doesn't stop you from creating new projects. You don't need to get the rights if it's a fully original work. You could also make a good argument here about how licensing is too difficult to obtain sometimes but that point is also lost.

>It means non-infringing works like youtube-dl get silenced and hidden from the public because they are inconvenient to media giants

This never happened. Youtube-dl is still up, if I recall for the short period it was taken down there were like 20 forks that popped up immediately.

>It means accepting internet censorship without oversight.

This also never happened and is contradictory to everything else you said, your complaint actually seems to be that there's too much oversight.

>It means DRM crippled hardware that you but are never allowed to own.

This is not related to copyright law. DRM is a purely technical means, not a legal one.

>It means DRM crippled software that can never be preserved and archived in a usable format. It means your culture and your history being paywalled off or locked away

No, it's not your culture or your history, you're not the author. What everyone misses with these comments is that some authors many not want their works to be preserved or archived.

>If you this this is an issue of "I just want free stuff" you clearly haven't been paying attention.

There's no coherent issue anywhere else in your reply and there's nothing to pay attention to. I've tried to make these comments constructive so maybe that can guide you towards what your real issue is.

> No, it's not your culture or your history, you're not the author. What everyone misses with these comments is that some authors many not want their works to be preserved or archived.

The moment an author makes something public and it gains significant traction is the moment it starts to be part of global culture instead of just only the author's.

Either publish the work with everything that it entails, including archival and assimilation of that work into popular culture, or don't publish at all. If you don't want society to archive your works, then it shouldn't even know of their existence. There's no middle-ground.

> This is not related to copyright law at all. I agree ISP monopolies are a problem, for any number of reasons.

It's a problem regardless. No industry should be allowed to force a 3rd party to cancel your service with them over mere accusations. The fact that there are countless examples of DMCA notices being sent in error, or intentionally sent inappropriately doesn't make the situation any better either. While I agree that ISPs should not be a monopoly and we should all have easy access to high speed connections at low prices from a large number of providers, that's not the reality we live in, it still wouldn't solve the problem since as long as there are finite amount of providers you'll still eventually be able to be kicked offline, and I shouldn't be forced from the provider of my choosing in the first place.

> Anyone can sue you anytime for any reason, that doesn't mean they'll win the lawsuit.

The people who decide not to publish aren't doing it because they fear random lawsuits from random people. Their fear is very specific. They decide not to create or publish their work out of fear of being sued under our overbroad and excessively punishing copyright laws.

Winning doesn't help if you're driven into debt and bankruptcy by the costs. While companies I've never heard of could randomly sue for any reason out of nowhere (a major problem on its own) the risk of that happening to me is very small. For people publishing music (even those within the traditional system) it's much much higher and it's already having an impact. (see: https://www.rollingstone.com/pro/features/music-copyright-la...)

I don't think this is unintentional. By increasing the risk of creating new works outside of relative safety of the RIAA's umbrella they can silence indie artists who don't want to sign their rights away.

> No, that doesn't make any sense. This is a blatant misunderstanding of copyright that I see so often. Copyright doesn't stop you from creating new projects.

See above - it does stop the creation of new works from people who either can't find the rights to do what they want, can't afford them, or fear the lawsuits they will face if they go ahead anyway. Sita Sings the Blues (https://en.wikipedia.org/wiki/Sita_Sings_the_Blues) is a great example of a wonderful creative original work that almost didn't happen because of our insane copyright laws. In the end, the creator was only able to get the work out to the public by releasing it for free and she's still at risk of lawsuits. I recommend reading the copyright section of her FAQ here https://www.sitasingstheblues.com/faq.html

> This never happened.

Strange take in a discussion of Duckduckgo removing links to the project form their search engine and immediately followed by how even though it was wrongfully removed before other people had forks so somehow that didn't matter. It mattered then, just like it matters now that DDG is hiding that tool from the public.

> This also never happened and is contradictory to everything else you said, your complaint actually seems to be that there's too much oversight.

The censorship issue is concerning what ISPs, hosting services, and search engines are forced to remove. No company should have the power to remove content from the internet without due process and right now, that's not happening.

Youtube pulls content down all the time in response to DMCA notices that should never have been enforced and that content stays down unless the uploader invests significant amounts of time and effort getting it restored which is still not always the outcome.

So far, the US has resisted site blocking at the ISP level (something the MPA/RIAA has managed to force on ISPs in other countries) but they keep asking for that capability here as well.

Oversight doesn't come in the form of aggressive takedown demands or automated systems blindly taking down content whenever a DMCA notice gets fired off. It comes from careful evaluation of takedown notices before content is removed or sites are taken offline.

Just recently, lack of oversight caused youtube to take down people's videos after youtube got DMCA notices that were not legitimate and youtube's polices made it very hard for real copyright owner to get the issue resolved. (https://torrentfreak.com/bungie-files-lawsuit-to-punish-send...)

> This is not related to copyright law. DRM is a purely technical means, not a legal one.

If DRM was a purely technical problem, we'd be free to use purely technical means to get around it and you'd be right except that it's illegal to bypass DRM or to post tools that would allow that to happen. The law which makes it illegal is the DMCA. When youtube-dl was forced offline the action was defended under that very same __copyright__ law. Their argument was that the tool circumvented youtube's DRM. It doesn't, but that didn't stop them from trying to make that case.

> No, it's not your culture or your history, you're not the author. What everyone misses with these comments is that some authors many not want their works to be preserved or archived.

Yes. It is. It's our culture. We all share it. If an artist wants full control over their work they should keep it to themselves. Once they release it to public it becomes a shared experience that belongs to everyone involved. Do you really believe that Superman isn't a part of our culture? That Mario Bros. isn't a part of our culture?

Copyright law was put in place to give special rights to creators for a limited time to encourage new works. Not because naturally they have the right to keep us from our culture. They don't. They needed special rights to do it. The deal was we'd accept that for the sake of new works, we would temporarily give up our rights, and in exchange once that temporary period of special rights ended, works would enter the public domain. They broke that deal with perpetual copyrights and DRM. Copyright law is being abused to hinder the creation of new works. It's time to change the system.

> There's no coherent issue anywhere else in your reply and there's nothing to pay attention to.

I can't force you to understand, but I'm willing to help you as long as you're willing to listen.

There are plenty of misunderstandings of copyright law from its detractors but this comment seems to be in places the other side of the coin (plenty of the DMCA's detractors misunderstand it but it's not as common to see a hardline supporter of copyright like yourself who doesn't seem to have heard of the DMCA) and in places failing to recognize that you're engaging in a discussion of whether the law as it exists is good or bad: claiming that our laws cause collateral damage to the public isn't always a "misunderstanding" of the law.

>This is not related to copyright law at all. I agree ISP monopolies are a problem, for any number of reasons.

ISP monopolies aren't due to copyright law but these disconnections are a consequence of copyright law: it is the only reliable way you as a third party can get an ISP to disconnect someone based on unsubstantiated allegations. Send an ISP letters claiming that a subscriber is attacking your systems in violation of the CFAA and they'll likely ignore you, maybe they'll ask you to prove it or better yet go after the person doing the hacking. Send an ISP letters claiming that a subscriber is file sharing in violation of the Copyright Act, and if they don't meet your demands (which far exceed anything prescribed by the DMCA) they're liable for contributory and vicarious copyright infringement [1].

>Anyone can sue you anytime for any reason, that doesn't mean they'll win the lawsuit. You could make a good argument here about how copyright law is too long, but that point is lost in the rest of the sentence.

They can, but the relevant laws and legal precedent decide whether the plaintiff can get a decent lawyer to take their case and whether the suit will survive your first attempt to get it dismissed. The doctrines we have around copyright make it easy to bring claims against infringers and get a court to hear it out. But unintentionally they allow for abuses as "small time" as internet subscribers to be dragged into multiyear lawsuits for file sharing when it later emerges that the plaintiff has - literally - no evidence [2] or as "big time" as Katy Perry getting baselessly sued for using a fairly generic sequence of chords and needing an appeals court to overturn the jury verdict against her [3]. In both of these examples, legislation or new legal precedent could raise the standards for bringing certain types of cases without hurting righteous plaintiffs. On the other hand, in both examples the righteous party eventually won. But in others they lose, or pay a nuisance settlement because the court fight isn't worth it [4]. Frivolous lawsuits are a big problem and aren't at all limited to copyright, but clearly there are features of copyright law that make it attractive for people looking to abuse the legal system.

>No, that doesn't make any sense. This is a blatant misunderstanding of copyright that I see so often. Copyright doesn't stop you from creating new projects. You don't need to get the rights if it's a fully original work. You could also make a good argument here about how licensing is too difficult to obtain sometimes but that point is also lost.

I agree that there's a very common misconception here, but there are two real issues with copyright law that are also being referenced. Copyright law makes it unnecessarily difficult to create derivative works based on very old works that would have passed into the public domain if not for relentless lobbying for longer corporate copyright terms. It might also protect "abandoned" works and works where the corporate copyright holder is defunct too aggressively, there's a good argument for both sides of that issue. But it's an argument to be had over how the law can avoid causing real problems, not a "misunderstanding" of the law.

>This never happened. Youtube-dl is still up, if I recall for the short period it was taken down there were like 20 forks that popped up immediately.

The article this comment thread is under is about precisely the issue of copyright lawsuit threats being used to hide youtube-dl from search results.

>This also never happened and is contradictory to everything else you said, your complaint actually seems to be that there's too much oversight.

The "censorship without oversight" refers to the DMCA notice and takedown procedure. Outside of extremely rare court injunctions, there is no authority that can monitor or stop flagrant abusers of a system whose purpose is to legally compel the removal of content from online services, which might be called a form of censorship.

>This is not related to copyright law. DRM is a purely technical means, not a legal one.

Again, this is "first paragraph of the Wikipedia article" level DMCA. If DRM circumvention were not illegal, it would be purely a technical means, but courts have repeatedly ruled that DRM doesn't need to be any good technically speaking to have legal force.

>No, it's not your culture or your history, you're not the author. What everyone misses with these comments is that some authors many not want their works to be preserved or archived.

Those authors need something else because stopping annoying archivists is not in the scope of copyright law. The purpose of copyright is to give authors a limited monopoly on the making and selling of copies of their work. And before the second half of the 20th century, we had copyright law with the understanding that creative works outside of IP law were part of the "public property", "common property", or the newer term "public domain". Now, of course, there are ways of protecting works more effectively than copyright alone by never selling individuals copies of works, only licensing them, and using legally protected DRM to discourage breaking the license terms.

[1] https://variety.com/2019/music/news/cox-communications-music...

[2] https://www.techdirt.com/2021/12/16/malibu-media-ordered-to-...

[3] https://variety.com/2022/music/news/katy-perry-appeal-dark-h...

[4] https://twitter.com/edsheeran/status/1511631955238047751

It's more nuanced than hand-waving away anyone that wants to talk about copyright reform with neo-capitalist 'everyone will choose to be a freeloader' hyperbole.

"If you could feed everyone on earth at the cost of baking one loaf and pressing a button, what would be the moral case for charging more for bread than some people could afford to pay? This represents the difficulty at which we find ourselves straining at the opening of the twenty-first century."

--- Eben Moglen, speech he gave in 2003 - refer: http://moglen.law.columbia.edu/publications/maine-speech.htm...

The natural state is that yes, you can share everything since, for things that are intended to be experienced by humans, there are no technical restrictions that do not succumb to the analog hole. So yes, not being able to consume everything for free is collateral damage from copyright and thus to justify copyright you need to demonstrate that their is more gained in return. This is why initial copyright terms were much shorter than they are now - because the deal has always been to encourage the production of more content that then becomes part of the commons, i.e. eventually can be consumed by everyone for free.

> You have to play ball with them or they will simply not be able to produce those things anymore.

They should stop, and we can freely share what's left. Their bullshit crowds out everything else anyway.

What is "everything else" in this context? You're saying that every writer should stop writing books, every artist should stop making art, every musician should stop making music... That is everything as far as our information society goes, there won't be any "everything else" after that.

Do you need me to go around looking for all the declarations from different big name musicians saying how they only really make money on touring, not on the record selling?

How many examples of companies and people working on free software do you need? Or authors that started out writing as a side-gig?

It's simply not true that copyright reform would lead to death of creative works. Most likely the opposite: we would certainly see a reduction in the quantity of works, but those remaining would be working out of pure interest, which would lead me to believe that the average quality would be substantially better.

I'm a musician and I write software, you don't need to preach to me about this. Record sales are indeed dead but there is no chance they'll come back at all without copyright or something like it. Also that doesn't invalidate copyright anyway. Copyright is also an important part of performance, for example the laws about what happens and what royalties you get if somebody else covers your song and profits from it.

Free and open source software is completely dependent on copyright to work, so that's a terrible example.

You can advocate for copyright reform all you want but you misunderstand who you have to please when you do that. It's all those lobbying groups. There's no other groups who would be invested in reforming copyright law. The "quality" you're advocating for here is completely subjective and not up to you or me to decide, even moreso when you try to dismiss all these things that millions of people actually like as "cultural garbage".

> You can advocate for copyright reform all you want but you misunderstand who you have to please when you do that. It's all those lobbying groups.

Or we can simply acknowledge that the whole system is rigged and continue to ignore the laws and get as many people to understand that piracy is the only reasonably moral alternative?

> You have to play ball with them or they will simply not be able to produce those things anymore.

You say that like it's a bad thing. I wish it was that easy to get them to stop producing all that crap, but it seems that the easier it gets to pirate things, the more cultural garbage they produce.

To be clear here, you're saying that you want all writers, researchers, artists, architects, musicians, composers, filmmakers, actors, podcasters, to just stop doing things? Would it make you happy if I just threw out my painting supplies and my clarinet?

If you only paint or play the clarinet to get a paycheck from Disney, yes.

So now we're going back to the central point. Before it sounded like you wanted me to "stop producing". Now from this reply, it sounds like you don't actually care if I stop producing, you're just objecting to me getting paid and demanding payment.

I'd be objecting anyone that gets paid to produce cultural pollution.

Are you getting paid to produce cultural pollution?

Please discontinue this style of rhetoric, it's not what this site is for.

The "people will starve without copyright" rhetoric is also quite off-putting.

That's not a rhetoric and it's also not even close to what I said or what I actually think. Please don't make things up, it's doing a disservice to yourself.

What you are implying is that people only create something if they can "demand payment" for it, no?

Nearly all of classical music was sponsored by and produced for wealthy patrons. Is that cultural pollution too?

The body of work that reached the present day is the refined version of that. All the duds were forgotten.

I'm sure back in the day, Salzburg people were also rolling their eye about yet another Requiem.

Jumping in on your back and forth...

I believe there is a kind of artists dilemma where, one way to put it, we differentiate "commercial art" from "fine art". We put a higher value on the starving artist whose passion for creativity is so consuming, and vision so singular that they will follow their muse irregardless of any financial success.

I exaggerate to make clear the point.

Put another way, if there were no more blockbuster films, indie films would nonetheless still get made. Some people (maybe myself?) would be okay with that.

> Writers, researchers, artists, architects, musicians, composers, filmmakers, actors, podcasters, even a lot of software developers

Plenty of those same people are fed up with copyright law in the US too. There are people releasing their works under more permissive licenses, publishing directly to torrent sites, etc. The fact is that most of the time copyright enforcement doesn't do a damn thing for the creators anyway. They rarely keep the copyright of what they produce. It does however often hinder their ability to work on things they'd like to, and it opens them up to legal risks from others who might claim their works are too close to their own. There's a very real chilling effect where creators may never even publish works out of fear that they'll be dragged into court against a team of industry lawyers. Our excessive copyright and patent system actually hinders more than it encourages new works.

Researchers? Seriously? What IP do researchers have?

Copyright reform is not “defunding copyright.” It’s a reform. It wouldn’t hurt individuals, unless they are super successful, and even then, not that much. And it helps everyone have higher quality, cheaper content and political freedom.

The problem is simply that the media is once again suppressing the lower/middle class for the one percent, and spreading FUD.

I mean even Google did not go this far though. So it sounds like DDG is doing more than the strict minimum required by law.

Google also has more resources to fend of lawsuits as well as existing deals with many big copyright holders. The letter of the law is not the only thing that matters, chilling effects are real.

As for the YouTube-DL part specifically, I'm pretty sure that's a side-effect of them being dependent upon Bing for the actual index: https://www.bing.com/search?q=site%3Ayoutube-dl.org

Also, _come on_; if someone gives up after one search term, which also includes advanced site-restricting syntax, there's no way they'd be able to operate youtube-dl anyway: https://duckduckgo.com/?q=youtube-dl&ia=web

Yeah idk why someone would type in the full URL of a site and not just the term they're looking for in a search engine...what a dumb way to test that on their part.

The point isn't that you can't reach the website by typing in the search term, the point is that they've deindexed the site. If you search for youtube-dl, neither youtube-dl.org nor yt-dl.org come up in the results, as far as I can see.

You can still access it from the results via other sites, like github and wikipedia (which even makes the real website pop up in the info square, funnily enough), but the search results themselves do not contain any links to the main website.

What if I want to search what sites have mentioned those urls?

That's still the wrong syntax.

Absolutely! Why on earth would you test a dumb way of doing things on a user-facing service?! slams hand on desk Irrelevant!

The article doesn't explain it well, but if you search for just `youtube-dl` (without the "site" prefix), the main youtube-dl.org site is still not present in the search results.

They don't rely solely on Bing. If a popular site doesn't appear, it's definitely a conscious decision on their part.

... or the domain list comes from Bing and they additionally feed some of that to their own crawler or...

This seems like an oversimplification of things you don't know.

DuckDuckGo is not solely dependent on Bing. They use Bing as one sourc of the, but they've run their own crawlers since at least 2012.

Not for actual search IIRC. Last time I checked, they only used their own index and crawlers for the "smart search" and instant answers. Almost all of their search results are still sourced from Bing.

That's pretty easy to check for yourself, usually whenever something does not show up on Bing it won't on DDG either. A good example of that was when the "tank man" picture disappeared from Bing and the exact same thing happened on DDG.

Yandex was also previously used for search, largely of Russian sites.

That relationship has reportedly been "paused":

https://www.protocol.com/bulletins/duckduckgo-yandex-ukraine

I use DDG as my default search engine, along with NoScript in the browser. Often when I visit a new website, I peruse the (long) list of domains that the site is trying to pull javascripts from.

I keep most of those source sites in UNTRUSTED status (including some of the big names in search/ads/etc). But I've always had DDG in the TRUSTED category because I had only seen its javascript before on the main DDG website.

(Unfortunately NoScript has a limitation that you can't tell it to "only TRUST javascript from example.com when I'm visiting example.com").

But recently I started noticing some websites pulling javascript from DDG (I don't remember which sites).

So now I was wondering if DDG is getting into the tracking business, since they're now having their javascripts load from third party sites.

Obviously this is anecdotal. But does anyone know if they are indeed beginning to track?

Can you give us a list (or partial list) of sites that are pulling scripts from duckduckgo? We can look at what they're trying to do.

I don't remember the sites; I'll try to find them again, and will share here if I do.

I remember seeing 3 sites within an hour, and deciding to change the DDG setting to TEMP:TRUSTED afterwards.

The nice thing about DDG is that if you're willing to sacrifice some functionality it can still be used without JS at all (which is how I use it)

> Unfortunately NoScript has a limitation that you can't tell it to "only TRUST javascript from example.com when I'm visiting example.com"

I was under the impression that the custom option allowed this. Am I misunderstanding the point of this option?

I see "Custom" allowing you to choose which elements (frame, fonts, etc.) to allow/block for the domain you're configuring.

But it doesn't offer the ability to say "apply these settings to the domain example.com only when I'm visiting example.com, and not when I'm visiting anotherexample.com which happens to load JS from example.com".

Maybe I'm still misunderstanding, but when configuring the domain in the custom settings, it does allow you to limit the custom rules to only the site you're currently on, via the "Enable these capabilities when top page matches" dropdown. The default is "ANY SITE".

I don't see any of these options in my plugin. I have NoScript 10, and it looks like there is a NoScript 11 out there; is that what you have? Maybe the feature was introduced in 11 and I'm missing that update; checking their changelog now...

Edit: Correction - I do have NoScript 11; but don't see those options.

I think you nailed it - I am indeed on 11. So good news! It looks like NoScript is attentive to user needs.

Edit: seeing your edit - the plot thickens. I'm on 11.4.4 - any difference there?

I had 11.2.11.

And you're right, according to their changelog [1], they added it in 11.3.

>> v 11.3rc1 + Contextual policies (different capabilities for the same origin, depending on the top-level domain) configurable in the CUSTOM panel (thanks NLnet for financial support)

Woohoo!

Thanks for following up and making me look, I now have a better setup!

[1] https://noscript.net/changelog/

Glad I could help! I would hate for someone to miss out on what has so far been my favorite feature of NoScript.

I run uMatrix and have noticed some DDG showing up on other sites as well. The sites in question appeared to be (at least ostensibly) using it as a "can I reach the internet" sort of check. If I blocked requests, it would say something to the effect of "no connection detected." I wish I could remember which sites they were, but I do remember seeing at least one call to improving.duckduckgo.com from a 3rd party.

Ok interesting. I'll try to find those sites I encountered again and check it out with uMatrix.

Isn't umatrix dead/unsupported since like a year or so ago?

Unsupported, yes but so far there is no adequate replacement and it remains the best we have.

> (Unfortunately NoScript has a limitation that you can't tell it to "only TRUST javascript from example.com when I'm visiting example.com").

uMatrix (which I'm using in desktop Firefox) works exactly like this. Plus it allows you to forbid/allow cookies, styles, images, scripts, media, XHR, and iframes separately (for each origin/domain).

Ok I'll set it up sometime soon and give it a try. Thanks.

It's officially deprecated, but it still works.

https://github.com/gorhill/uMatrix

Isn’t that non-origin?

The origin/non-origin difference only applies to uBlock Origin, not uMatrix.

Ahh, good to know. Thanks.

Any alternative to uMatrix?

uBlock Origin in advanced mode also supports this (although only scripts/frames/images, not the full uMatrix list).

uMatrix really is the best dead extension. I still hope someone will revive or recreate it (or add the same interface to uBlock or one that is just as good).

> So now I was wondering if DDG is getting into the tracking business

Anecdotal of course, but I've been seeing more and more DDG billboards. Those things aren't cheap, and my trust in them has declined the more I see them advertise in the traditional market.

So where does one from here for everyday search? Google is out. Bing has many of the same problems as Google. Startpage blocks my VPN. Brave has always felt just a little "off" to me, but maybe they're worth a try. Any others I've missed that are worth looking into?

I found Kagi[0] from somewhere on HN -- they make pretty strong privacy claims, and are in a closed Beta stage right now (you can give them your email, and they'll send you a signup link within a week or two). They're planning to charge a fixed rate for their search engine once they're out of beta later this year.

So far, it seems to be working really well for me! Results are pretty excellent, and they support the DDG bang queries (like `!g`) if you ever need it

[0] https://kagi.com/

I second this. I use this full time now. A helpful HN user told me about hyperweb for iOS which I use to make Kagi my fulltime search engine on iOS. I have been VERY happy

> they'll send you a signup link within a week or two

Is it though? I think I've been on the wait list for a few months now.

How do we know these privacy claims are true? What if Kagi was Chinese or Russian, would you still trust it and why? And how do we know Kagi doesn't end up the same way as DDG?

I trust Yandex more than I trust Google, because Yandex doesn't give a shit about me. I'd certainly use an English-language Chinese search engine.

(Kagi dev here)

Kagi is privately owned US corporation.

> And how do we know Kagi doesn't end up the same way as DDG?

Being a paid service means Kagi's incentives are very different. Instead of selling your data or profiting from feeding into ad-tech food chain, we are interested in selling you a subscription. This changes everything as the number one thing in Kagi's universe becomes what the users want, different to DDG, Google and other ad-supported search engines.

> How do we know these privacy claims are true?

Sadly I can not think of a way to verify them (let me know if you have one). But most plainly, we have no incentive or reason to do otherwise. Note that the only private information Kagi asks of the user is an email address, need to create an account, and those concerned can use whatever email they want.

Thanks for the explanation.

> But most plainly, we have no incentive or reason to do otherwise.

One reason is: to improve your services. Other reasons could appear over time.

> Sadly I can not think of a way to verify them (let me know if you have one).

The privacy claims could be verified by recurring external audits.

> The privacy claims could be verified by recurring external audits.

Not in reality. Nothing prevents a bad player from misusing user data once an audit is done. External audit would only be a good cover.

One way for Kagi to have more incentives for privacy of their users would be moving to EU where there laws much more stricter.

$10 per month (last heard), if they plan to start charing.

Set up searx and aggregate results from the ones you want (e.g. startpage you can route through a different proxy or vpn). Lets you clean out the crap and rewrite redirect urls to the original ones, etc.

Yandex is pretty good for image searches.

I'm biased so here's an independent take on your options: https://seirdy.one/2021/03/10/search-engines-with-own-indexe...

If they stick with billboards for advertising, I personally don't mind it. The issue to me is with tracking-based advertising(/anything).

I actually noticed that the quality of the search dropped (from pretty much parity with google sans-cookies), then noticed the new billboards and radio ads afterwards. I've seen tv commercials since.

Suck Suck Blow has many redeeming features. One that's GOLD imho;

duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/

Running a hidden service is just so jolly gentlemanly. And it works in the total absence of JavaScript and no matter what utter lies I tell it about my randomised-per-request UA, and cookie black holes. The obvious dark side is that it's closely connected to Amazon.

ublock doesn't correctly show beacons and pings

beacons and pings fired upon activating a link, happen after the document change, so ublock associates them with the new document, even though they are initiated by the old document

We don't use third-party scripts on our site and I don't know of any sites using our scripts either.

I suspect this is largely due to DDG using Bing under the hood, which has led to similar weirdness in the past, eg all major porn sites disappearing from the results in Singapore (while Google still showed them, mind you!).

Still super disappointing though, and yet another reason why trying to build a better search engine on top of someone else's tech is a non-starter.

Having worked on Google's indexing system, I can't imagine how much it would cost to write all of the crawling and indexing code from scratch, and then run it on over the visible web. You need to bootstrap somehow form an existing index if you want to get anywhere in any reasonable time.

If there's a market for alternative search engines out there, AWS is already serving so much of the web that Amazon should really provide crawling and basic indexing (for a fee, of course) so it's done once for everyone, with Amazon Lambda processing to allow search engines to customize their indexing. I'm not sure if it would make them much money, but it makes more sense to start up a search engine on AWS using Amazon's crawl and basic indexing vs. using a search engine competitor's crawl and indexing.

That's an insanely good idea. Since behind the scenes, Amazon would know who owns each website that it's crawling, an AWS search database would inherently be better-curated and more trustworthy than anything an external crawler could put together. Having your website on AWS would be crucial for SEO.

Awful for competition and whatnot.

Google and Bing can already crawl AWS sites just fine. As long as Amazon doesn't provide any first-party search engine, just providing search engine infrastructure, I think it would be a net win for competition.

As it stands, if you want to bootstrap your own search engine, you need to base your search engine on either Google or Bing's index, or perform the herculean task of making your own crawler+indexer+search engine. If Amazon can commoditize the back end, and let AWS-hosted search engines provide differentiators late in the indexing pipeline and on the search/serving side, I think we'd see more niche search engines spring up.

You know what? Not that crazy of an idea.

Amazon is a weirdly plausible competitor to Google. They have a surprisingly robust ad business, and obviously AWS.

https://www.cnbc.com/2022/02/03/amazon-has-a-31-billion-a-ye...

Amazon ran A9.com as a search engine from 2004 to 2008. https://en.wikipedia.org/wiki/A9.com#A9.com_search_portal

Yes, A9 was one of the competitors for some of my time at Google, but I'm talking about Amazon basically creating some search engine infrastructure and a framework for building search engines on top of. Selling shovels to the gold miners, in a sense.

The crawling and initial indexing stages need a lot of scaling and performance tuning work, which is a core competency for the AWS engineers. Furthermore, essentially zero people say "I want to build a startup around my new crawl scheduling idea, but it's too time consuming and expensive to write the rest of the search engine." Many more people say "I have a new idea for ranking" or "I have a new idea for content extraction, but it's too expensive and time-consuming to write the rest of a search engine".

Google and Bing already expose "custom search engines" which are basically just filters on top of their search engines. I'm talking about something similar, but much deeper customization.

Also, I was at Goldman when Google Search Appliance was EoL'd. GS had plenty of internal content that needed to be indexed, absolutely needed to be kept private, and wasn't well indexed by any of the replacements GS evaluated. For companies that keep lots of documents in AWS storage, it would be good to open up the space for Google Search Appliance replacements. Amazon could write their own, but given that Google abandoned the space, the profit margins aren't great for a huge company. However, creating a search ecosystem within the AWS ecosystem makes the AWS ecosystem more attractive. Maybe forgoing the search profits and inviting in search providers would be a net win in selling more AWS compute and storage.

Unfortunately I suspect this would be selling gold shovels to dirt miners and the economics of it are not compelling.

I don't see how it's a non-starter. DDG is still a better search engine than bing, if for no other reason than bangs. If a censored search engine is the non-starter then every well-known search engine is a non-starter.

Maybe that's the real problem. There have been some recent articles on HN about how "search is broken" and maybe this article falls under that. Because what really gets me is that there is plenty of legit content on "pirate sites" and blocking them completely cuts off that content.

Someone should make a search engine that only indexes sites that Google and DuckDuckgo do not index. It would serve sort of the same purpose as like, lists of banned books.

It will be a lot of spam with a bit of useful sites sprinkled randomly.

You want a list of spam sites, phishing, malware, and SEO-driven gibberish with back-links?

Yeah, I guess that would probably be useful for something. Academic research into just how awful people are, perhaps. And, of course, as a blackhole list.

Just make sure you get a bulk discount on disk space. Such a list would be huge.

To be fair, “SEO-driven gibberish with back-links” is an apt description of Google’s actual results these days.

"You want a list of spam sites, phishing, malware, and SEO-driven gibberish with back-links?"

Yes, exactly, I want exactly whatever I search for, including that if that's what I search for.

If I search for something which exists, then I want the knowledge of that existence. The value judgement is no one else's business unless I search for "X but only where approved by any governments or companies with any political or financial interest in controlling what I know."

When it turns out that I would never choose to search for that, that is the proof that it's wrong to be doing it.

You don't really get the knowledge of its existence if it's one of thirty million hits. A search engine that includes everything is a kind of Library of Babel. The info is there but it doesn't help you.

I am the only one who may say that.

No-one wants to see 99.99999% of the spam sites that gets purged in Google's filter. Look at all the complaints of the 0.00001% that gets through.

I want whatever I say I want.

There is no reason for legitimate filtering not to be voluntary, like the default porn filter that you have the option to disable.

I still have access to my spam folder, and unsolicited communication is not the same as search results. If I search for something, then I searched for it. I am THE ONLY person who may judge the value of the results.

The search engine's job is only to tell me what's there, not to tell me what I want.

Most spam emails do not even reach your spam folder. You wouldn't want to see those purged emails either.

You can't say that for anyone else. No one can say that for anyone else.

And, repeating, unsolicited communication is different from search results.

But in fact yes I do wish I could trust that my spam folder was actually complete. I find false positives in there all the time. But my point was that even if it's incomplete, you still have a spam folder that you can choose whether to look at or not. The fact that some stuff is filtered by other parties before reaching my client or server is irrelevant.

I do want to see those emails. Which is why I run my own mail server where I control what does and does not get rejected. BTW Google and Microsoft do reject or worse, silently drop tons of legitimate mail for such great reasons as the sending server having a bad actor living on the block some time in the past. Mail filtering is a really bad argument why you wouldn't want to see what is hidden from you by the incumbent providers.

Running my own crawler is unfortunately not as realistic.

And you are welcome to it. But don't be surprised that few other people also desire that service, or that no companies are rushing to provide it for you.

Banned books are too dangerous for being find publicly, and list of banned books is too useless thing. Better to create a hidden torrent tracker for things nobody else wants to host.

I appreciate the sentiment but such an index would be 99.9% spam, malware, and phishing.

Not to mention some incredibly horrible NSFL shit

Not to mention, Game of Thrones spoilers

In the eyes of some of those in power, youtube-dl is "malware".

How do you know or are you just assuming that?

This is the same line of thinking as "someone should set up an alternate social network where users don't get banned". Except that 99.9% of bans at Facebook, Twitter and the like are completely justified, and so that's what the dominant content on your new platform will be.

> Except that 99.9% of bans at Facebook, Twitter and the like are completely justified

Gonna need a big citation on that. One of my Facebook accounts got banned because I accidentally left my VPN on. Whilst I take responsibility for the terrible opsec, the fact is people get banned all the time for doing things that are anti-Facebook rather than anti-social (anti each other). I would definitely use a social network that doesn’t feel like a police state.

Well for starters 99.9% of Facebook users don’t use a VPN.

We aren't talking of 99.9% of Facebook users, we are talking about 99.9% of banned Facebook users - and a VPN is just one thing that can get a non-malicious user banned.

Funny how the narrative on DDG has changed. I used to get downvoted to oblivion for merely mentioning they used Bing under the hood. Where are all the people that used to defend them so vigorously? I wouldn't be surprised if it's the same people now hating on them.

> I wouldn't be surprised if it's the same people now hating on them.

It seems like those people are still here defending ddg. They are even attacking the blog. Weird.

I use ddg as my default search and hope they are "privacy-first" as they claim ( thought I have my doubts ). At this point what other option is there? DDG search is noticably censored since they rely on bing. It's obvious to anyone who uses ddg regularly. Not sure why people here are making excuses for ddg or pretending otherwise. What I would give to get the google of old.

It's sad looking back on what google used to be. All the freedom and optimism of the 2000s is definitely gone.

Searx. Use Searx. Does your search engine of choice fail to get results that the others pick up? Searx grabs all of them and compiles them together. Don't miss another search result!

Also, proxying increases privacy. Also, it's open source and self-hostable.

Being open source doesn't mean the people running the public searx instances aren't tracking you.

Self-hosting makes you lose all the benefits of proxying.

In the end be it ddg or searx you can only rely on some third-party trust.

Yup, you choose one or the other. As with all things, there's you can't have it all. There's a balance.

Still, it's a valid criticism. How is anyone supposed to know which of the numerous searx instances has what they want. Seems this would really benefit from an instance run by a transparent non-profit organization. You still need to trust someone, you always do, but that doesn't mean that that trust can't be earned rather than just assumed.

yandex

I'm trying startpage right now. Looking for alternatives to ddg, liking it so far

TPB 1337x and yt dl aren't filtered for me?

Serious Question: How is the DDG search structured? Is it a cosmetic skin over Bing, or is it aggregating from other sites like Yahoo, ecosia etc additionally?

If it is just Bing under the hood, how does it exist as business entity. I am sure MS will take some action to consolidate their search share rather than seeing splintered.

They aggregate from "over 400 sources" and run their own crawler: https://help.duckduckgo.com/duckduckgo-help-pages/results/so...

Do they list their "sources"?

It reads like they are saying a website = source, and the hyperlink for the sources is dead.

That's just their corporate propaganda. For the most part, it's Bing with some additional features.

Their crawler (DuckDuckBot) doesn't have much of an impact on the search results, it's mainly used to provide instant answers. [1]

[1]: https://seirdy.one/2021/03/10/search-engines-with-own-indexe...

To be fair the instant answers are usually quite good!

> If it is just Bing under the hood, how does it exist as business entity.

They provide an alternative branding, targeted at "privacy-aware" users and hipsters.

DDG is Bing, they use its API to get the search results. They augment it with other sources to provide the "value added" part, but that's a tiny part. DDG doesn't want you to know that it is Bing, but Bing is what it is.

DDG does run ads just like Bing and Google, so it's just a way for Bing to get more search ad inventory out there.

Once upon a time there were other sites that did the same thing with Google, but eventually Google decided they didn't need third parties to drive search traffic.

HN constantly mentions DDG is really just Bing in disguise and they're essentially the same. However, that can't be entirely true because they produce different results for the same search term.

https://duckduckgo.com/?q=obscure+search+term&ia=web

https://www4.bing.com/search?q=obscure+search+term

That being said, when Bing censored "tank man", DDG's image search also produced 0 results for "tank man".

It is Bing, but they seem to use different ranking and filtering on the index.vIf you make complex queries it is quite obvious. Since they have zero transparency the details of their system and deal with bing is unclear.

Did you try in incognito? Your search gave me nearly identical results.

Very different ones here.

Between geotargeting and A/B testing and possible differences between the Bing API and public website you really can't conclude that DDG is not just what they get from Bing based on one example.

I don’t condone the actions of the Russian government; however:

If you want better results from pirate sites, try Yandex.com. Quite good for finding torrents.

>Quite good for finding torrents.

That's because there's no DMCA law there and they're not taking down thousands upon thousands of pages like Google is forced to do every day. [1]

[1] https://www.lumendatabase.org/

Yandex has an agreement with copyright holders and removes sites with pirated content from results. But those are Russian copyright holders and they mostly care about content in Russian.

Also Yandex removes sites that are banned in Russia, for example BBC in Russian or Navalny's site.

I used to be able to find MP3s with Google, until one day it simply stopped working. Then I moved to Baidu, which worked for a while. I now use DDG to find movies, and yet looks like the curtain may be closing on that one too.

I fear it might only be a matter of time before Yandex stops working too.

there are telegram bots for it though, personally I don't use them but people swear on them

btw Baidu is extremely slow due to GFW

true that. I tried to look something up on Baidu once (from Germany) and gave up after it loaded for a full minute

Russian government has always appreciated https://en.wikipedia.org/wiki/Kompromat so be careful what you search for.

I love how we have to preface our approval of something foreign with "now I'm not a Nazi, but...'

The obvious reason I used that disclaimer was the fact that Russia’s government decided to invade Ukraine.

why would you look for torrents in search engine? you will anyway find most of them dead

you are much better off with Knaben on EU domain, same results can be achieved with add-ons in qbit, then there are special software not worth my time (Jackett)

but I agree I find myself using yandex much more nowadays as alternative

Yandex made it possible for me to find extremely rare movies that are not to be found elsewhere. So I have found it very useful.

What is with this blogspam?

YouTube-DL comes up just fine in search results (https://duckduckgo.com/?q=youtube-dl&t=h_&ia=web).

So does Pirate Bay (https://duckduckgo.com/?q=pirate+bay&t=h_&ia=web).

The headline is outright false.

the complaint in the article is around the serach engine's "search within site" feature.

Compare the following two:

- https://duckduckgo.com/?q=site%3Athepiratebay.org&t=h_&ia=we...

- https://duckduckgo.com/?q=site%3Aycombinator.com&t=h_&ia=web

The YouTube-DL website notably does not come up in those search results, it's their GitHub page that does. The website does appear in the sidebar, because it pulls the sidebar data from Wikipedia, and YouTube-DL's Wikipedia page does come up.

Note that if you do a google search (https://www.google.com/search?q=youtube-dl) their website is the first result.

Both the website and the GitHub are top two results at my end, in India.

Interesting, mine are github, videohelp.com, and then ottverse. I imagine that adds credence to the theory that it's because of Bing's index removing them - maybe they use a different index in India?

Their main website is not coming up here (sweden). Only the github and some random fringe-sites, obvious scams and readthedocs.

Not anymore. I'm also using it in India, the original website is no longer listed. Top result is the repo

Still same here. It even has a dedicated snippet.

https://ibb.co/tZJcB1M

Same here in Germany

Not sure what you mean. Someone on urban dictionary defines blogspam as

> A blog where the author paraphrases or copies from the original article/webpage in an attempt to increase his or her own traffic.

Another person as

> loaded with so much foreign material, it refuses to function properly

Since neither of those fit, I guess you mean it more like a random blog that thw owners 'spam' everywhere such as onto link collectors like HN?

Torrentfreak is quite the opposite of that, if that's what you were accusing them of in the first place. One could say that your comment's "headline is outright false"... https://en.wikipedia.org/wiki/TorrentFreak

I'm just waiting for the day they announce an NFT or a "trusted partners" program with establishment media entities. Come on, DDG, you've come this far, so truly jump the shark for our amusement.

Which would be ironic since Nfts cannot be censored (they exist on a decentralized censorship resistant database)

I don’t need these results but I don’t want to support this behaviour/trend.

Can anyone recommend a decent non-Google alternative.

https://search.brave.com/

https://presearch.org

Cryptocurrency? No, thanks.

I've been using Kagi (https://kagi.com) for a few months and it's been great. It's free while in beta but will be funded by subscription fees afterward.

The killer feature for me is the ability to up-rank and down-rank sites in my personal results, which has been really helpful for curating my tech-related searches. It allows me to quickly find high-quality docs and eliminate the garbage.

I’ve been using Kagi as my primary browser for a while. My only complaint about so far has nothing to do with Kagi itself, I just wish I could set it as my primary search engine in Safari.

You can, using (unofficial) Kagi for Safari extension.

https://github.com/marcocebrian/kagisearchsafari

I've set it as my primary search engine in Firefox, it looks like there are instructions here about how to do the same in safari: https://kagi.com/faq#default

sounds like an AD

kagi? are you willing to repeat the same mistakes?

promoting unknown proprietary software, as if it was Moses, yet again

very suspicious, is kagi made by the people who operate DDG? or it's totally "trustable" set of people?

> The killer feature for me is the ability to up-rank and down-rank sites in my personal results

how does that work? is that feature tracked by kagi? are they using it to train a ML model?

Self-hosted SearXNG [0]

Have your own search aggregator.

[0] https://github.com/searxng/searxng

Check out https://you.com - lets you control your sources and is privacy focussed

Your choices are limited, and very few independent indexes. Most options are Bing alternatives: https://seirdy.one/2021/03/10/search-engines-with-own-indexe...

That was fascinating, thanks so much for sharing it

And here goes my reason to use DDG. Not because I pirate stuff, but because I hate censorship. Its only a matter of time till they implement the same filters as GGle.

And in no way those filters stop dedicated ppl.

DDG sucks in many ways. Besides the engine performing quite poorly, it also relies on third parties and so will return filtered results they may not even control. They also never supported IPv6 and are hosted at Microsoft or Amazon.

IMO, there is no credible search engine today.

> are hosted at Microsoft or Amazon

I agree with most of what you are saying but this is silly. I get "Microsoft and Amazon bad" but basically saying "host your own infrastructure worldwide and index your own corpus or you are a bad actor is rather inane.

They may not host their own infrastructure, even though I think something like DDG should. But they can choose a better network, better technically and ethically.

https://duckduckgo.com/traffic

Have a look at a 50 day average and you can see their mistakes come to light. I've shifted off them recently.

I used to be DDG for everything, then it became anything non-work related, then on phone with FF focus, now nothing.

VPN and clearing cookies after browser close except for a few certain sites has replaced them.

What is an uncensored search engine then?

If I search for something, and the search engine does not tell me about things which ARE there, then that is a defective search engine.

It's fine to keep kids from getting ahold of any sharp objects but if I need a knife I need a knife and it's ridiculous for anyone else to decide to lie to me about the existence of knives.

We are doing the Chinese firewall to ourselves just a bit slower.

Private search aggregation engine

https://github.com/searx/searx

Exactly.

Installed that recently and been using it for a few weeks - it's great to self-host one's own search aggregator and I'm very impressed with the results.

So much for a Google alternative.

Quit using DDG when they decided to start censoring stuff. Been using brave search ever since, it's actually pretty good.

Didn't Brave get caught inserting their own affiliate links into search results when you clicked on them?

I would rank that pretty low on my list of things to worry about a search engine doing. Actually, to me, that would be relatively fair in terms of monetization of a search engine product.

I would turn the feature on if asked by a reputable and well-meaning company. Especially in the case of a search engine seeing as they are providing the link afterall.

Search engines rely on trust. Dark patterns that include shady things like exploitative URL insertions on clicking links are highly relevant.

I'm continually surprised by the amount of attention that HN gives DDG. They have no unique or interesting technology. The "privacy" claims are all self-attestation.

I've started hearing ads for DuckDuckGo on NPR, of all places. Could be completely coincidental, but I find the timing suspicious.

Do you think they're gearing up for an IPO?

I think they've advertised there for a while.

I've seen DuckDuckGo ads on network TV.

What is going on with this company in the past several months? Completely destroying any goodwill they had

I've never understood how links can fall victim to copyright. A link is just a signpost instructing your browser where to go next. There is no copying involved, anywhere. Subjugating links with copyright law is morphing reality into fantasy.

Unfortunately, Bing (and thus DDG) removes proper websites from their index for unknown reasons, without any way to get this sorted out [0]. My blog (10 years, no spam content) has been delisted for 2 months now. The Bing webmaster tools don't display any error. After using DDG for 2-3 years, both as my main search engine and as OpenSearch for my blog, I had to switch back to Google.

[0] https://techcommunity.microsoft.com/t5/microsoft-bing/help-w...

That's out-of-order on two counts. I see there are plenty of others in the same boat. Our thus far smaller but 5+ billion index has you covered: https://www.mojeek.com/search?q=site%3Awww.espositosoftware....

Thanks, but that is not my site. Mine is darekkay.com and it isn't indexed by mojeek.

OK, thanks. Why did you think that? Looks like, at least some, is indexed: https://www.mojeek.com/search?q=darekkay+site%3Adarekkay.com

I've searched just for site:domain (without any specific keyword), which both Google and Bing support.

Thanks for pointing that out. We'll add support for that too.

https://news.ycombinator.com/item?id=31059882 : DDG: We are not “purging” any media outlets from results

>Gabriel Weinberg @yegg Apr 17, 2022 10:18AM UTC Hoping to clear up some misconceptions about our private search engine. ... <

Gabriel Weinberg, @yegg CEO & Founder @DuckDuckGo - Nitter : https://nitter.net/yegg/

The mentioned sites aren't filtered from DDG for me

I really appreciate these reports, because torrentfreak usually includes a list of the sites which were blocked, so I can add them to the list of torrent sites that I visit via unfiltered DNS and VPN proxy.

I had forgotten about YTS until the recent BREIN (.nl) announcement, and I hadn't heard of Fmovies until today.

Circumvention of these measures is so easy that I feel like it's making a haves/have-nots division on the Internet. Those of us with two or more clues are barely inconvenienced, and everyone else is blocked hard.

What about archive.org? Will they delist that too? That definitely contains a lot of copyright-infringing data. Where will this end?

Pirate Bay actually contains a lot of non-infringing items too.

Who watches the watchers?

> Pirate Bay actually contains a lot of non-infringing items too.

Wait, what have they got? Actual Linux ISOs?

And public domain content, and content licensed for free distribution, and abandoned software. The good stuff is there too, but the legit free stuff isn't all bad.

You can literally get Windows keys on archive.org, not that I use Windows anyways

10yo me would have loved this information. Linux FTW

Well, after using DDG exclusively for web search since maybe 2014, I'll have to jump ship. I was never chuffed about them not using their own index, but removing priate sites and yt-dl, and the improving.duckduckgo.com business is exactly the type of behaviour I left Google for DDG over.

I've liked what little of Brave Search I've used, but I'm not in love with Brave as a company either so perhaps I'll audition some of the paid web search options.

It's interesting that Qwant never comes up in these discussions, they have their own engine and are privacy focused, looks very similar to Google but just never seems to be brought up?

Same results on Qwant. No thepiratebay.org and not youtube dl's official site. The results seem very much identical.

Hmm this will affect me, I always type tpb in ddg to go there. I’m also happy my provider is small and freedom loving so it isn’t blocked (neither is RT for that matter).

DuckDuckGo has a cheat sheet specifically for youtube-dl so it seems quite unlikely they are actively trying to suppress content about youtube-dl:

https://duckduckgo.com/?q=youtubedl%20cheat%20sheet&ia=cheat...

There’s a lot of hate in this thread towards a company trying to do the right thing.

> https://duckduckgo.com/?q=youtubedl%20cheat%20sheet&ia=cheat...

"This Instant Answer was made by the DuckDuckHack Community."

> There’s a lot of hate in this thread

You would do yourself a favor if you tried to engage with criticism instead of deriding it as hate. If you think something is wrong you should be able to come up with convincing counter-arguments.

> towards a company trying to do the right thing.

A company that claims they are trying to do the right thing.

Have they? They whitelabled MSN/Bing search. It's probably Microsoft.

Just switched. Lets see how well brave search works out :)

I switched when they decided to "curate" their search.

I absolutely do not trust Brave. I tried their browser on my iPhone for a few months. Always chose the most private settings, don't save history, etc, so whenever I opened the app there was no evidence it had even been used before. After one particular update, I started the app and it opened up literally dozens of tabs. Going through them I realized each tab was a site I had visited in the past. Left a nasty review on the App store that they never responded to, but they pushed out another update almost immediately. Never again. I wouldn't be the least bit surprised if years from now we find out the CIA was behind Brave, like they were with Crypto AG

Brave was started by one of the Founders of Mozilla.

With respect to the Mozilla project (not so much to its current state) - that does not mean much. Different people were involved with it, and that was 25 years ago.

Less than 10 years go... but ok

It was also a time of Mozilla growth not decline like it is today

Hell I would more charge Mozilla with being CIA front group than Brave based on their modern actions

Proud Boys was started by one of the founders of Vice media too.

That doesn't make them any less of a hate group or less shady.

I could be mistaken, but I believe that DDG sources data from Bing and other large search engines... If their sources, such as bing decide to remove a site, DDG gets affected by this as well.

Ok, this is the start of the end for ddg, it was good while it lasted.

DuckDuckStop

Snake oil marketing? Its results have become less relevant to me now. I am using Kagi and Neeva for search results. Kagi is still in the beta but it does offer reasonably good output.

I ditched DDG after it started to "downrank Russian disinformation". Thanks, I'll decide for myself where real disinformation belongs to. It became just another tool of western propaganda. I'm not sure if Google does that, but at least they don't paint themselves as good guys anymore.

Russian disinformation is effectively spam. It’s akin to if your search results about copyright policy got polluted by SEO’d, backlink-filled, MPAA-funded blog spam about the evils of piracy and the Joy Of Disney that pushed down legitimate information about copyright policy.

The problem is that with only one side of a story, you never get a truthful story. The usefulness of having both sides of the story cannot be overstated.

Otherwise the side telling the story doesn't even have to try and keep it realistic and can forgo any notion of truth. Allowing for more blatant lies to spread as truth.

It is a very complex problem that I would say is best fought in early childhood development, by teaching our coming generations methods of discovering truth.

Stopped using them a while back. What is everyone using nowadays? Any good alternatives? I use a ra-pi with pihole for my dns. It blocks good amount of stuff

Core part of the article is at the bottom:

> TorrentFreak reached out to DuckDuckGo to discover why these domain names are not showing up in its search results. At the time of publication, we have yet to hear back. It wouldn’t be a surprise if the move is copyright-related, though.

So there isn't anything else known than "domains are delisted".

I don't see how this is excusable vs just adding an optional safety filter like for porn in the image search.

Censoring my search results is the opposite of what I want in a search engine. I will not be using ddg anymore.

It seems duckduckgo is a microsoft proxy: as far as I know, its advertising agency is the one from microsoft.

Is it actually DDG that is censoring them, or is it Bing and DDG is as well because it is using Bing results?

Proprietary software

Refused every independent audits

Political censorship

They are now moving to that other project, so predictable.. intelligence really is dead in the US

This change must be region-limited, or just takes awhile to roll out.

In Canada, I tried searching for 2 of the sites on DDG the story said were removed, and they both still show up.

Well, not using DDg anymore, based on this and the comments down below. Any recommendations?

Maybe I’ll give Lycos a try again xD

Try https://you.com - private and also feature rich

Same issue, youtube-dl.org doesn't show up in web results

Can you help me understand what you mean by it doesn't show up? It seems to be there like all over the place? In web, on the github app, in the reddit app results, etc.

https://you.com/search?q=download%20youtube-dl

At least when querying "youtube-dl" it shows a small snippet to the right where one can see the URL..

I tested every mentioned site, nothing is filtered in DDG for me. Hoax or just based on country?

Is there an alternative search engine that actually searches the web with removing search results?

https://you.com lets you like/dislike sources to see what you trust

Spam scam

Quite ironic that the only 'free' search engine left is Yandex.

So much for freedom and democracy (TM).

I hope you are not implying that Yandex does not censor its search results.

Really? Just searched "pirate bay" on DDG. First result is thepiratebay.org.

Seems like searching "fmovies" also gives some results.

That's odd because even Google shows the homepage for YouTube-DL.

We need a p2p, independent, distributed, FLOSS search engine.

duckduckgone

Damn it! You beat me to this quip! Here's your damn +1. Grrr...

Well, now I'm definitely switching search engines.

And I have removed duck duck go as my default browser

*search engine

What are you switching to?

isn't duckduckgo just a skin on top of bing?

They claim to have results from over "400 sources" but then don't list them.

It's Bing.

Doesn’t DDG just use Bing under the hood?

They show up in the search results for me

they've also started completely censoring nubile/18 (aka legal) porn images since a couple weeks

the fate of every successful venture, huh? getting screwed by political agenda and values TM.

I thought yt-dlp was recommended over youtube-dl these days. Any chance DDG suppressing a less-preferred program?

It's a search engine, not a popularity ranking system.

so is ddg becoming a google ?

Probably worse than Google at this point. At least Google doesn't hide behind "privacy respecting"

https://web.archive.org/web/20220313015243/https://www.washi...

a search using;

anyterm :: torrent

still returns alot of results for me sofar.

and so does youtube-dl;