Hacker News Reader

You say:

> Each of [Bing’s search partners] have to show bing ads, forward the users' IPs as per Bing's terms of service (to fight ad fraud etc).

But DuckDuckGo says:

> [W]e never share any personal information with any of our partners. The way it works is when we call a partner for information, it is proxied through our servers so it stays completely anonymous. That is, any call to a partner looks to the partner as it is from us and not the user itself, and no user personal information is passed in that process (e.g. their IP address). That way we can build our search result pages using these 100s of partner sources, while still keeping them completely anonymous to you.

https://help.duckduckgo.com/results/sources/

>Each of them have to show bing ads, forward the users' IPs as per Bing's terms of service (to fight ad fraud etc)

The ToS you are referring to is for companies without custom deals in place.

You can read DuckDuckGo's privacy policy here: https://duckduckgo.com/privacy

> I like DuckDuckGo. But unless they build their own search - I think the fundamental problem has not been tackled. There is no good, independent, private search alternative.

There isn't going to be, because it's really hard and expensive to do that. I don't see how any startup could even begin to compete with their own search engine.

The barrier to entry is extremely high: Bing probably would not have survived long enough to be profitable if it had been a startup. Microsoft put a lot of money into building it.

Subscriptions and micropayments would not be a solution for DDG because too few people would be willing to pay. Ad revenue is the only model that has sustained search engines in the long term.

Respectfully, I think you are definitely overestimating the cost of building a search engine. Do you have experience in search specifically? Asking in good faith here.

I think you're absolutely right that it's not trivial, but it isn't a completely unsolvable problem only addressable by the major spyware corporations like Facebook, Google, etc. I think you could build a decent search engine in about 6-18 months if all living expenses were paid and you could just lock yourself into your apartment with no distractions, with maybe a team of 1-3 other extremely talented engineers.

Wouldn't there be a ton of search startups if it were actually this easy?

The only honest answer to that question depends entirely on one's interpretation of the strong efficient market hypothesis.

I think competing with Google would actually be a more difficult problem

It's not so much the difficulty, though yes it is hard, but it's generally extremely difficult to raise capital as a late entrant into an established industry. Unless there's something disruptive. That said, it doesn't take that much capital to build a new search engine with today's computing infrastructure so a few have tried over the past 5-10 years but none were as good as Google (a minimal benchmark). They found a lot of initial interest then died out quickly (sorry I'm forgetting the names, maybe Powerset, Blekko and another). Without doing any "heavy lifting" and also thus not requiring much capital, these so-called "private" search engines have gotten some traction.

You will need data on relevant results for a broad swath of queries to fine tune your ranking models against, and will need to spend millions on infrastructure just to crawl and store an index of the entire web.

I see crawls from DuckDuckGoBot so chances are they're trying.

It would be quite easy to only index Reddit and StackOverflow.

Yes, you're basically right...but the problem isn't building your own search engine, it's in getting search ads privately as well as just being transparent and honest about how your service works -- none of which is true for DuckDuckGo sadly. Their business model is built on sending your personal data to bing when you click on ads on their search results page which link to Bing (moreover they say they never send your IP or personal information to a third-party, but presenting an ad on your search results without disclosing that the ad links to Bing may not "legally" violate that clause but it does in spirit). Ads are also localized so even if they aren't sending your IP address (which according to Bing they are supposed to be doing, but in their terms they claim not to), they're sending your location data. DuckDuckGo further refuses to clarify what data they do send (i.e. how accurate the location data is, what is it even) to retrieve search ads. They aren't transparent about how they work and their business model is fundamentally built on non-private search ads from Bing. It's not "true privacy" nor sincere to say you're not saving any user-related data when you're sending a lot of it to firms like Bing who do save it. The only search engines close to being truly private are epicsearch.in (part of the Epic Privacy Browser) and maybe some small, interesting efforts like private.sh, neither of which have search ads.

* Dealing with a proprietary cloud service is never going to offer privacy. If you want to be confident you have privacy, you need to use public, community efforts which you can become part.

* DuckDuckGo could be saying "We have a client from Kyrgyzstan, please give us an ad". Bing could be saying: Here is the heading, the body, and the URL. DDG could be dumping that in the page. Your privacy is not compromised.

* DuckDuckGo could be saying "We have a client from Kyrgyzstan and a client from Brazil and a client from Texas searching for 'plastic sponge bathtub' or 'green sponge bathtub' or 'green sponge beach'" in response to a search for "green sponge beach" from a Brazilian. Bing could be responding with several answers. DuckDuckGo would then send the relevant answer to the client.

* If you click on an ad, you have no privacy, end of story.

Does anybody have any infos about Qwant?

It's the only EU-based search engine with its own indexing engine ( https://en.wikipedia.org/wiki/Qwant )

I don't care if it's EU or not, but is the index real or kind of half-fake? (like "yes, we have an index, but it's 10MBs big...")

I started using Qwant ~2 months ago and it isn't bad: the snippets that are shown in the search results often don't contain the search terms that I used but the links were so far good - but I did mostly lookups for e.g. functions and other reference data - the few times that I challenged it with exact searches its results were similar to the ones of Google, meaning that it didn't return the websites that contained such exact sequences of characters.

I would like to know too. I read someone on social media alleging they also used Bing behind the scenes, without attribution, but I haven’t researched further.

You might be right :(

By searching some random words like "when to buy fruitcakes" (without double quotes) I get almost the same listing in Qwant as in Bing. The results are a bit too similar... .

No knowledge, but at least for my searches it has a definite edge against ddg. (I am European.) What I especially like is that in many cases qwant returns nothing if there is no exact match of my query (typically I made a typo.) I kind of like the feeling that the engine tries to search what I tell it to search, not what it thinks I want to search.

I like to be able to search things I have no idea how to spell. It's pretty easy to make a guess at spelling words and I want to at least have a suggestion below when something is wrong.

That's as easy as !g

> What I especially like is that in many cases qwant returns nothing if there is no exact match of my query

I agree - I admit of having been surprised the first time that it returned "nothing" but in the end I did like the feeling of knowing that it did search what I wanted to search.

I'd rather 100 people know 1% about me than 1 person know 100% about me. Even if the only thing duckduckgo did was spread knowledge about me thinner, it would have value. Google already has most of my email (because most people I send emails to use gmail) and my youtube viewing history. Giving them my general search history as well isn't something I'm keen on.

Duckduckgo isn't perfect but they're better than google. If I knew of something better than duckduckgo, I'd use that instead. But I'm not going to let perfect be the enemy of good.

Aren't DuckDuckGo searches a mixture of Bing, Yahoo, Yandex and their own crawler?

I don't know a great deal about their infrastructure and would love to learn more about it.

I don't know about the DDG searches, but IIRC Yahoo is just Bing now

This article is from 2017, but has some more technical detail behind DDG than other articles do:

https://www.wired.co.uk/article/duckduckgo-anonymous-privacy

So a mix of Bing and Yandex?

https://help.duckduckgo.com/results/sources/

There are basically 4 search engines left in the whole world - Baidu (China), Yandex (Russia, recently officially with KGB management), Google and Bing. That is all, full stop. Everyone else is a fancy GUI for on them. Google has more than 90% market share and each of the rest has like 1-3%. Building new engine is prohibitively costly and hard. So the lesser evil is Bing, by simple elimination.

AFAIK, DDG uses Yandex engine for searches in Russian.

You can't include Baidu in the list and say Google has 90% market share. Saying Baidu has 1-3% without specifying "in the USA" is misleading. They've got a billion users in China.

> But unless they build their > own search

IIUC, they have built their own search, but use Bing(among others) for paid ads.

If this is the case, it seems that it’s the business model that needs innovation and not the search engine itself.

Try to run this test. If you query "what is my ip" in duckduckgo, or any of the other ones I mentioned, you will notice in the description of one of the top results this IP: 207.46.13.147. It's a BingBot IP [0]. It's a good enough test to spot where results are coming from.

[0] https://whatismyipaddress.com/ip/207.46.13.147

Building even a mediocre search engine involves a gigantic index in >40 languages that is constantly refreshed, plus image and video indexes, porn, spam, malware filters, complex ranking, freshness, originality algorithms, constant quality analysis and continuous maintenance and improvements for the infrastructure. They are not transparent about breakdown of queries or details of their deals with Bing and Yandex. They try really hard to create an illusion that it is a real search engine, but seems like it is basically smoke and mirrors.

Wonder how viable it is for a web search engine to support itself on paid subscription. It could also be $X per N search result pages.

> Try to run this test: If you query "what is my ip" in duckduckgo (DDG)

I tried that, and did not see 207.* anywhere on the page. It displayed my IP address above the results list, and then a list of reasonable results for that query, without any IP addresses listed in any of the descriptions.

I see it in the 3rd result:

https://i.imgur.com/HbhvISk.png

This means DuckDuckGo uses Bing for its results. But it doesn't mean DuckDuckGo is forwarding your IP to Bing.

A duopoly will be marginally better at implementing user-friendly features like privacy than a monopoly, because there is at least nominal competition.

However I think we need to start asking deeper questions - why is it that concentration is so prevalent in the software industry?

Software is often said to have "natural" monopoly characteristics, but are those characteristics really a product of "nature" in the way that say the laws of physics are, or are they a product of IP laws and the regulatory choices made by the government?

I feel it’s because of network effects as well as inability to steal IP easily.

Network effects purely because once you have enough mass of people on board, you can easily attract more people and it is easy enough to distinguish yourself from your competitors that it deters people from switching and/or even being able to make a comparison.

Second you can’t easily steal IP. Anker making a next generation charger will have to face copycats who went to their OEM and were sold the design or the product itself from the backdoor. It’s simply difficult to keep things under wraps when it has a physical + human component. Not so easy in software. To substitute Instagram you’ll probably [0] have to write everything from scratch. Computers when configured correctly will not exactly sell your work from the back door.

[0] I mean source code is still stolen but it’s still marginally harder to use source code than just selling the stolen designs. Plus stealing source code works only once because the victim will then clamp down harder and then continue to innovate it’s way out of the mess.

> forward the users' IPs

Seems doubtful since that would be contrary to DDG's main marketing message and their reason for even existing.

AFAIK qwant is using its own engine

Do you have a reference about DDG using Bing?

It's not mentioned on their site, or their wikipedia entry, or anywhere else I could find, and the results were different when I tried a few searches in them both just now.

Are they "Using Bing" in the same sense they "Use Wikipedia" to put info boxes on the results page sometimes?

DDG does use Bing for _some_ of their search results, but it's far from just a reskin.

_most_ of their search results. They don't give a breakdown.

DDG uses Bing, Yahoo! and Yandex for non bang searches[1].

As of October 2019, Yahoo! Search is once again “powered by Bing”[2].

To me, results are as crap as Bing's.

[1] https://www.reddit.com/r/duckduckgo/comments/6uaibg/comment/...

[2] https://en.m.wikipedia.org/wiki/Yahoo!_Search

Bing is really good for porn for some reason

"Uses" is a weird word here. Because some people are suggesting that DDG is a reskin of Bing. But if we look at what DDG says[0], they talk about it the same way they talk about Stack Overflow. No one is crazy enough to say that DDG is a reskin of SO. What they are actually saying is that they are aggregating results. Which I don't understand why that would be a bad thing.

[0] https://help.duckduckgo.com/results/sources/

Money quoute right here: "We also of course have more traditional links in the search results, which we also source from a variety of partners, including Verizon Media (formerly Yahoo) and Bing."

SO and the other 400 sources are only used for instant boxes and other widgets. The real organic search results are all from Bing.

Full quote

> In fact, DuckDuckGo gets its results from over four hundred sources. These include hundreds of vertical sources delivering niche Instant Answers, DuckDuckBot (our crawler) and crowd-sourced sites (like Wikipedia, stored in our answer indexes). We also of course have more traditional links in the search results, which we also source from a variety of partners, including Verizon Media (formerly Yahoo) and Bing.

I'm not sure where you're getting

> The real organic search results are all from Bing.

That paragraph __DOES NOT__ say that

The paragraph clearly says that.

> more traditional links in the search results

It's super obvious to me what that refers to - the actual search results. The rest is just fluff.

Uses as some of their main sources:

> In fact, DuckDuckGo gets its results from over four hundred sources. These include hundreds of vertical sources delivering niche Instant Answers, DuckDuckBot (our crawler) and crowd-sourced sites (like Wikipedia, stored in our answer indexes). We also of course have more traditional links in the search results, which we also source from a variety of partners, including Verizon Media (formerly Yahoo) and Bing.

https://help.duckduckgo.com/results/sources/

The results are very similar but not identical to using yahoo or bing.

I may be wrong, but I read before on HN than bing is used for 99% of the search results.

"We also of course have more traditional links in the search results, which we also source from a variety of partners, including Verizon Media (formerly Yahoo) and Bing"

https://help.duckduckgo.com/results/sources/

That doesn't suggest anything that the OP is saying. Google does similar things. We wouldn't say that searching DDG is like searching Stack Overflow. Or it is like searching Yelp.

There's a difference between getting results from sites that contain content (Wikipedia, Stack Overflow, Yelp) and sites that are pure search engines (Bing).

There's also a difference between reskinning a search engine and aggregating other engines' results with your own.

Sure, and AFAIK Google does neither of those, so it would be wrong to say "Google does similar things".

> AFAIK Google does neither of those

> it would be wrong to say "Google does similar things"

This does not follow.

It also isn't what I was suggesting, but it would surprise me if Google didn't look at what Bing and Yahoo were doing (and DDG).

This isn't true about DuckDuckGo. You've come to an erroneous conclusion based on bad interpretation of information.

They survive based on ad revenue. OTOH they provide an ad-free and to my mind a less annoying "lite" interface as well.

<https://duckduckgo.com/lite>

Please edit your post as you’re spreading FUD and are clearly incorrect.

wtf is that true? ip logging literally defeats the purpose of "privacy"...

I’m not sure it is true.