Hacker News Reader

Comment by yegg | Hacker News Reader

First, it is misleading to say our results just come from Bing. That's far from the case in actuality. Please see https://news.ycombinator.com/item?id=31490994 for a more detailed explanation on that.

On other search engines, they all rely somewhat on either Google's or Bing's web crawling: Qwant, Bing and Brave, Google (and Bing for images). This is easy to see as a webmaster since you don't see their crawlers much (if at all). Only Google and Bing are doing full scale web crawls. However, search is a lot more than traditional web links -- in fact it is about half now from instant answers that can come from dozens of sources and indexes (which the above comment gets into).

View on HN

> First, it is misleading to say our results just come from Bing. That's far from the case in actuality.

That's just corpo-speak. For the most part, Duckduckgo is Bing with some additional features. That's true to the extent that when Bing decided to censor the Tank Man image, it was removed from your results too. [1] Not that you guys refrain from censorship yourselves. [2]

The crawler (DuckDuckBot) doesn't have much of an impact on the search results, it's mainly used to provide instant answers. [3]

[1]: https://www.theregister.com/AMP/2021/06/04/search_engine_tia...

[2]: https://nitter.net/yegg/status/1501716484761997318

[3]: https://seirdy.one/2021/03/10/search-engines-with-own-indexe...

This is Josep M. Pujol from Brave Search.

I'd like to correct some factually incorrect information regarding Brave Search.

Brave search crawls the web through the Web Discovery Project and has its own crawler, which fetches a bit more than 100M pages daily.

Brave search uses Bing API and Google fallback for about 8% of the results shown to the users, the remaining 92% are served from our own index, when we launched almost 1 year ago the number of results from 3rd parties was 13%.

There is no need to mention "multiple source" when a number can be given. The underlying theme here is not if DDG provides no value on top of Bing, it does, no one is questioning that. The question is whether DDG would be able to operate if Bing were to shut DDG down tomorrow.

If Bing and Google were to disappear tomorrow, for whatever reason, Brave search would continue to operate, that's the independence Brave search is building.

What factually incorrect information was posted? Maybe I missed it.

Yegg said "they all rely _somewhat_ on either Google's or Bing's web crawling" and you confirmed it by saying "Brave search uses Bing API and Google fallback for about 8%". So... which part is factually incorrect?

Edit: Misread the second part, removed that portion of my statement.

Brave indeed brings high hopes in this field!!!

I was just hopping that, since it was based upon tailcat, it would create a bigger disruption by being opensource like Gigablast.

(However, i understand that, from a financial point of view, that's a risky move. But like DDG... i hope that maybe someday it will be)

> The question is whether DDG would be able to operate if Bing were to shut DDG down tomorrow.

No, that doesn't appear to be the question at all. The original post appears to be an attempt to smear DDG by posting misleading information that you know will confuse users into thinking that their search engine sends PII to Microsoft when you know it doesn't. The original tweet doesn't appear to mention Bing shutting down at all. Here's the entirety of the tweet:

"This is shocking. DuckDuckGo has a search deal with Microsoft which prevents them from blocking MS trackers. And they can't talk about it! This is why privacy products that are beholden to giant corporations can never deliver true privacy; the business model just doesn't work."

I see nothing in there questioning whether DuckDuckGo will still be around if Bing goes under. I also see nothing in yegg's response above that has anything to do with this irrelevant question you mention.

There is plenty of comments discussing on the provenance of DDG results, including from Gabriel himself, which is the one we both have participated in,

"it is misleading to say our results just come from Bing."

Discussing how many sources can one bring together it's a distraction to not discuss the degree of dependency between DDG and Bing. More-so when claiming that others suffer from the same, which is factually incorrect for Brave search.

Yes, i've read most of the comments...

But what i would REALLY like to see on DDG (beside it becoming fully OpenSource - someday) was having it taking the step of not relying on Bing.

That is a terrible shadow over you.

You should embrace the momentum you're having and step up and do your own thing. Your own index. Yes, you have to build/rely on others such as wikipedia... naturally

But please... NOT on big tech!!!

Or one day, someone else will get there and eat your lunch (Honestly, Qwant is a great alternative that apparently does not rely on big tech)

I maybe wrong, but i think this is what everyone one wishes from DDG!!! I believe everyone wants DDG free from Big Tech

(but everyone is free to correct me).

Qwant is also using Microsoft's Bing Search API.

"Qwant uses different programming interfaces such as those of Microsoft Bing, Twitter, YouTube, or iTunes" -- https://about.qwant.com/en/legal/classement/

WT...?!?!? I was not aware of this!!!

Is Gigablast, really, the only "alternative"?!?!?! (in the meanwhile, since there is no advantage, i'll be back to DDG....... for the time being...)

Still, my point remains: we could, at least, have a version of DDG that would only use it's own index!!!

Isn't that possible? (I would be using it!!!)

Mojeek - totally independent, no-tracking; but I'm biased.

Independent take: https://seirdy.one/2021/03/10/search-engines-with-own-indexe...

Mojeek is one of the very few players that are building their own index, my respects.

Holy shit, stop calling everything anyone in this thread says "misleading". Your answers are so off-putting. GP didn't say

> [your] results just come from Bing

You're the one being misleading here by suggesting that they did say that. Nobody is interpreting what they said as "All search results from DDG are just straight from bing". You're nitpicking words in almost every one of your responses.

If there are truly only two full-scale web crawlers left, and they are Google (ew!) and Bing (ew!), then it's high time there were more options. I naively thought DDG was exactly that, with a 100% focus on privacy. The cake is a lie.

Care to give some numbers instead of handwaving? What percentage of the queries are answered from bing index? Or even better what percentage of queries that resulted in an ad click were powered by Bing? A ballpark estimate is fine.

Reply on news.ycombinator.com