Update: Still investigating, but have made some progress -- Determined that on desktop there was a link to Techdirt up continuously via our About module (when you search for "Techdirt"). And now the traditional web link is back up as well (for desktop and mobile): https://duckduckgo.com/?q=techdirt&ia=web.
Over the past fifteen years, search has become "universal" with dozens of indexes and modules throughout the page. Also it is important to understand people click/engage with things on the page roughly half as much each position down, so by the time you get to the bottom it is like 100 times less than the top. This means that things on top, increasingly non-web links, have become more and more important.
In this context, as mentioned in the other comment, the largest modules on mobile and desktop we power ourselves, that is, local and knowledge graph. AI will be the same. We do use Bing as the primary source for traditional web links, but not for all and even when we do it sometimes looks different in various ways. In something like this, we can re-insert this link if that is what is needed.
> Bing is our largest source of traditional web links.
I think "traditional web links" is the main (and I think should be the only) product of a search engine, and it seems like you rely on Bing quite significantly to serve search results.
- maps
- directions
- local business listings
- about boxes
- basic facts
- conversions
- news
- videos
- images
- shopping
- sports
- stocks
- weather
- flights
- entertainment (e.g. cast)
- etc.
What we really want to know is if for links alone - what extent do you index? Let's ignore the knowledge graph, let's say we just want a simple word match.
I'm surprised you didn't have techdirt's homepage indexed at all. "techdirt" did not return "techdirt.com". To me this means you aren't indexing the home pages for popular websites at all - let alone their content - and rely effectively exclusively on bing with maybe some "caching".
Talking about links alone - exact word matches with the domain name (or page content other than your knowledge graph), what do you index?
What specifically do you do other than adding sites manually to a list and I assume "caching bings results"? To me this would still be 99.999999% ~= 100% reliant on bing for search results.
This isn't meant to be an attack in anyway, I'm just looking to clarify what's going on.
> We do use Bing as the primary source for traditional web links, but not for all and even when we do it sometimes looks different in various ways
I don't know that it's reasonable to demand that they provide you an exact percentage.
Personally I'd like to know if that means "we have a manual list we re-added". I think what myself and probably many people are most interested in is "do you rely on bing for < 95% of word match searches (no knowledge graph)"
> when we do it sometimes looks different in various ways
IMO this is also vague - I assume this means they're knowledge graph augmented? I really want to know if they have their own indexer (vs a manual list + caching bing results) for text matching.
This is context-dependent. If you're looking for a link, then you're looking for a link; but more often than not, you're actually looking for answers.
Say you're looking up an exchange rate or the value of a stock; converting units; checking the weather; or just want to look up an actor's photo. Why would you click and wait for a (likely, slow & ad-infested) page to load, if the search engine can provide factually correct data without any extra steps? How often do you read past the first paragraph on Wikipedia?
If you want to dive in / verify any of that information, it's still all just a click away. But having to only do one thing instead of two is almost the definition of technological progress.
I want room for both.
There are times I want the search engine to stop "helping" (sic) and just give me a straight search based on my terms.
There are other times where I'm looking for something specific but I don't remember exactly what it is, so I want help correcting/ narrowing down my search.
One potentially relevant line from the article:
>I love that first one. Microsoft, a company with a $2.5 trillion market cap, “may not have enough resources” to crawl and index Techdirt? Cool.
I have to imagine DDG's valuation wouldn't quite hit 2.5 trillion if it went public. It's not absurd that they aren't able to fully duplicate those efforts (or that they don't need to in order to have a decent product).
They fixed the problem so it shows ability to. Maybe the issue is “detect what parts we should do ourselves”? How do we detect censorship beyond blog posts like this? Can we diff various indices on a regular basis?
The whole point of using Bing as the base is so that you don't have to make and keep an index of everything yourself, so DDG wouldn't have another exhaustive index to compare to (and if they would, they wouldn't need Bing). They index some things to add onto Bing, but they don't try to capture everything.
Furthermore, any specific site missing from the index does not necessarily mean a flaw - okay, you would diff some indices, and find out a long list of what Bing has excluded... and then what? For every 'fixable censorship' case (accidental or intentional) there will be thousands of spam or malicious sites which should be excluded, as nowadays the key part of search is not finding everything but throwing out the results which want to be found but shouldn't. Again, DDG wants to piggyback on the effort that Bing is doing to filter the index, and if they want to second-guess all Bing's filtering (as opposed to just making a fix for this specific case) then they have to replicate and improve on all of the (huge and expensive!) Bing's filtering effort; which goes counter to the reason for using Bing which is to avoid all this expense.
Bing is their initial and primary source of data for web links. Not the only one either, but the primary one. They maintain their own index from it.
That’s not “a mirror” by any stretch of the imagination.
What appears to have happened here is some form of replication bug where in a way yet to be determined a removal from the original data source was unintentionally replicated.
Data synchronization is hard when you’re not directly mirroring data. This kind of bug crops up
Not really. That’s the point. Data replication is hard, especially when you have to consistently update from the same data source.
Bugs happen.
What is their other data source? And how did it come to suffer from the same problem?
I keep seeing insinuations that DDG has some other secret source than Bing, but it's always hand waved and never explicitly named. Incidents like this seem to strongly imply that only Bing really matters at the end of the day.
Personally, I don't even mind if using DDG is the same thing as searching bing so long as it actually works, but they still can't get a simple search like "office -microsoft" or "headphones -best" right. I even thought they acknowledged the problem and were looking into it at one point.
It’s not an insinuation, it’s a straight fact direct from the founder.
That you don’t understand data replication problems—or know the inner workings of a private company’s codebase & data funnel—doesn’t make it some conspiracy.
It just means you’re uninformed.
I'm glad to be corrected. What is their other data source?
All the stuff they claim to do themselves is <1% of whatever value DDG provides. Lucky for them it’s just enough to be able to use terms like “largely” instead of “entirely”, “other sources” instead of “only source”, etc.
They would fast become outdated however, but that’s a different problem. One solvable with another source.
Long story short - at some people in the past someone proxy-mirrored all content of SaaSHub. You'd open a page like "someshady-proxy-mirror.com/duckduckgo-alternatives", and it will mirror saashub.com/duckduckgo-alternatives. The same happened for all all pages. Soon after that, Bing (and DDG respectively) dropped all SaaSHub links from the index and kept the proxy-mirrored content! WTF.
After a week-or-two of "fighting" I managed to block the proxy-mirrors; however, I never got SaaSHub back in Bing's index.
I'm a single founder and feel helpless with this Bing/DGG issue. Any help would be appreciated. Thanks.
If some tech kerfuffle is happening, there's a good chance someone involved will see it. Its like a wider ranged subreddit, or a more orderly Twitter.
https://news.ycombinator.com/item?id=36899187
https://news.ycombinator.com/item?id=36899072
“This shouldn’t happen, we are working out why it did” is probably the only official response possible right now.
Elsewhere in the comments he’s addressing misconceptions & outright falsehoods about how DDG works.
Edit: results appear on DDG, for me, apparently.
I'd reported a problem with the "lite" interface about two months ago. It was not only acknowledged, but fixed, in just over half an hour:
(We definitely don't want fake/bait stuff here either so I share your feeling on that - this just isn't a good way to express it, at least not on HN.)
<https://news.ycombinator.com/item?id=35669888>
You might take his suggestions to heart.
I don't really like to be cheeky/sarcastic, but I don't really like people telling me how to behave, I do me, you do you, especially if you are an admin
It’s either people are compatible with a community or they are not, asking to change means people have no personality and you can just shape their behaviour at will, i guess that’s true for tech people, but not the rest of the world
We're just trying to have an internet forum that doesn't suck—it's nothing personal! The problem we face is that large internet forums end up sucking by default - that's where the arrow of entropy points - so if we want to avoid that outcome, we need to spend a lot of energy to deflect it, and we need users to help with that.
There's nothing intrinsically wrong about posting unsubstantive/flamebait comments on the internet - there are other sites where that is common and expected. We're simply trying to play a different game here, and for that we have different rules*. Just like baseball rules say you can't tackle the pitcher, and football rules don't let you whack the ball with a stick, HN rules say not to be snarky, not to post flamewar comments, and so on (https://news.ycombinator.com/newsguidelines.html).
Btw, what you said about "changing others" is correct in a way - I post like this as a moderator because I'm trying to persuade users who are breaking the rules to change their behavior. (Not make them change - just persuade them that it's in their interest.) But you're wrong that this isn't doable—it does work sometimes. You decide whether or not it works in your case, of course, but HN has a long history of users switching to using the site as intended, once they understood why it was in their own interest to do so.
Here's why it's in your interest: HN is only worth visiting in the first place because we have these rules and most users follow them. Flamewar, for example, can be exciting for a while, but in the end it's just repetitive and boring. The idea of HN is to satisfy curiosity, and for that we have to avoid the snarky/flippant/sensational/indignant/etc. types of internet posting, all of which destroy the site for that purpose. It's in your interest to participate in the intended way, because then you're contributing to the reason why it's worth visiting at all. I'm sure you wouldn't leave fires burning in a campsite that you enjoy, or (less dramatically) drop trash in a pleasant park. The way we see it, asking people to play the intended game here and stick to its rules is sort of like that.
* this is also why I often ask people "Please don't do this here". It's not "please don't do this" or "it's wrong to do this" or "you're bad for doing this" - it's just "please don't do this here."