My take on how DuckDuckGo can outperform Google | Hacker News Reader

My take on how DuckDuckGo can outperform Google

daniloedu.medium.com

163 pointsdaniloedu187 comments

The elephant in the room: DDG's search results are primarily sourced from Bing.

Unfortunately, as long as DDG depends on third-party crawlers, the suggestions to improve search results (& "the algorithm") seem far-fetched & naive.

(DDG does have its own crawler, DuckDuckBot, but apparently it's only used for very specific functionality.) https://help.duckduckgo.com/duckduckgo-help-pages/results/so...

> For DuckDuckGo, it may be tricky to resolve the issue permanently as long as it relies on Bing. https://torrentfreak.com/duckduckgo-restores-pirate-sites-an...

> According to various online forums, the best way to ensure your site gets indexed by DuckDuckGo is to submit it to Bing and Yandex. https://www.jessesquires.com/blog/2022/03/25/my-website-disa...

Moved from DDG to Brave Search about 6 months ago and haven't missed DDG at all. I can still use the !bangs I was accustomed to.

Given that Brave uses its own search index, and only a few times do I need to fall back to Google, it feels good to have a measure of independence from the Google/Bing dominance in search

Brave search does feel like a differens search engine, i really appreciate their goggles and forum post features.

is this related to the browser or do they have a standalone website for search? Never knew about it, looks like its https://search.brave.com/

It's standalone website. Not tied to their browser

Brave automatically changed my default search engine on private tabs to their search. I don't know if it was a bug or feature, but this kind of behaviour keeps me wary from committing.

Anecdata for anecdata: that didn't happen with me, I'm still on DDG.

Similarly, I moved to Kagi and couldn't be happier

Same, Kagi is great. I tried to make do with DDG and uBlacklist, but with my rules file reaching 269 domains in about 3 months, I gave up.

Love being able to customize the ranking order for specific domains.

> Brave uses its own search index

Did not know that. I immediately grabbed some recent "poor result" Google searches from history and tried them at search.brave.com — so far ALL results are better. Reminds me a bit of Google from ten years ago.

Google's decline into its present inability to NOT show only "assumed popular content" created a huge time sink in my life that wasn't there before. Not a fan of the over- and inaccurate use of "disruption" but it was a legitimate description for what Google did to other search engines, and this feels like a similar level of improvement. I'm frankly amazed at how good the search results are in my initial test.

> I immediately grabbed some recent "poor result" Google searches from history

Did you revisit every query from your recent Google searches (or) Is it that you have a good memory that you were able to remember poor results from looking at the query?

I'm asking this because, I've been contemplating a 'Search Engine Wall of Shame'[1] where people can submit their poor search results for the engines to make actionable changes towards improving them.

[1] https://needgap.com/problems/207-search-engine-wall-of-shame... (Disclosure: It's my problem validation forum).

Worthy project, thank you.

I re-opened several Google SERPs from my history and compared to Brave.

I often find NO results relevant to my query on the first three pages of search Google results, often seemingly because there is a "more common/popular" aspect of the topic I am searching about, and everything is about the common and related aspects and none about my use case.

Thank you.

Would you be willing to submit your bad search results to such forum? I welcome you to post your bad search results with the queries to that needgap thread for the time being unless there's a dedicated forum for that.

I am willing, but the odds actually doing so are low (busy and disorganized).

That's fine, Next time you come across a bad search result, I hope you remember this conversation.

Why hasn't DDG grown internal search competency? It's been years, and there's still no sign of search being a first class concern at DDG. That seems like a number one must have for any search product company.

Rebranded Bing with a new interface isn't hard tech. It risks so much on the business relationship. They should be staffing for this yesterday.

Hire more search folks, DDG! Also, ditch that awfully long name.

would also like to hear from DDG ppl. On another note, Really appreciate DDG's map functionality using apple map with iaxm query parameter set "maps" and q set to whatever you want to search for:

https://duckduckgo.com/?q=Africa&iaxm=maps

It is refreshing to be able to use a usable map on tor without having to accept google big term of service.

> Why hasn't DDG grown internal search competency?

Maybe because it's already a profitable business model as it is?

Because internal search costs a lot of money. Google has millions of search servers costing billions of dollars just for the hardware.

Maybe they could partner up with https://www.mojeek.com/ as an independent search engine with own crawling technology.

They are welcome too, but perhaps are restrained by their stakeholders.

Looking at the suggestions in the blog, and for transparency - Mojeek CEO:

> Stop trying to look like Google.

Agreed. We and our users still believe that "10 blue links" have great value. But drowning these out with too many of things like Ads, Videos and Answers on the page helps pretty much just Google.

> Arrange that algorithm to make it less vulnerable to SEO hacking.

Admittedly we don't yet have that problem yet. Still there are plenty of measures that we provide, and will expand upon to mitigate that. Without going into details these generally amount to giving users, and API customers, more control over searching and ranking.

> Discard AI-generated text.

A good idea. But can also be done on the SERPs. Do users benefit from AI answers? Mojeek is a search Engine not an answer Engine.

> Results in other languages.

Bing?

> New opportunities.

DuckDuckGo is doing a great job of providing new and improved privacy products. We stand with them on many things. I am sure they appreciate the opportunity also in providing more informational (search) diversity from and with a fully independent search stack (infrastructure, crawler, index, ranking).

> DDG's search results are primarily sourced from Bing

That isn't an accurate characterization of what we do now.

https://news.ycombinator.com/item?id=32360642

This seems to be the gist of the response you've linked:

> mobile searches are the largest category of searches, and local searches are the largest category of searches within mobile. Instead our local search content is a combination of our own indexes in partnership with Apple, TripAdvisor, and others.

This suggests Bing isn't used for local searches, but it is still used for mobile and desktop searches that aren't local. If I am searching for a tech problem or how to tie a knot or recipes or the history of hair metal, I'm getting Bing results wrapped in DDG. Is that inaccurate? Is Bing still used for searches that aren't local searches?

We've indexed billions of pages already at you.com to avoid this situation. The majority of our apps like Stackoverflow and Medium etc, we've indexed ourselves.

> So, that’s another opportunity for DuckDuckGo, by ranking the results based on the content’s quality and relevance, as Google used to do before.

This makes it sound like that's easy. Yes, SEO twattery is an issue, and with the growth of the web it's also much more prevalent and the stakes are higher. Google from 2022 has a much harder problem to solve than Google from 2000. DDG can't "just" solve that just as Google can't "just" solve that.

Let users just rate that search result, so if someone else searched for the same thing and didn't find it relevant, they can thumbs down. If they found it relevant, they can thumb up.

Would the votes apply only to the specific search query?

Site reputation score could be factored in.

There should be a relevance score, and it might be cross-referenced to other similar search queries that mean the same thing, but that's a machine learning project all on itself

That only works if people are using the same search term AND looking for exactly the same thing... If they're looking for different search things but using the same search terms (i.e. some people are using sub-optimal search terms), that's just going to add effective random noise to the ratings.

Given how location-aware some things sometimes need to be (including spellings / alternative words, which in my experience, Google is a lot better at than DDG), I don't see how that works at all.

Nobody said I can't use a SIMILAR search query to have a slightly lower bound for relevance. Machine learning can identify if two queries mean a similar thing in the context of search

What makes you think a user rating system wouldn't be gamed?

Especially since it has been tried and that is exactly what happened.

Exactly this. Pagerank is/was another form of voting.

You don't just average user ratings, you need to have a smart system.

First, each user needs to have a history or they won't be considered in the rating almost at all. Then, each user's history gets analyzed for average rating vs. the average userbase so you can determine whether this user upvotes everything or only rates the sites that are terrible.

So each user gets a score of "threshhold" of what rating site they will thumbs up. Some users will thumbs up a site that's 50% (mostly useless, but not outright spam), while some only upvote sites that are 80% (good content at the very least).

Then regressed for this variable, you can calculate their feelings on each site and see if it correlates well with other users. The more ratings, the more each additional rating is weighted.

This will reduce the number of ratings required for a statistically valid sample for each search query (since you need to tell users if it's relevant for the thing being searched, not just quality of the site). But you also need to fight bots as well.

This system may not be perfect, but it would add value to a search engine

Mate, I'm going to pay $1000 and get a bunch of Filipinos in a LAN centre using that SaaS company that routes your requests through other people's VPN software to click a bunch of thumbs up. Watch as my cialis video tops your search for EC2 instances.

We already have that. At least I can hope for a variety when random users are also involved.

My Filipinos will win.

Not OP, but allow users to follow others and create an aggregate filter. That leads to its own problems, but if results are done with a weighted preference depending on if have been noted as "good" by people I've followed, or to a lesser extent those they've followed, it may produce better results.

I’m not sure the solution is to create a search results social network…

Why? Perhaps only add people you know and trust IRL, or have some other sort of reputation system.

That’s way too much effort to use a search engine. I’m very technical, and I don’t see myself being interested at all - forget mainstream users…

You need to do it with user accounts so the search engine team can remove fake clicks and ban the user. Users with a low amount of ratings won't have any say in the rating of the site, since they will have a low confidence.

They will first need to rate a few hundred sites (valid ratings, because if they don't have high validity with the rest of the sites in the database the confidence this is a valid user will be low) to give the spam site a high rating. So they will have already built a database of good ratings before they can give the last vote that they get paid for?

Also, you can manually remove a spam site that uses this tactic and all their money will have been wasted.

Even better, let users leave comments on on search results. That stops all the "search query +reddit" searches because now Google is reddit. The discussions are happening on the search engine response page.

I disagree; Google isn't even trying - after all, Pinterest is still polluting image searches despite not even trying to do anything malicious.

However even if we assume that detecting spam content is hard, detecting the funding of said content is much easier - most of this content is there to get you to buy something or look at ads, and they use analytics tools to track conversion.

Use the presence of ads, analytics, affiliate links and other marketing tools as a proxy for "spam" and downrank them, or offer the user a "non-commercial" option like Kagi does.

Now you move the problem from old-school SEO "make my content rank higher" to "conceal my ads/analytics/etc from the search crawler so I don't get downranked". That is a very effective strategy because advertising has to be served via third-party code to protect against ad-fraud, but third-party scripts are trivially detectable.

Now the spammers and their partners have a problem - the only way to fool the crawler is to self-host the ads, but doing that would expose the advertisers to ad fraud. You're pitting the spammers against each other in a game where the only winning move is not to play and the extra overhead will make the spam less profitable (or not profitable at all).

99.9% of the web uses ads, analytics, affiliate links. I don't think anybody is interested in a search engine that only looks at 0.1% of the web.

This is the case because we optimized for it with a profit-centric internet.

If search engines start penalizing this kind of content then the 0.1% might grow. Not to mention, most of my web search usage nowadays revolves around open-source library documentation, blogs, GitHub and occasionally StackOverflow, so being constrained to the 0.1% of the non-commercial web doesn't appear to be a downside at least when it comes to searching work-related issues as a programmer.

A lot of the high quality content that exists on the internet only exists because the advertising pays for the people who create it.

Be careful what you wish for. If you make sites with ads rank lower the void will just be filled with crypto scams or affiliate marketing. Remember it's a lot easier to make a spam site than a high quality one, changes that disincentive high quality sites from existing won't necessarily do the same for spam sites.

Quantity of tracking and advertising is a great proxy for spam content. If a page contains 10+ affiliate links, I don’t want to see it. If it contains just one it might be more legitimate content like an author talking about their book and link which gives them a commission on it.

Look forward to multiple sites with just one link on each page then. Any kind of well-intentioned rule you set can be figured out and gamed.

I've been using crowd-sourced tools like SponsorBlock and Return YouTube Dislike, and I really like them. So far they've resisted attack using some kind of consensus algorithm. I've also been using the search engine Kagi, which lets me downrank and even eliminate certain domains and sites which I believe to be low quality. I wonder if it's possible to marry these. If the majority of users dislike a domain, it gets downranked. Of course bot detection and perhaps the use of logins might be required, but compromises must be made if the desire is a better search engine.

Such shifts are’t free. If spam is less profitable, you get less spam.

I follow a lot of YouTube creators who product genuinely excellent and valuable content. They typically limit their sponsorships to one short segment per long video. At the same time, their video descriptions often contain numerous affiliate links, preceded by a disclosure that the following links are to be used if you’re interested in purchasing one of the products and you want to support the creator.

Those links honestly do not bother me at all. I barely even notice them since video descriptions are hidden on YouTube, apart from a few sentences at the top. So how do we make search engines that distinguish legitimate creators like that from spammers who create fake blogs with fake reviews and tons of affiliate links, all using a fancy blog template that makes the site seem trustworthy?

YouTube descriptions started being useless since they became a copy/pasted generic blurb of text (irrelevant to the video at hand) and a dumping ground for various social media or similar links, so no surprise that you aren't bothered by spam links in something that nobody looks at anymore because it's already been spammed to hell. But there was a time when descriptions were all manually written and were an actual summary/overview of what's in the video.

> So how do we make search engines that distinguish legitimate creators like that from spammers who create fake blogs with fake reviews and tons of affiliate links, all using a fancy blog template that makes the site seem trustworthy?

It's a whole other problem for videos so I'm going to approach this from a website search results point of view: we simply use presence of commercial content as a ranking factor - if the commercial content is all that matches your query then you get it, but if something non-commercial matches your query equally well then it comes first. This encourages non-commercial content without banning commercial content entirely. Ads/marketing/etc are noise, so it would be normal for a search engine designed to serve the user to rank low-noise content above noisy content, right?

I don't know what content you consume, but most commercial content I find is garbage, while non-commercial content shines because it's produced by someone who actually cares.

First, what's wrong with having Pinterest images in the search results? For the types of queries where they show up, they usually seem to be pretty relevant.

Second, many high-quality sites use ads and analytics. What are you going to do? Downrank the New York Times because it has ads and uprank your local conspiracy theorist because he doesn't?

Why would you want a pinterest reupload of an image, instead of the actual source?

They've played games in the past with requiring a Pinterest account. Same with the Quora spam.

Pinterest hides the actual image from the search results in a sea of unrelated images, and blocks you from accessing the images without a login.

It’s impossible to find the actual image from the search results if the source is pinterest now that Google removed the direct image link from the search results.

> Downrank the New York Times because it has ads and uprank your local conspiracy theorist because he doesn't?

If the only difference in ranking between the NYT and the local conspiracy theorist is the presence of ads then yes absolutely. Realistically speaking there's no way the local conspiracy theorist will get the SEO "juice" of the NYT (including domain age/reputation, backlinks, etc), but if he does then he deserves his place. I don't want a search engine to be the one deciding what is considered the "truth".

Yes. Why hadn't I thought about it? Everyone talks about how easy would be to solve this problem. Well if it so easy do it and become the next Page and Brin, instead of suggesting other to do.

It's humans (SEO) vs AI (Google). Humans win so far.

It's simple. Whenever I query for "X", under the hood duck duck go queries "X reddit".

Jokes aside I think fixing the SEO hacking as the author mentions would go the longest way. It's really bad for stack overflow posts. When I Google a programming error the first few results are copy cat sites with the exact Stack Overflow questions and replies. I don't understand how SEO has evolved to favor that over the actual, more popular, stack overflow page.

> under the hood duck duck go queries "X reddit".

Both Google and DDG already do that, you just format the query correctly. Once you are an expert in a field you will begin to understand how bad reddit is for information.

But “X Reddit beats Google” (ironically on google) is the biggest meme right now. Are you telling me it barely works?

Everyone on HN is catching up to this, but what we are all realizing is we want actual user curation, trusted user curation.

Reddit is currently a good proxy because the best recommendations on Reddit for a question are upvoted reasonably democratically by the type of people you’d trust. For the time being the gamification of Reddit isn’t bad enough to be an issue here.

But that’s just a matter of time. As more and more people start using Reddit as the most authentic source of dependable information, folks are gonna start gaming that to the core as well, basically this is going to be a game like the Red Queen.

One possibility is that the search engine itself starts becoming like Reddit maybe? They clearly know what result you’re clicking, they could start making user similarities (or ask you for an explicit list of your peers) to find similar users and rank results based on what people like you click. Obviously google can do it easier since it knows YOu. But even DDG maybe able to do something like user curation or site blacklisting (and have popular lists like exist for tracking avoidance that you can subscribe to). This is the only solution IMO.

For example for any coding related search I’d be happy to rid of every top 10 result in google except SO. All the rest is garbage.

Maybe there’s also an upvote button. But the score is not universal but personalized to you based on votes from people you actually might trust.

Reddit is already a cesspool for many issues including politics, news and race/culture based questions.

Agreed. There are certain topics which for which one cannot find anything resembling unbiased and factual, and it's getting worse as the owners continue to turn Reddit into a sanitised, ad-friendly social platform. If one is using Reddit to find relevant information, they first need knowledge about which subreddits can be trusted, and this is in constant flux.

> Reddit is already a cesspool for many issues including politics, news and race/culture based questions.

So? The blast radius of those issues doesn't fully spillover into the non-political forums.

You want to know which of two different products in a particular class is better? /r/<productclass> won't be filled with political or culture-based proselytising.

Looking to buy a new laptop/car/power-tool/chair? Good advice is to be found on reddit and you may never even see a single off-topic discussion in your search.

Probably wont be long until the spammers figure out how to invade those spaces too though.

Everywhere is a cesspool for those issues.

SO, GitHub, often Reddit, and pure technical doc sites would be on my list of non-garbage from Google’s top 10.

(engineer at Neeva here)

I couldn't agree more which is why we recently launched our Onebox product bringing you forum content directly inline with your search results.

https://neeva.com/blog/introducing-neeva-onebox-community-co...

I just gave Neeva a shot and it's really great. I love that I can up and down rank domains, and it doesn't appear to filter domains using some shadowy ideological algorithm. I don't seem to have access to that Onebox feature yet but I imagine it's being rolled out by region.

One question: why are there no dates under results? Even for articles where the date is near the top. This is something I rely on heavily to filter out the older results. To provide an example, I searched for "keychron k2v2" and found this article from Sep 26, 2020 (https://appleinsider.com/articles/20/09/26/review-the-keychr...). I then filtered to show me only results from the last year, and this result still popped up. This suggests the Neeva engine doesn't detect dates at all.

Updating my comment...

We're actively working on this!

We should have significantly better coverage for dates in results in the next few weeks. Stay tuned!

Amazing! Thanks for the update!

This is why google tried so hard to succeed in social media.

Content written by humans has been moving to walled gardens like Facebook, Twitter, tiktok, Reddit.

Most of these don’t let google index them properly, as they would prefer uses create an account on their site and use their own search.

Brave Search has a Goggle that does exactly what you want:

https://search.brave.com/goggles?goggles_id=https%3A%2F%2Fra...

I've noticed Google has started to ignore the Reddit postfix more and more. Now I have to add site:reddit.com to get reddit results

DDG is a dead end. The good news, if you already tolerate DDG mostly because of the bangs, you'll love Kagi (https://kagi.com/), which supports them and has far superior ad-free search results.

Disclaimer: No connection except as a satisfied paying user ($12/mo).

+1 to this. Also no affiliation but I joined during their beta and have been a member ever since.

I appreciate their stated commitment to avoid censorship in general, but would prefer an explicit dismissal of ideological filtering. By the way, their pricing page is currently offering premium service for $10/mo.

Unrelated - I use kagi too! I was watching a supervised learning series on YouTube by Killian Weinberger here [0] and he used an email from kagi as an example of spam.

[0] https://youtube.com/playlist?list=PLl8OlHZGYOQ7bkVbuRthEsaLr...

That is curious, can you give an exact time point in video?

23:05 onwards in the first video

I’d like to try out Kagi but having every search I do tied to my real world identity (via payment information) with no ability to search privately really kills it for me.

Yup, also my main reason for avoiding them.

Kagi does not tie searches to an account, as a matter of fact it does not log them at all (and why would it when you already pay with your wallet)

https://kagi.com/privacy

Kagi claims they get search results from google, isn’t that not allowed?

Google offers this as a paid service: https://developers.google.com/custom-search/v1/overview

My biggest issue with DDG search results was always lack of 'context'. If i type some keywords related to recent political statement or an event, even if a bit generic one, I'd like to see information about that, the most recent one, from my country, not some years old results from USA.

Like typing 'Tusk HiT' in Google, should yield me a ton of articles about what Donald Tusk(a famous polish politican) has recently said on HiT('History and Present days' - new, propaganda school subject and highly controversial book of mostly fiction and opinions that was made bu our 'government').

If I type it in Google I get newest and relevant results. Is it at the cost of my privacy(they know my rough location)? Sure. Do 99% of users care? Not really.

If i type it in DDG, I get Songfacts about 'Tusk' by Fleetwood Mac... It's not even comparable. For every day use, it's simply a no go for most of the world, which is not really technical enough to even realize they can/should use bangs. DDG will never get higher market share unless they realize, people rarely actually know what they search for, they won't know if they wanna see a Wikipedia article or something else. They just want relevant results, and DDG doesn't really deliver that.

Often I want exactly the opposite of "relevant" results - I want results related to what I searched for, not to where I'm located nor to what's currently happening.

Someone on HN complained they couldn't Google their acquaintance around the time of the Olympics, because one of the participants had a name which was different but similar, and Google just couldn't grok you'd want to look for someone else than an Olympian...

The vast majority of my searches are either relevant events or non-contextual to begin with and I rather have the rare case where your example happens to be when I have to add extra search terms, instead than the other way around.

I honestly see HN and Reddit complain about Google results a lot and I wonder if y’all the minority use case tbh.

I could have like two buttons for the two use cases, or a checkbox. For sure a minority, hence why Google doing what they're doing...

Recently the son of a radio personality died. I was curious to if the family had released a cause of death and I was stunned by how many spam/bot-written sites where out there churning content for a fairly obscure death. If there are that many for the son of someone on the radio, I can't imagine how many junk sites there are for actual celebrities. It is a huge problem and I'm not surprised even google can't stop all the crap from rising to the top.

For me, DDG just needs to add blacklists/whitelists to search results that users can customize and I'll stay. I'm looking at switching away because SEO is so bad.

I feel the same way about job boards, but we're the product in both cases.

For this I use Google Hit Hider. It's an amazingly well-built addon. Works on DDG as well.

Kagi has this and it's amazing. The domain ranking is a range so you can completely blacklist something or you can make it appear higher in the results

Yes, so much autogenerated nonsense SEO sites in every search for me on DDG for about a year now, also thinking of moving back to google. Sure, !g, but if I use it every time, why not just google?

Every time I try g! to bypass SEO or “no relevant results”, the google results are just as bad as the DDG ones. I’ve found myself trying g! more often this year, but it has worked zero times for me in the last > 5 years.

https://search.brave.com/help/goggles

2022-06-22 https://news.ycombinator.com/item?id=31837986

I use DDG as my default and rarely use !g. My only wish is that DDG had better “immediate results” like Google, where if I am looking for a conversion or a time in a different city, it just displayed the result rather than me having to go to one of the first results and type in the desired conversion there. I love how Google does that, but hate everything else Google does so it isn’t enough to cause me to switch back.

I've been on ddg for years, have it defaulted in all my browsers/machines, and I can't remember the last time I used !g.

Not sure what you mean about immediate results though... if I search for, say, "10C in F" I get "10 Celsius = 50 Fahrenheit" at the top of the page. And "time in mumbai" gives me "7:37 AM" plus the offset from UTC and the date. Also "weather in chicago".

Same for conversions of lb/kg, mph/kmh, and some other units I just tried, as well as currencies. But either it doesn't do mpg<->L/100km or I'm formatting the search wrong.

It also doesn't answer "hello in french" with an immediate translation like !g does. But I'm ok with missing translations because languages that I want to look up are in wordreference, are easy to make shortcuts for, and generally have better translations and more thorough info.

When I do have to use google directly I sometimes find it _harder_ to get results because it doesn't just show some links and summaries, it seems to want to try to give me "solutions" which aren't always quite what I want. I've been told the results get better the more you use it (ie. the more info you feed to the machine, it learns what you want) but I almost always get what I need on the first attempt searching with ddg and usually within the first few results. If it's not there, my experience has been that I'm not going to find it with !g either. (Usually something to do with really obscure programming problems or error messages.)

An example of where it's lacking in immediate results is searching for Tottenham. DDG provides relevant news articles and a box for the city itself. Google provides the result of the most recent match for Spurs as well as upcoming matches.

I kind of wish somebody would do a command line app for these things. Currency conversions, time in x place, weather, unit conversions, date diffing, etc.

Most of it could even be done offline.

KDE KRunner has some of this, though not on the command line. I'd prefer to have a search engine running locally that handled this kind of thing, just because the browser can display richer results.

apt install units

$ units

You have: 37817.55 per second You want: per day * 3.2674363e+09 / 3.0605034e-10

Or 3,267,436,320 requests per day.

I was profiling redis performance, and I got approximately 37817 requests per second for MSET on 8x terminal windows when running redis-benchmark -P 2000 -c 6 -s /var/run/redis/redis-server.sock

Google is optimising for ad revenue.

Is it possible to take the google results and apply a domain score overlay on it to get rid of the spammy SEO sites? ie a community curated whitelist of sites that get preferential treatment

If a person had the knowledge/skills to get a spammy SEO site to rank high in Google, they would likely have the knowledge/skills to manipulate a community curated whitelist/blacklist.

You could have a group of humans vote on whether sites can be included in the whitelist and a main criteria is that sites nexus to an actual human involvement. Could that work?

Theyre optimizing for ad revenue by having their search results be good enough that ads are worth looking at.

For example, they may show a copy of stack overflow because it has more ads - for many users they will still click through and deem the copy site sufficient.

Same thing with other SEO sites that blatantly copy other websites

Stack Overflow has its own ads, it doesn’t appear to use Google. Why would they optimise for those clicks?

That’s the point - people are saying google is promoting the SEO sites that copy content from stack overflow and use googles ad system

Do you have any evidence of this? It would be a huge scandal if Google was found to be boosting organic SERP rankings/placement based on the site's use of Google Ads and I have no reason to believe this is true.

No evidence, just conjecture.

No ads are worth looking at.

The search algo is just a community curated whitelist, it's just more efficient than a human maintained one, using more metrics and optimising for relevance instead of ban/don't ban a site.

The problem Google has now is either: it's algorithm is broken and it isn't showing relevant or quality results, or, maybe, the majority of the web is just trash and there are lots of relevant but very few "quality" results

With everything happening in the tech world today, I think there are opportunities for players that have users as their priority in their culture and don't consider them as a product.

We will never see this happen, but I wish there was a legally mandated separation of web indexing services from the result ranking services. So user-facing search engines would be forced to rely on independent back-end companies for indexing the pages and storing results (in a neutral and interchangeable way), and launching a competing search engine would not involve a multi-billion-dollar investment.

> So user-facing search engines would be forced to rely on independent back-end companies

Interesting notion.

So, if a state or business wanted to suppress certain information, they'd only have to kneel on the neck of one indexing company, rather than many. Having said that, if the index is certifiably clean, fair and uncensored, and access is free, I can see huge value in a public global web-index.

By the way, "Duck Duck Go" seems to be a terrible marketing name.

Yeah, they wanted the quirky catchy name but forget that it also needs to not be terrible.

No need to say why [particular brand] of search engine can outperform Google. Really, any search engine could outperform Google by doing the following:

1. When I search for "X Y Z", ensure that the resulting documents contain X AND Y AND Z. The plus signs and quotes don't work. No exceptions to this rule, unless the user specifically requests "OR" searches, fuzzy searches, stemming, synonyms, etc. But no one will request those things because those things suck. (Recommending spelling corrections are okay, however. Just don't automatically perform them.)

2. Do not show me what "People also ask". Fire the product manager who is keeping this shit in the results.

3. Do not mix images and video and news with web results. I see the "images" etc links. I know how to click on things.

...

Has anybody had this experience with Google lately: Fighting and fighting to get it to NOT show 800 billion non-matching results, and then finally getting the actual result you want, only to have it be accompanied by the warning, "It looks like there aren't many good results for your search." Bad design award of the year, for that one.

FYI, Google recently announced they're improving exact searches:

https://blog.google/products/search/how-were-improving-searc...

The thing about DuckDuckGo is that the search results are trash compared to Google, or even Bing.

For example if I search for something super simple like:

"what is another word for happy"

Google lists synonyms at the top of the page from Oxford.

If I do the same exact search on DDG

I just get the definition of happy and no synonyms.

Today, I would bet that the majority of searches aren't to find websites but to find answers to questions.

DDG sucks at answering questions.

DDG works great if you search for `happy synonyms`. It even has an equivalent list of synonyms embedded in the results page.

Actually even with "what is another word for happy", the first result is a wordhippo synonyms page.

Though I agree that question based queries are probably more popular these days, and DDG isn't handling them as well. I imagine this doesn't bother the tech / privacy crowd too much.

That’s a one box / instant answers query. A DDG search for “happy synonym” (no quotes) provides synonyms and antonyms.

The google result shows more antonyms than the DDG one, but the DDG result has a click through to rhymes and sounds-like words.

So, it is a draw in my book. I don’t try to ask english language questions to search engines though (and find it uniformly produces worse results than listing keywords, but YMMV).

Based on autocomplete suggestions, it looks like google auto rewrites your query to mine.

Arguably, it would be good to be able to search for pages containing phrases like the one you typed, instead of just ignoring user intent for people that have used search engines before.

shrug

Yeah, but if you want to compete with Google your going to have to serve the answer faster. No one wants to click on webpage to find an answer to a simple question.

This shows Google is way better at understanding what the user wants when searching.

"what is another word for happy" "happy synonyms", and "happy synonym" all give the user what they want right away on Google doesn't matter how you search it.

With DDG it does matter how you search it.

Another example is another simple query like "florida capital". Google in big letters tell your what the capital of Florida. It shows you a map of the city that you can click and shows basic information on the city.

DDG gives you information on the capitol building and the first link is too the Florida Capital Bank website.

Why would I ever use DDG over Google? Other than privacy which I don't think is that big of an issue as people make it out too be on Google.

Didn't you mean to search for "what is the capital of Florida?" ;)

Because that, amusingly, gives me an info box with the answer. Either way, the Wikipedia page for Tallahassee is the third link on the page.

It is amusing that DDG seems to be assuming I spelled "capitol" wrong, and pushing the correct results down as a result. I know the difference.

Growth of searches of DDG seems to stagnate? Any idea why? https://duckduckgo.com/traffic

I like DuckDuckGo, but I'm getting to the point where I just automatically do !g similar to adding Reddit after a search, which defeats the entire point of DDG.

The problem is I often search weird esoteric things that just don't appear well in DDG. Things about a game from 20 years ago. Specific, niche mods for said games. Indie music artists. Bits and pieces from old television or movies. Lesser known product brands. Almost anything that isn't very well named.

For those things, DDG's first couple (or several) results are often quite poor as they reference something that almost sounds related and is very popular/common, but actually has nothing to do with what I needed.

I like privacy and take easy steps to get it where I can, but I may end up ditching DDG.

Brave Search has been a nice alternative over the last few months. Still has the !g option if you need but been working well for standard queries.

https://search.brave.com/

Brave state that it's 'built on top of an independent index' but once you compare a few queries, it's obvious that they're just using a slightly modified version of Google results.

At least for me google search results have degraded drastically over the past years, especially over maybe the last. I haven't used it more than a few times in the past few months. I find brave search to have the original google quality, allowing me to find things right away. Maybe it's because google is adapting towards natural language queries, and brave is still "old school", and I haven't adapted, I don't know...

Second this. I have been using Brave as my default for half a year now and I am quite happy with it. It is remarkably free of spam, probably because it is not actively targeted by spammers (yet). I use Google if I am looking for something to buy, because that is where all the ads and deals are.

I think any search engine with smallish market share has an interesting advantage that few people are going to do search engine optimization targeting minor search engines. So, if one uses a different ranking algorithm, even if it isn't technically better in any sense, it's still an advantage in that it's using signals that people aren't deliberately trying to subvert.

Indeed! There is a connection to Goodhart's Law here. https://en.wikipedia.org/wiki/Goodhart%27s_law

So "just make it less vulnerable to SEO". My goodness. That Google/Bing/whatever didn't think of this.

For me DDG was as useless as Google: they both ignore my keywords whenever they feel like it.

I've sponsored search.marginalia.nu a few months (it is awesome for what it is and sometimes gave me better results than Google/Bing/DDG) and now I am a paid Kagi customer.

Yes, they too sometimes ignore quoted keywords, but they treat it like a bug when I report it.

Yeah I don't think a full generalist search engine is doable with my resources. Fun to see how much I can do, though, and I do think maybe there is room for more specialized services.

I'm just totally amazed that a one man project absolutely can compete with Google on providing an enjoyable search experience, and can even compete with Google on search results in some areas.

I use DDG 100% of the time and maybe 5%-10% after I have to add !g to get better results.

But maybe 1%-2% of the time I find DDG's results seem utterly unrelated to what I was looking for, like it randomly rolled a dice. I do mean completely unrelated. It's not a big deal but it certainly is a puzzle.

That seems to match my experience. Not the !g usage, that's much lower for me. We're at a point where if DDG can't find something, then neither can Google. Google will just drop keywords if it can find something, and frequently it will be the most important keyword that gets dropped.

The unrelated result seems about right to me, 1-2%. It depends on the subject, it will typically be search terms that have been SEO'ed to death. The same search on Google will often yield similar poor results, or an entire page of ads.

If Kagi is as great as people claim then that's great, we do need more competition among search engines. It might hurt DDG a little, but I would hope it move more people from Google to alternatives.

The key point for me is really how bad Google have become. The search results aren't amazing, Bing and DDG can easily compete. They have vastly overdone the ads. Some results are now almost an entire page worth of ads and no actual results. The important part is to have good alternatives to Google.

Before you try to differentiate yourself with new/niche features, you should make sure you're bringing table stakes. The biggest thing stopping me from using DDG is, sorry, the shittiness of the search results. Often I try 3-4 different queries on DDG before giving up and trying my original query on Google and getting the answer in the first result.

If there were a "Google with privacy" I'd switch instantly. But right now DDG is "Google with privacy, plus every other search fails".

Who cares about “how DuckDuckGo can outperform Google”? For some users, it does. Their value proposition is privacy, which for me was appealing enough to move from Google. When I found out they were actively censoring search queries, it was enough for me to move elsewhere. Every solution does not need to be THE solution for all users. There is nothing wrong with being a niche platform focused on catering to a specific demographic. In fact, it probably allows the platform to better cater to that exact demographic.

Their mobile browser makes search faster. It has a nice autofill email feature to proxy your own with a @duck.com domain, stripping the trackers from them. The !bang feature has been useful for me socially, like in conversations, making search quick enough to be a part of it.

Now if they can pursue some tactful strategies like the article hopes for, they can eject the Microsoft ads that they have been grinding their teeth over.

They need to put more resources into reliably handling the hidden service [1]. Since NY Times, BBC and even Facebook rolled out hidden services for privacy I see it as a sign of a mature organisation that is genuinely committed to values. DDG walks the walk, but the service is sometimes a bit slow.

[1] duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/

>"Total Internet Users Worldwide" *numbers are estimates.

According to the world map infograph in the article there are 4.5 million more people using the internet in Australia than there are people in Australia...?

So I'd say the estimates aren't great, or they're likely estimating number of internet connections rather than the people using them.

Using one single search engine really holds you back, in terms of results, and privacy. Use a metasearch engine. I recommend Searx/SearxNG. It's open source, so you can run your own instance if don't trust public ones.

https://searx.be/ for example.

I don't really like DDG's results but I keep using them just because I can scroll through results with the J key.

evil-mode

If DuckDuckGo really wants to beat Google, they could start by fixing the way the minus sign works in their searches. For example, if I type "cats -dogs" I don't want the first five result I get back to be all about dogs. This is the single biggest reason I ditched DuckDuckGo for Brave Search.

> if I type "cats -dogs"

OK, I'll bite. I tried the search: all articles on the first page are about cats, not dogs; most have the word "cat" at the beginning of the title.

I'm not literally talking about cats and dogs. Cats and dogs is DuckDuckGo's syntax example. DuckDuckGo's minus sign functionality is a well-known issue.

https://www.reddit.com/r/duckduckgo/comments/w5dzol/minus_si...

OKIC.

I never used DuckDuckGo because they used Yandex as one of their backends. I just did not want my queries, even anonymised, sent to Russia. They "paused" this practice only in 2022. Too little, too late.

DDG is so much better for technical searches related to programming

I find that, whatever the topic, the second page of DDG results is much worse than the first. Relevant articles on page 2 are often dups of articles on page 1. And page 2 articles are much more likely to be irrelevant.

I also dislike the scrolling behaviour of DDG once you've clicked for "More results"; keyboard scrolling (down-arrow) on page 2 takes me to the top of page 1. So I often !g instead of clicking "More results".

I don't know about others, but Google provides me relevant and useful results each and every time. I have no plans to switch to DDG or Bing or anything else anytime soon.

DDG doesn’t track you, but does it really matter if all the results have tracking cookies?

I don’t see a difference in the level of ad targeting I’m getting, even though I changed from Google to DDG.

> but does it really matter if all the results have tracking cookies?

No it doesn't. Just tell your browser to reject tracking cookies.

The killer feature would be to rate this search result. If I could see other users hate the site, I won't spend time looking at it. Thumbs up and down would work just fine.

Others may love it for a different reason than I hate it though. It would need to be more than a simple up/down to be useful.

I just want to see if it's a spam/scam page and avoid those results with less than 50% rating. The same way I used to see thumbs up/down on YouTube videos to see which videos are clickbait

I know 25,000 people in India who love spam sites.

If they upvote a lot of spam sites, their account will have a low confidence rating, and won't influence each site's confidence.

If they actually downvote a few hundred spam sites before casting a vote for a spam site to promote it, then they will have built a 99.5% good database already and one site can be manually removed. The only way to have a high confidence in a user if they both downvote bad sites and upvote good sites. If they did that work to verify their accounts to be valid, then they will have done more good than harm

> Maybe including a !Bang to search into TikTok could be a killer feature.

Seriously? I have no use for results that point into a walled garden where I don't subscribe.

Do you have to be completely ignorant of how DDG works to write this article? They don't index or rank anything. Microsoft is doing that.

It's not just how DDG works, but how search works and how difficult the search problem is. This was a complete waste of my time and people voting this up are clearly equally clueless or haven't even bothered to read it before voting.

A choice quote from the article:

> So, that’s another opportunity for DuckDuckGo, by ranking the results based on the content’s quality and relevance, as Google used to do before. I consider that there are some methods to accomplish this; one could be to rank the content through the community or actually use artificial intelligence to categorize the academic/relevance level.

LOL.

Yeah but I'm accustomed to the HN ignorance of the form "why do they not simply rank the document I want at the top?" People may underestimate the difficulties of web search and I understand that. But how can they not understand that DDG is just a marketing outlet of Bing? That doesn't take any thought at all.

Fair. I guess different "ignorance" ticks off you and me differently. I dismissed that aspect, since I assumed it's possible people are simply interpreting "DDG" as "any search engine that's not Google".

> But how can they not understand that DDG is just a marketing outlet of Bing?

People on HN knows that Bing powers much of DDG, but how would the average user know that?

In order to provide the best ranking DDG would need to track beyond their results page and see if the persons' question is actually answered.

For a privacy-focussed search engine that's a very hard goal to achieve without betraying their core value prop.

Google can do that through

- Google analytics

- Chrome

- Youtube embeds

- Google DNS cache

- Android DNS

- Tracking repeat searches

- Tracking similar searches

- Google fonts

Every other player is just so far behind it's unlikely we'll see anybody beat google in their core desktop and mobile search but there are certainly niches that can be attacked.

The reality is that satisfying all privacy concerns and building a great search engine are not compatible.

We've been working on many of these suggestions at you.com

Giving users control over their search results is a big part of fighting SEO spam.

The absolute flattening of the "daily searches" chart is interesting and I'm curious how it has continued.

I know a handful of us immediately switched out of DDG when they decided they were going to artificially down-rank Russian web sites from results[1], but I imagine the overlap between DDG users and people who care about this is quite small.

[1] https://slate.com/technology/2022/03/duckduckgo-russian-disi...

It is likely due to the most recent period being a partial time span. These articles come out every few months, and the most recent period is always low.

As for the Russia thing, as far as I can tell, they still show Russian propaganda sites (and often as the first result). They just don’t let SEO’ed sock puppet sites drown out all the other results.

I imagine the number of people that switched away because of that is vanishingly small.

A couple of times I tried Yandex and found out that it gives sometimes better results than google

They lost me when they decided to filter out disinformation from search results. I’m good on needing a search engine to “protect” me from wrong ideas.

Short of them completely reversing course and throwing out an apology, I’ve moved on from DDG. Companies that curate content for political reasons are a dime a dozen and we don’t need another one.

Why would you want a search engine to provide you results from websites/companies that are known liars, that you very likely would never know was propaganda etc?

It's not that I want to visit these sites, it's that I want people to stumble unwittingly to and be convinced by such sites that are on "my side"!

I'll try to answer your question in earnest. "Why would you want a search engine to provide you results from [websites] that are known liars?" Claiming someone is a liar is almost always a gross oversimplification and a form of propaganda in and of itself. Propaganda is partially defined as selectively presenting facts to influence people, which is exactly what you're telling people to do by shutting out opposing viewpoints.

If you are going to then say that people are too stupid to discern real from fake information, and therefore need the loving hand of Big Tech to tell them how to think, that's a pretty negative take. Asking Google or DDG if something is misinformation is like asking Monsanto if it's safe to drink Roundup.

> If you are going to then say that people are too stupid to discern real from fake information, and therefore need the loving hand of Big Tech to tell them how to think, that's a pretty negative take.

The first part of it is pretty busy proving itself true though. Don't get me wrong, I don't think big american tech companies are good curators of this, but previously extinct diseases are showing up again because some people are literally too stupid to vaccinate. There is far too many people who now believe the earth is flat, and so on.

So yeah, people are too stupid to discern real from fake information, and why wouldn't they be? If the school system fails to teach you how science works, then you're going to read sources that say vaccines are dangerous and you're going to see doctors who tell you vaccines are dangerous, because vaccines are actually dangerous. The problem with anti-vaxxers (not all of them, some of them think Bill Gates is inside the vaccine and that's just crazy) is that they read the information, but they don't seem to understand the information. Because vaccines are dangerous, but not vaccinating is more dangerous, which is why we vaccinate. This is completely anecdotal, but some of the anti-vaxxers I've talked with even seem to understand the whole vaccine > not-vaccine, but then don't fully seem to grasp the scale of it. They'll go "yeah, I know it's a little selfish but why would I risk my kids to help others?", which is very understandable, but what they don't consider is what happens when everyone thinks like that. Because vaccines aren't just > not-vaccine. That little > is actually on the scale of society breaking bad, but that part of the information doesn't make it through the noise.

I don't really think the world wide filter is the solution. I think we should get systems that are better at prosecuting people who wilfully spread harmful information instead. So that it doesn't take 10+ years for people who claim things like the Sandy Hooks parents are actors to face the consequences of their evil. On the flipside, I don't really want worldleadersarelizards.com to be the first results on my search engine either.

> Claiming someone is a liar is almost always a gross oversimplification and a form of propaganda in and of itself.

I don't buy this. Seems like fake information to me.

Now, to be clear, I do agree that tech companies must not be trusted to make decisions about politics. But the reason for this isn't because there isn't such a thing as a liar. I know you said "almost always", and didn't say "always a gross simplification", but still.

There are liars. They are people who tell lies.

DDG search are heavily censored anyway.

Reply on news.ycombinator.com