New ways we're tackling spammy, low-quality content on Search | Hacker News Reader

New ways we're tackling spammy, low-quality content on Search

blog.google

228 pointsxnx186 comments

The expired domain abuse thing --- that's interesting because finding that signal was one of the few things I got done while I was at Google... in November 2006! And I vaguely remember being told that the fix went into the codebase after I identified that it was happening.

Interesting you bring this up. We saw something similar happen at another FAANG where the fake review detection mechanisms that had worked so well were suddenly removed from the codebase, among other quality control safeguards.

Safeguards can hard to fight for when they make key numbers go down but improve something else that doesn't (immediately) show up in metrics.

Agreed, metrics such as customer satisfaction, customer retention, and brand trust.

I wonder if Google does something to detect fake Android reviews or coordinated review bombing? It seems to me, at least anecdotally, that they don't have effective mechanisms for this.

I mean, that was before the Doubleclick merger aka takeover that seemed to plunge Google into profiting from the problem instead of trying to fix it.

If your fix contributed to reducing ad revenue I can bet my money it was intentionally ignored. This is the only thing that matters to them.

I wonder if this will make a big change. I bought a company domain which I realised after a while had a different meaning before. It's also been linked to from Wikipedia. That means I've been enjoying the old free Google juice for quite a while.

The post alludes to it, but one of the targets here is AI-generated content.

> We’ve long had a policy against using automation to generate low-quality or unoriginal content at scale with the goal of manipulating search rankings. This policy was originally designed to address instances of content being generated at scale where it was clear that automation was involved.

> Today, scaled content creation methods are more sophisticated, and whether content is created purely through automation isn't always as clear. To better address these techniques, we’re strengthening our policy to focus on this abusive behavior — producing content at scale to boost search ranking — whether automation, humans or a combination are involved. This will allow us to take action on more types of content with little to no value created at scale, like pages that pretend to have answers to popular searches but fail to deliver helpful content.

And I’m reading this to say they can’t reliably detect whether something is AI-generated or not, so they’re not going to try. Whether human or AI or otherwise procedurally generated, sounds like they want to target this kind of low quality long tail search stuff.

This seems good though. Reminds me of https://xkcd.com/810/

Yes, but not quite.

It's that we can't tell the difference between the AI generated SEO spam and the human generated SEO spam, not that the AI has started to generate actually useful websites. And there was already so much human generated SEO spam that it couldn't be effectively moderated, but now the problem is a hundred times worse.

I believe it's more about manipulating pagerank with networks of "fake" websites.

Also, Google launches a tool to AI-generate web pages at scale. Maybe Search will be able to detect those Google generated pages.

https://thehill.com/policy/technology/4492468-google-paying-...

And to close the loop of shareholder value, Google will introduce Search Plus - a subscription-only search product that will hide AI content.

I would really love for Google to provide advanced search capabilities or filters to superusers who know EXACTLY what they are searching for. I wouldn't even mind paying for it tbh.

Have you checked out Kagi? It’s exactly what you’re looking for and more. They have a blog post on the front page rn about their integration with Wolfram Alpha.

> integration with Wolfram Alpha

That blog post was about the WA founder joining them, I don't remember reading something about integration.

> "By integrating Wolfram Alpha’s extensive knowledge base and robust algorithms into Kagi’s search platform..."

https://blog.kagi.com/kagi-wolfram

The more I read about kagi on hn the more it feels like organized shill campaign

We just had our daily stand up and I was asked to downvote you.

Hi!

Just Pay Kagi, they already have all of it and it works, it's in production.

Personally, I think they deserve the money, they offered a better paid product and never sold advertisements, which I think is a scummy model.

That is a niche product, Google doesn't want to have niche products for some reason. Google is great at making niche products, but leadership doesn't like them and always shuts them down...

Google has been A/B testing filters. I had it for a single search before it was gone never to be found again.

Oh why ... it is a good idea.

I guess we should be glad that Google make bad product decisions though, to hurt the company.

They used to give users much more control—the product has been deliberately crippled for a long time now.

That's a niche demand with a low chance of serving general audiences. SGE could be a potentially suitable user interface for this use case but I don't think they will integrate it into their entire stack anytime soon given the technical difficulty...

I would really love Google to let me ban certain domains like photo banks which feel attacked by Google searches.

I’m not paying Google any money though. And $10 a month is over my budget

Might I suggest the plugin uBlacklist. I use it to block a whole bunch of domains from google search that I don't want to see anymore.

You are an angel!

You don't need to use a separate extension to filter out search results. You can use this tool to generate filters for uBlock Origin: https://letsblock.it/filters/search-results

Is there a cli tool for that? I have over 300 domains in my ublacklist list.

You can try using ChatGPT or a similar tool to apply the same pattern to all domains in your list with a prompt like:

    "Use this pattern:

    example.com##a:has(.b)

    to create separate rules for these domains:

    example.net
    example.org
    ..."

eHow. I have literally never found a single useful tutorial from that site and they come up in just about every Google search.

Quora too.

used to be possible around 2011-2012 but worked too well so was obviously removed.

As AI generated content takes over the web, algorithmic search will become increasingly useless. In fact, for many things - like what to buy - it already is useless. Your best bet is asking your family/friend/neighborhood groups.

If I had trillions like google does, I'd invest in hiring a whole bunch of humans to curate a search index that is actually useful. It's something a new player will never be able to scale up to.

Right, LLMs will change the world, we know.

Algorithmic search has been useless for 5+ years. LLMs will just generate those spam pages faster, but they're already there.

It seems like it shouldn't be too hard for Google to create an AI filter that recognizes spam.

They obviously have the training data. Even if they needed to manually label a bunch of it.

The problem must be that they make too much money from the web pages that are just "500 words rephrasing the headline plus constantly-refreshing ads between every paragraph"

> As AI generated content takes over the web, algorithmic search will become increasingly useless.

This honestly seems unrelated to their poor search quality—hell, I'm even open to AI generated content for some queries. I blame catering to their clients and attempting to manipulate the content on the internet in the name of SEO for why I find useful results buried beneath products and ads.

> Your best bet is asking your family/friend/neighborhood groups.

Where should the ones you ask get information about the best options?

It seems much more likely that someone bought something for the wrong reason (easiest to find, being fooled by fake reviews etc) and then are recommending it onward because they are reasonably happy with it.

> Where should the ones you ask get information about the best options?

You ask multiple acquaintances who've purchased different things, and focus on the specifics of their experiences with those products or services, and figure out whether those things would work for you.

> and then are recommending it onward because they are reasonably happy with it.

If they're reasonably happy with it, why wouldn't I be reasonably happy with it?

> I'd invest in hiring a whole bunch of humans to curate a search index that is actually useful.

This is going to be a honey pot for the antitrust authorities. Google's main defense is that its ranking is done in an algorithmic way and human rating can affect the result only in an indirect manner following their rating guideline, which is carefully reviewed by lawyers. When you add some subjectivity from random human, they occasionally make some unfair mistake then it makes all the way through the media outlets, you know what's going to happen.

> If I had trillions like google does, I'd invest in hiring a whole bunch of humans to curate a search index that is actually useful.

Curated and sorted link lists were a big thing back in the early days of the WWW. So ... back to the roots, yes?

> As AI generated content takes over the web, algorithmic search will become increasingly useless

I'd love a way to search for things, and exclude anything created after 2020.

Sure, it won't help me find reviews of new restaurants, or understand new products or programming languages.

On the other hand, if I'm interested in reviews of a novel written in 1988, or how to solve some particular problem using calculus, that information isn't likely to go stale.

Haven't asked it for any fiction reviews, but ChatGPT is nothing less than a superpower for math-related questions. (If the last time you tried it was a year ago, try it again before posting something snarky in response.)

Google definitely isn't needed for that.

> If I had trillions like google does, I'd invest in hiring a whole bunch of humans to curate a search index that is actually useful.

The temptation for a for-profit company to corrupt that index would be too great to resist forever.

What modern capitalism really seems to do is produce cycles of improvement to gain marketshare followed by enshittification to cash in. The average of the cycle leans more towards the enshittified state than the improved state.

"To better address these techniques, we’re strengthening our policy to focus on this abusive behavior — producing content at scale to boost search ranking — whether automation, humans or a combination are involved"

But this isn't the problem.

People are just generating boatloads of "AI" generated pages with a bunch of ads. They don't need to be #1 in the search list, they just need some of their pages to appear in the search list frequently enough that the ads google places on those pages turn a profit.

Which google also knows, but again they're selling the ads, so they're getting a cut of it as well.

As long as google doesn't penalize pages on the basis of how many ad services and similar they use (and penalize google ads at the same rate) this will continue, and as long as it continues google profits from it. I get that a number of the engineers might not like that, but enough of them do that they long ago threw out basic privacy, accuracy of search results, and even the relevance of ads.

How much does one earn from Google Ads?

It's funny how they avoid using 'AI' term in the post when mentioning the reasons of spam. Given that Google is now an AI company, it may create a bad impression.

My "bar" for how Google does here is how it treats recipe pages. I absolutely loath what SEO has done for recipe pages, because now essentially every recipe page looks exactly the same:

1. Tons and tons of content about the history of the recipe, how humans first domesticated wheat, when the ancient Egyptians first discovered yeast as a leavening agent, etc. etc.

2. Then, after thousands and thousands of words, there is a short recipe for bread at the bottom of the page.

In fairness, this is a hard problem to solve. Some sites (rarely) have good information on techniques or things like substitutions in all that copy. But the vast majority of the time all that copy is just useless drivel to appeal to search engines. So my question is whether a great but pithy recipe for, say, focaccia or whatever would make it to the top of the search rankings, or will Google always favor the verbal diarrhea that sullies all recipe sites these days?

Is the long text purely for SEO or because a longer page gives more opportunities for ad impressions and higher engagement metrics?

Recipes are not copyrightable so the only thing that sets each site apart is the surrounding story and nonsense described above - otherwise every recipe is flagged as duplicate content.

I think forcing a user to scroll past 20 ads is a huge part of it. Usually by the time I get halfway down the page I give up on the idea of cooking.

I sometimes wonder how much of the costs associated with running a website are to pay for all the bandwidth for ads, SEO experts, etc… when a simple site with HTML and a dash of CSS could likely host recipes very cheaply.

I also frequently wonder what the excess cost (monetary, environmental, etc.) is for the modern web’s habits. Optimized schema? Nah, just chuck everything into a huge JSON blob. Need one function that you could honestly write yourself? Import this huge package instead.

It's not that hard to solve if you have a "I like this / I hate this" signal from the user.

Unfortunately for Google, G+ was probably their best chance at something like that. Now they just have stuff like time on page to try and figure that out.

No I could just make another bot to rank myself higher.

Google already has to implement bot protection for its ads.

https://www.justtherecipe.com/

Perhaps this is a case where Google could rip the recipe off the page and display it in Search.

Download or buy a bunch of recipe books and then search inside them. They're of much greater quality. YouTube cooking videos are also superior in quality to anything you'll find on Google.

They’ve known about this for years and have done nothing about it because it’s working exactly as intended, and there were few decent alternatives available to us.

Now we’re supposed to believe they definitely haven’t been abusing their monopoly, have just realised all this and are taking this grave threat to quality search Very Seriously Indeed, to ensure that you, dear customer, can find what you need. Couldn’t possibly have anything to do with encroaching threats to their core business.

Google: it’s been obvious for a good many years that you think us all a bunch of fucking peons. I, for one, will enjoy watching you and your clusterfuck of a product portfolio go the way of the Yahoo while I funnel my $10/month to Kagi.

I'm interested in entirely private, self-hosted web crawlers. The sort of instance you could set up on a $12/yr VPS to provide search results for small, specific portions of the web that you wouldn't get rich details from Google, Bing, et al. Personally curated search.

I don't know of anyone who does this well, unfortunately.

As much as I appreciate what Kagi is trying to do, I don't think I necessarily want a replacement for Google. Or if I did, it would have been Google from the 2000s.

I want something different and better.

YaCy might fit your bill: https://yacy.net/ Although, I'm not sure if a $12/y machine has enough space & RAM. But with YaCy you can setup your own crawls and only index the part of the web that you want.

I am interested to see the results of this. I had Kagi but got rid of it due to the cost. The results were a little better overall, but asking Gemini or Bing gets me to the same place faster most of the time. If I have to search manually, independent of the search engine, I am going to have to open like 3-8 tabs to get my answer most of the time. The difference between having Kagi and not having it is a couple more mouse wheel scrolls and 2-3 more clicks most of the time.

Try out Perplexity. It’s so much better for me personally usually for such things.

For me using Kagi is also a way to help push sites that actually respect the user. My hope is that if it gains enough traction it will force some of these bigger sites to abandon some of their dark patterns. And in the meantime, I get a much better experience with no ads, and constant new features, which makes search feel new and interesting again.

As long as Google doesn't have keyword search it's useless to me. Bring back quotes around a series of words, with an exact match. Bring back +WORD and -WORD to include exact matches and exclude exact matches that I am _not_ interested in seeing.

I DO NOT GIVE A FUCK about "popular" or "trending" matches. I want to see results that EXACTLY match what I am looking for.

Why is this such an impossible ask for search engines? 15 years ago it was the norm, not the exception. If SEO is the problem, then BLOCK THE SEO TRASH AT THE SOURCE.

You can use "word" instead of +word.

-word is still a thing that works a bit.

Too little, too late. I've stopped using Google search altogether since I subscribed to ChatGPT and Claude for a nicer, ad free experience

not sure why google doesn't launch google search pro with functions similar to kagi with more customization...

Someone should roll out a new tool to rerank Google’s results by the inverse of the number of ads on the page of each.

So... Kagi?

The acid test: can I find a brownie recipie without any fucking around?

...nope, I still can't.

Just opened incognito window and entered "brownie recipe"... the first link is to "Love and Lemons" site which seems pretty good - there is a pretty small introduction, but then there is ingredients list, steps, etc.. top 5 links all look pretty good to me. (At least in web design area... I have no ideas if recipes themselves are tasty or not)

Maybe it's a regional thing (I am in the US)? Or your profile is messed up? Try searching in incognito window.

The first time I visited "Love and Lemons" (the first non-sponsored link I got) I scrolled down looking for the actual recipie until I gave up. Right before you made this comment I checked again and found the link to the recipie under the introduction as you described.

Most if not all recipe sites have a jump to recipe button, as de facto required by search engines.

Way too much fucking around.

to click on a button?

Yes. I wasn't searching for a button, I was searching for a brownie recipie

If the developer has to put that button on the site, they’ve already failed. They know it’s a poor design, and are trying to add features to point to that will absolve them, instead of fixing the core problem which wouldn’t force the user to look for a skip button. First time users of a site have no reason to look for a button like that, when the standard control of every other site for decades has been to scroll.

I also just went to the site. When I went to tap the jump link, a modal popped up and blocked me. I dismissed that and tapped the link and it jumped me down the page to yet another request to sign up for a newsletter I don’t want. Now the top 3rd of the page is a video that started to auto-play; I had to tap to dismiss that. When I scrolled into the recipe section I got yet another modal trying to get me to sign up for an account so I can save the recipe… as if bookmarks don’t exist. I dismissed that, and now I can see the recipe. There are 3 ads in the middle of the recipe that break it up, some with more auto-playing video, and the scroll keeps readjusting itself based on ads and BS popping in and out. The bottom of the page also has an ad banner to dismiss, a bookmark icon that bounces occasional (which is certainly just another attempt to get me to create an account), links to other recipes are sliding in and out depending on the direction I scroll, to try to get me to go other places on the site… and to top it off, the Pinterest flag blocks content, depending on where I scroll, while sometimes being blocked itself by the aforementioned recipe slide-in.

How can anyone defend all this? It’s horrible.

I'm not defending it. It's a result of google's ranking system+ad system. All these descriptions are not for users to read, these are for google crawler to rank higher in search+to get more money from randomly placed ads. If you don't do this, chances are high your website will rank lower compared to someone that did this considering other factors are the same. I'm just saying that out of the all bad options, this option with a button is less bad, you skip all(most) unnecessary stuff with a button click

I think the main point is that all this trash is a problem Google created, and has no incentive to fix. The web would be a much better place with these practices weren’t the foundation of SEO.

Yes, issue is why this button even exist instead of actual content being first and foremost

Is that really a ranking signal?

https://duckduckgo.com/?q=brownie+recipe&ia=web

Returned Love and Lemons first too. Never heard of them, but their SEO must be amazing.

I'm pretty sure it's because they integrated some google thing in order so show that rating and reviews right below the search result. It has the highest rating/amount or reviews ratio so it feels like this logic is putting it to the first result. Will be not surprised if most of these comments and reviews are some language models talking to each other, it's a bit uncommon to see THOUSANDS of comments, ratings and reviews on a recipe site, probably SEO optimisation.

Kagi gave me the same

So kagi displays spam in the same way as other search engines do

I get BBC Good Food as the first response, which is even better.

Just checked out Love and Lemons. Sorry but I can't say it impressed me at all. The vast majority of the page is blogspam garbage. Nobody needs a ten-paragraph introduction and an explanation of why you're using cocoa powder and vanilla in a brownie recipe.

Interestingly the first result for "brownie recipe" for me is BBC Food which is about as close to no-BS as you'll find.

Your brownies are probably going to suck. Just because you can't be arsed to read more than two sentences about how to do something doesn't mean others can't.

It's 299 words (so close to a round number!) of worse-than-useless filler before the recipe. You get two amazon sponsored links and one blog plug for some author with that. There's also an interstitial you have to click through.

I can see why GP was not impressed.

The very first thing on the page is a "Jump to recipe" link. If you too are a victim of dwindling attention spans like the rest of us, you have the option to skip straight past the beautiful photos and background.

Looks like these spam pages work because some people actually like them...

That would be a major problem for hand curated indexes.

> So this fall, as I flipped through Michelle Lopez’s new book Weeknight Baking, her recipe for Boxed Mix Brownies, From Scratch caught my eye. Would I finally be able to make homemade brownies that would be just as delicious as the ones from a box?

I can’t even fathom how anybody thinks that this is fine on any level (from the mentioned site, the first result about brownies for me, and one of the first few paragraphs). For a while, I rather pay for cooking and recipe books, because wasted time on these texts would worth way more than that money. Especially that even the recipes themselves are terrible most of the time on the internet.

But after a thread a few months ago here on HN, in which people praised w3schools how it’s really a good site, I’m not surprised on anything. If the people who really should know how these things work, and that there are better free alternatives for all content on that site, even encourage this bullshit, then this won’t improve at all.

The other advantages of buying a recipe book:

- you can take notes in the margins

- you can find your favorite recipes quickly because the pages are all stained and wrinkled

- you can reward the author for all the time they spent testing and honing those recipes

The real depressing thing here is the amount of whining about "wasted time" from folks looking for free cooking instructions on the 'Net. It's almost as bad as the "gimme teh codez" jerks on forums.

FWIW, I like this brownie recipe (and have purchased the book it comes from, twice): https://www.seriouseats.com/bravetart-glossy-fudge-brownies WARNING: contains even more words than the "love and lemons" page.

They’re not free, and never were. Tech bros just telling you, that you pay more, if you don’t pay in cash.

I was recently gifted a recipe book via Amazon. I was excited until I opened it up—it was just low quality LLM generated recipes, poorly typeset. There's no escape.

Actual issue is why "jump to merit of page" link is needed at all, not ones ability (or lack of) toactively filter content while reading

The article says they're announcing in advance, so maybe whatever they plan to do hasn't happened yet.

Never tell a Googler what the metric is.

Cue a flood of aspiring L6 promo packets with detailed numbers on improved brownie recipe ranking.

Hey, as long as it gets Sally’s Baking Addiction[0] ranked at the top I’m all for it.

0: https://sallysbakingaddiction.com/

My go to site for baking recipes is Smitten Kitchen.

https://www.google.com/search?hl=en&q=site%3Asmittenkitchen....

I realize adding `site:...` is probably fucking around. But trust me, her brownie recipes are worth it. :-)

Isn’t the point of Google search to not have to do that?

Sure, that's why I added the disclaimer. I'm just trying to help Fricken out with some good brownie recipes.

Google lists dozens of recipe sites when I search for "brownie recipes" and honestly, how many different ways can you change a basic brownie recipe. So I'm sure almost any of them will do. Any seems like reasonable result. I'm not sure what algorithm Google should use to pick the best or most reliable recipe site.

I just happen to know, having cooked many of SKs recipes, that they are good recipes that almost anyone can follow.

They're not trying to find a good brownie recipe. They're using it as a test phrase for search engines.

My go-to site is King Arthur's 200th Anniversary Cook Book. Literally zero fucking around, packed with every classic baking recipe known to man, and gobs of straight-forward baking chemistry lessons a five year old could understand. Empowerment.

Katherine Hepburn has the brownie thing smacked down. Look no further.

I'm sure you have a recipe already, but I'd like to highly recommend Stella Parks' brownie recipe, and even more highly recommend her blondie recipe with malted milk and browned butter.

Go back to a simpler time... `before:2015 inurl:blogspot.com`

I googled MIL-C-44072 and it popped right up.

http://everyspec.com/MIL-SPECS/MIL-SPECS-MIL-C/MIL-C-44072C_...

I had to squint to see the actual content

That’s what she said about you, too. Hey-o!

I just use the Wikimedia Cookbook: https://en.wikibooks.org/wiki/Cookbook:Table_of_Contents

Maybe some browser extension that extracts just recipe from all the ads and superfluous narrative crap?

Simply add a user call-back rating system, then subtract a factor of 10 for anomalous feedback traffic inconsistent with unique regional ISP blocks and human users.

They can't do this as 95% of the internet would disappear overnight including a lot of Google ad revenue.

The proposed project has a paradoxical use-case defined, and as such will not function as expected.

Where do we send the invoice for stating the obvious? ;-)

lol

If you think more deeply, you'll realize why that won't work.

It encourages cheats to identify themselves, and automatically identifies hostile population clusters through a clique algorithm.

i.e. people get a local version of the results consistent with their local cultural values. You are correct in that the cons/spammers on this forum wouldn't notice anything change from their perspective.

Maybe we should deploy it on the ISP DNS servers for fun =)

It'll cause a slew of even more direct manipulation services to pop up to either up-rate the current site or down-rate competitors. Not to mention further fueling cancel culture.

What they should do is let people set preferences for how they want domains ranked in their personal results, and then use longer term trends on how those lists evolve.

Indeed, initially the stats are random, but Google already blacklists domains that hammer their services. People have tried the click-army approach many times.... only to discover their efforts are fleeting due to cross-checking the rank with CDN and DNS traffic.

If this worked Yelp would have a fair ranking system.

Yelp does not filter the vitriol because their business is shaking down businesses to remove defamatory content.

Statistically, sociopaths are <3% of the population.

They make one mention of obituary spam but then fall short of mentioning other specific issues they are going to fix. I’m not convinced this is going to lead to real change (like the last announcement). Let’s see what happens.

Most of the low-quality content I used to see on Google was their ads.

The best way to improve Google search results is to stop and use Kagi[0]. Seriously. If you are reading this article and these comments then you must be somewhat interested in search results. Do yourself (and the search world) a favor and subscribe.

[0]https://kagi.com

Kagi is head and shoulder better at finding things that are useful. Google is going the way of AOL.

How do they plan to monetize it? How many people will pay for Search? Will they have an ad-supported version? If so how would their direction be any different when they face the same problems?

It’s a monthly subscription that apparently a lot of people are willing to pay for, feels like an honest business which is shocking in tech.

We don't know if they're profitable doing this. One day, they might go kaput as well.

at least they said they are

In the early days a user could go into their setting, see the cost per search, and the total cost of their use per month, compared to what they paid. I threw some extra money their way when I was my use exceeded what I paid. But that was before they changed up their pricing structure.

How much is "a lot"? The HN bubble doesn't count.

>20,000

https://blog.kagi.com/celebrating-20k

Nearly 24k paying users and ~400k searches per day

https://kagi.com/stats

This comment convinced me to subscribe to Kagi. I've been hearing so much about Kagi on HN that it's time to see what all the hubbub is about.

If you end a sentence with a question mark an LLM will attempt to answer your question. It's great.

There is so much Kagi references on HN that I’m slowly beginning to feel sus about it.

I’m not saying this is happening (I’m not saying anyone is a shill). It’s just making me feel a little “hmm” about it.

It's not just hacker news! They were up in arms about some actual analysis of search quality here: https://danluu.com/seo-spam/

Another common response is

>Your table is wrong. I tried these queries on Kagi and got Good results for the queries [but phrase much more strongly]

I'm not sure why people feel so strongly about Kagi but, all of these kinds of responses so far have come from Kagi users. No one has gotten good results for the tire, transistor, or snow queries (note, again, that this is not a query looking for a daily forecast, as clearly implied by the "winter 2023" in the query), nor are the results for the other queries very good if you don't have an ad blocker. I suppose it's possible that the next person who tells me this actually has good results, but that seems fairly unlikely given the zero percent correctness rate so far.

For example, one user claimed that the results were all good, but they pinned GitHub results and only ran the queries for which you'd get a good result on GitHub. This is actually worse than you get if you use Google or Bing and write good queries since you'll get noise in your results when GitHub is the wrong place to search. Of course you make a similar claim that Bing is amazing is you write non-naive queries, so it's curious that so many Kagi users are angrily writing me about this and no Google or Bing users. Kagi appears to have tapped into the same vein that Tesla and Apple have managed to tap into, where users become incensed that someone is criticizing something they love and then write nonsensical defenses of their favorite product, which bodes well for Kagi. I've gotten comments like this from not just one Kagi user, but many.

I'm not really a fan of this person's dismissive attitude, they're trying to avoid dealing with domain rank adjustments, understandably so the results are comparable, but the adjustments are part of the value add.

The testing methodology is also odd, looking specifically for yt-dlp when asking for "download youtube video", when the web based downloaders that most of the big engines turn up on top make more sense for the average user, both due to not having to install a command line tool and due to not necessarily being on a device the tool can be installed on. I've often used the top Kagi result, and it has worked fine, yet has been labeled as scammy by the evaluator. IIRC that site also ranks high in Google's results. For adblock they expect specifically ublock origin, and consider ABP to not count despite it also being a very popular blocker.

Them complaining about why it's only Kagi users reaching out to him also seems unnecessarily aggressive, eg I've found their article through you, in the context of Kagi, so if I were to comment, it'd obviously be in that context.

However, how likely is this article to be brought up in the context of Bing, Google, ChatGPT or DDG, where there are many more carefully thought out and less dismissive comparisons and discussions out there?

fwiw I agree with you that the evaluation criteria here are not ones that I particularly care about - I'm generally pretty happy with AI-first and then expanding to search engines if I can't get what I want there. My main intent was to note that Kagi users are just genuinely that enthusiastic, since GP at least lightly implied there might be some astroturfing going on.

I think it's just that search engine quality comes up often, and through the recent very popular Kagi threads, a lot of HN users have ended up at least giving it a try.

In a sense it's targeted advertising, since HN users are very likely to be power users who care about a paid search engine for one reason or another. But it's still organic in the sense that the articles posted are ultimately just the regular search engine discourse or otherwise not content specifically crafted for the purpose of advertisement.

Well you're right. I'm a shill for kagi. The experience has totally converted me. However, I'm not affiliated so I'm an organic shill - healthier and less toxic.

I feel it’s much more that people on HN are able to evaluate that Kagi is much better than Google, and really want other people to see that. Unfortunately, shitty search results are usually good enough for most people, so it’s hard to justify the switch.

I'm a happy Kagi user. It's not ground breaking or anything I'd even recommend to most people. It's a slightly worse Google search, with substantially better power usability. Normal, every day searches could be done on either Google or Kagi and the Google results are slightly better especially with regard to geo based critera. I don't have to specify "Name of City" in Google nearly as often as I do in Kagi.

However, when I'm trying to find some archaic technical solutions that I know is out there on the web somewhere, Google just can't really do the job any more at all. Exact search doesn't even work! Kagi's filters DO work, and the ability to black list stackoverflow scrapers cuts out a lot of the useless noise. It's at those times when Kagi earns its place.

I talk about it here, but I first stumbled across it on Mastodon. I think it’s more that the type of users Mastodon and HN attracts are the same type of people to pay for a search engine and be excited about a search engine.

I have long hated that “free with ads” is the first and only business model anyone thinks can work on the internet, so I want to support companies that are willing to push back against that. Ads in search create a conflict of interest that I don’t like. Ads in most products create a conflict of interest I don’t like.

I pay for YouTube Premium for a similar ideological reason. I want to support the business model that doesn’t rely on showing me ads. In a perfect world, this would mean that the features and algorithms of YouTube would change to prioritize me as a user, rather than pushing constant consumption, but with Google its baby steps.

There's no lasting "paid, without ads" model.

There's only "free, with ads" and "paid, with ads eventually".

https://www.amazonforum.com/s/question/0D54P000079nP9wSAE/ho...

There's no lasting pizza either, because tomorrow it has been eaten. So you should never buy a pizza or any other food, because after a while it has literally turned to shit.

free with ads seems to be lasting business model

I’ve subscribed just yesterday. Cannot be happier.

I trust the non-ad-based company that I'm paying for quality results more than Google, that's for sure.

Edit: but I'm glad Google is noticing their results are getting to be shit.

I finally took this advice a couple of weeks ago. It's been great. Zero regrets.

I prefer not having my full search history linked to my personal info and payment details, but that's just me.

The privacy, spam, signal to noise, and user experience equation is different for everyone. For me it's a good tradeoff since they (say) don't store your history and the ad profile Google builds on you is probably more damaging to your privacy than a photo of the entire contents of your wallet.

> they (say) don't store your history

They don't show your history.

> the ad profile Google builds on you is probably more damaging to your privacy than a photo of the entire contents of your wallet.

I can take measures to prevent Google from linking together a full profile. It sucks that it's necessary, but once you log in it's effectively game over. And your payment info has enough stable identifiers that it's trivial to link you to pretty much anything else you've paid for.

---

The core misconception here seems to be the idea that there is such a thing as a honest for-profit corporation.

That's why I said it's an equation, a balance, if you will. Sounds like you are all the way over on one side. We need the Stallman's of the world but most people are not that committed to a single issue.

> The core misconception here seems to be the idea that there is such a thing as a honest for-profit corporation.

And I would say there's degrees. Even if all companies eventually slouch into a google-like organization, at least you can jump on the bandwagon until that happens and find another when it does. And maybe none of the wagons are headed to a strict privacy, user controlled data, and free-software utopia, but maybe not all of them are headed to Gomorrah.

Kagi explicitly promises not to store your history[1] and then takes your money based on that promise. If they do store your history, then they are committing fraud on a grand scale.

[1] https://kagi.com/privacy

I see lots of promises, but very little proof. While, again, running a business model that massively increases the inherent risk compared to the competition.

One of the best things about paid business model for search is that it aligns incentives and decreases the risk of data mining as the business makes money through subscriptions, without any need for ads/selling data.

Doing one doesn't prevent them from doing the other. It gives them more data about you to sell, while limiting the dataset to gullible rich people who think that you can capitalism yourself out of the problems of capitalism (the prime advertising target demographic).

But what is the incentive?

Money?

You are the one claiming that Kagi is somehow special, trustworthy, and immune to the same influences that drags down everyone else.

Like all the companies that realized that quietly selling highly targetable user data is more profitable than their actual services? See all the "DNA analysis" companies selling data left and right

> I can take measures to prevent Google from linking together a full profile

> I see lots of promises, but very little proof

Yet you trust google and any company that might do business with google? This view is some sort of mind game you're playing alone.

Who said anything about trusting Google in the first place?

My point is that Kagi's business model requires more trust in Kagi than Google's requires in Google.

How would you go about proving such a thing?

Evert VPN app promises privacy but actually they don't honour that promise

> I prefer not having my full search history linked to my personal info and payment details, but that's just me.

From Kagi Settings page, search history toggle:

Save My Search History

Currently this option can not be turned on. Kagi does not save any searches by default. In the future we may add features that will utilize your search history and then we will allow you to enable this.

I encourage readers to think through the privacy practice implications of firms that may see your information "first party", and firms that make their money leveraging your information among third parties and the entire adtech ecosystem.

That's exactly what a honeypot would write.

I thought the same thing at first. Then I realized Google already has this info about me and more. So I just split up my searches between Google, DDG, and Kagi.

This is a bit tongue in cheek, but it is also kind of true:

The easiest way for Google to detect spammy, low-quality content is to ... simply add up the number of Google Ads placements on the page.

Those flooding the internet with garbage content are doing so because the verbiage is tightly intermingled with Google Ad placements, that's the business model.

Google -could- go a step further and deactivate those accounts, but that would impact their revenue, so nah.

In general, I believe Google's ranking system attempts to be hands-off and not hard-code what factors might correlate to good/bad rankings, but I do wonder if they had to put in a special rule to ignore any automatic quality signals related to their own ad system (or competitors!). It would make sense to do this for revenue preservation reasons, but I wonder if they also might have to do it for anti-trust reasons.

While we're just shooting the breeze, there is still something to what you're saying here.

It's a long held complaint that Google's ad/analytics/tracking scripts slow down page load/page render times.

It's also long held that Google prioritise speed in rankings.

My guess is that they'd be excluding their own scripts and ad load times as part of that page load calculation, even though that is not a genuine result and would down-rank pages that utilise competitor ad platforms.

Google is really bad at large scale coherent strategies. Google search ranking teams are almost completely isolated from Google ad teams. It is like two different companies and cultures, Google ads is run a lot more like Amazon.

I feel like I've read dozens of announcements from Google regarding its efforts to improve search quality. That can't be literally true, but the sense of deja vu inculcates a certain skepticism.

In a links post I just published (https://jakeseliger.com/2024/03/05/links-a-moon-landing-rapi...), I wrote:

Still, the Google search monopoly is under more serious threat than it’s been in the company’s history, and that may inspire real change. Amusingly, I used Google search to try and find a video of Sergey Brin saying that he’s un-retired to come back to work on AI, and Google search didn’t easily find it—but it turned up a bunch of spammy YouTube videos.

It's an arms race, so "we are improving search against spam" is a thing they will post when they fall behind sufficiently. This goes all the way back to the "Panda" update ten years or so ago. It's been a thing for ages because they've got a Red Queen problem.

There was a freakonomics podcast episode about this the other day. Basically, plumbers with names like “AAA Plumbing” tend to be worse. Plumbers who can’t compete on service tend to try to directly game the rankings.

In this case, 'tend to be worse' could simply mean they receive more negative reviews, because the overall volume of jobs they pick up is huge compared to plumbers who didn't try to manipulate where they appeared in the directory. Happy customers leave fewer reviews than angry ones.

Ironically this blog feels like it was written by an SEO bot. There's basically no meat to what it's saying and spends each paragraph rephrasing the title without adding anything.

The preferred term is GenAI

So far their solution seems to be ranking Reddit for every query.

Until Pinterest is gone from search results, I won’t believe them

This is a perfectly reasonable point/benchmark. I don't know why someone thought it was so bad it needed to be flagged.

I read that as "New ways we're tacking spammy, low-quality content on Search"

An eye opener for me was that I didn't question it. Has Google really fallen so low?

My gut feeling to this is "fuck you and I hope you fail for only tackling this after chatgpt threatened your success"

Reply on news.ycombinator.com