Is Google Getting Worse? A Longitudinal Investigation of SEO Spam in Search [pdf] | Hacker News Reader

Is Google Getting Worse? A Longitudinal Investigation of SEO Spam in Search [pdf]

downloads.webis.de

276 pointsDanielleMolloy274 comments

I don't understand Google anymore.

Surely people can relate to the situation where you end up on an article based on some technical query you have. The article repeats your question 7 times, has endless casually-related filler text that still does not answer the question and then ends with: try to unplug it.

It is so freaking obvious that it's a malicious content farm, but Google with all of its technical might seem unable or unwilling to detect it. If tech can't do it, organize some type of curation or feedback?

Same for image search. You search for "red flower Thailand" and flowers of various other colors from various locations appear. The idea that Google is spectacularly good at subject detection from imagery does not seem to actually work out in practice.

Most people's search queries consist of just 2-3 words. Nowadays Google consistently just drops the last word as if it knows better than I do what I need.

High value elaborate articles on various topics do not rank. Instead, dated articles do. You have to manually bookmark high quality content as you see it, because you'll never find it back via search.

Is everybody asleep at Google? This is not a small thing, this is your bread and butter. Teens are using Tiktok for search, you're in real trouble and better start cleaning up your act.

> Same for image search. You search for "red flower Thailand" and flowers of various other colors from various locations appear.

Now, if any of these flowers are next to a red dress, tapping the dress will reveal links to places you can buy it.

Google is not asleep. It has just got its priorities wrong. (Or rather, incentives in this organization seem to reward not what users like me appreciate.)

Huh, oddly one of the harder things for me to search for is places to buy things, unless they're very mass-market consumer goods.

Not like Amazon doesn't have the same Google problem. Except they openly discuss how the user doesn't have a choice.

I would gladly pay a commission for a service where I vaguely describe a product and preference and have the best version of it delivered without having to spend hours researching and hunting for deals. Of course this immediately runs into severe perverse incentives and similar issues.

The thing is, all the stuff you’re listing as things they suck at properly detecting, filtering, categorizing, they used to be extremely good at. The google search that exists today is markedly worse than the Google search of 5 years ago. Wtf happened to cause search to just rot away into a useless mess of results that used to be very high quality? It’s such a night and day regression that I’ve legit wondered if this is a sign of the Mandela effect.

What you describe here in some order could be pointed at the fact they let someone who used to focus on Ads run all of organic search AND ads. This used to never happen, they had proper separation of church and state. The other part to blame imo is turning down the link graph, over reliance on NLP, and attempting to continue to prop up traffic to old media sites.

https://www.wired.com/story/prabhakar-raghavan-isnt-ceo-of-g...

Couldn't agree more.

And it's not just Google. Amazon is a mess. Social media is a mess. It all used to kind-of work but it's rapidly falling apart.

I wonder if they've misapplied the Youtube algorithm to searching. Other people who liked the pictures of red flowers also liked green flowers and purple flowers, so let's include that in the results, since that will probably generate more engagement and more engagement is obviously always good.

no, the rank Google ad buyers and sellers.

probably easy to confirm with a web crawler and generic searchs

they fundamentally changed their rankings to favored Google ad dollars

it's really not a deeper technical problem.

It always starts at the top. Sundar sure seems like Google's Balmer. He's good at keeping the lights on but from an outsider's perspective he doesn't seem to have any vision.

As an example of how bad the searchability is nowadays, I’ve been creating and expanding my own knowledge base (something like a personal wiki with links to interesting content I find) for about a year. It seems to work very well despite the effort it takes to keep it organized.

I've maintained my own hosted wiki since 2004, and yeah I'm glad I didn't completely outsource information management. It's definitely getting hard to find certain things I know I've seen.

Now I just need some kind of open source search engine to run on it ... (a bunch of text files that render to HTML, and ideally following the links 1 or 2 levels deep)

~20 years ago Google desktop search was a fantastic piece of software ... very fast and accurate on your local files. I don't think something like that exists now, and maybe never existed for Linux

Search engines are extremely modular and Unix-y. You have a bunch of indexed corpora and you intermingle them at ranking time, with respect to a query. But unfortunately there is no real incentive to provide something that has measurably good results and is also open to your own data and modifications

The incentive is to make a walled garden out of it

Try Docfetcher https://docfetcher.sourceforge.io/en/index.html

State of desktop search is very bad but this is what I use. It is of acceptable quality.

> Now I just need some kind of open source search engine to run on it ...

Here you go: https://yacy.net

Same here, but in my notes app. Been doing it for years.

And if I may expand a little, not just for crappy Google, I also create alternative local knowledge bases at work.

I can't find anything at work. Everything is spread out across chat, Wikis, SharePoint, email. All having different owners, content may at any time disappear or move, there's constant authorization headaches.

Whenever I come across something useful that I expect to be of some future use, I make a local copy. File, web page, wiki, anything. Because our information systems are a massive failure.

I think this is typical of most work places. Our ability to maintain good knowledge bases has essentially collapsed. Every coding task at work I basically start with reverse-engineering the system because no one really knows how it all works anymore.

I was about to ask you to share your list, but that got me wondering: is there any tooling for curating, sharing, and most importantly, consolidating curated lists of sites (based on tags rather than categories), such that the consolidated list is then searchable?

This is basically what I use my Wikipedia User Page for. Keeping links to all the news websites I happen to have found reasonably interesting content at. Stories I like that Wikipedia will probably not accept. Articles that I want to keep, yet might not be accepted for an article, or I have little faith they'll remain in the article. Probably just need to dump my bookmarks in occasionally.

Honest question, how much of this is simply due to Google slowing showing more & more ads on Page 1 of search results?

There use to be a time when paid placement was only 1-2 results.

It’s frequent now that the top 5-6 results are paid placement.

(And when I’m doing a search for a specific product I know I want, competitors are bidding up those search terms which is annoying because I’m being shown not what I’m explicitly searching for)

I have a hypothesis that a lot of this is the result of hyper focusing on short term reward. Just think about how we measure a top exec's performance. It is often dependent on cutting costs and increasing revenue. Cycle through that a bit and if the previous person did their job well then they cut a lot of fat. In fact, they probably cut as much as they thought they could get away with. So next person comes in and they gotta start cutting more than fat. Of course you could go other avenues to increase revenue but cost cutting measures are the easiest and quickest.

from a technical aspect, I'd bet their ranking algorithm now just favors ad placement , full stop.

it's largely just a product of ourouboros

Kagi just renders those toxic sites into a grouping called "Listicles" which I then ignore. It's far from perfect, but it's clear that a company with far less money and access than Google doesn't find this an impossible problem to address.

So I would suggest that Google knows what it's doing, it just makes them money.

Their bread and butter is advertising, not search

Also, malicious content farms pay Google (to show their ads), you don't. So for Google, they are the customer they need to serve, not you.

Well done on putting this far more succinctly than I could have. You’ve absolutely nailed the top three issues with Google search in a nutshell.

Thanks, I might as well keep going then :)

Google has commercialized a huge amount of search terms. I'll use biology as an example. You search for particular species and search results prioritize products that kill the species. You search for a particular plant and you'll have a hard time learning about the species as it only shows cultivated versions and products related to how to care for them.

Pure information/knowledge for the sake of learning and curiosity is de-prioritized.

Back to image search, it's unable to figure out original sources or doesn't care. Pinterest is the well known manifestation of that.

Google shopping results: completely broken. Click through on the products and half the time the price, availability, discounts and stock do not match.

Everything is so goddamn broken, and nobody at Google seems to care. I can't explain it, but it's been going on for a good 6-7 years or so.

I'll explain it. Giving worse results sells more pay per click ads.

I've seen this in many search terms. Purposefully push down the things that people are looking for and they will click on ads.

Those sites that have a ton of extra info? Many of them also sell in content ads right?

The ad machine took over long ago and you are forced to play the ad auction game if you want to show up in results, and if people are looking for something they are forced to scroll past a lot of ads.

I watched this happen in chunks over time. It wasn't always like this, but it is now, and it's been a slow creep over time.

It's because Google has never been focussed on search quality imo. No search engine produces high quality results anymore.

Especially since you get the clear spam sites that somehow reference your query in the page content (where they've just spammed loads of keywords, but also pretty sure some spam sites are doing something dynamic with it).

never???

There are people younger than Google here, so might as well have been never to them.

Of course content farms rank highly on Google, they're filled with Google ads, so why would Google demote what makes money?

Google has maximized advertisement $ and that's all.

We're all technically minded here but very few people really understand how technical choices add up to greater detriments.

and that's today's Google. they minimized the index and maximized the searches that yield profit through Google ads. those websites you hate? they monetize Google ad words.

>You search for "red flower Thailand" and flowers of various other colors from various locations appear.

https://imgur.com/a/sJCECzQ

Looks mostly red to me. A little pink too I guess.

FYI, you read the parent far too literally. I think most interpret the example as an off hand illustrative example, not a literal one.

the one that pisses me off most is installing/configuring a software package... Top articles always end up being "apt-get install foo" and never address the configuration at all.

ChatGPT 4 is fantastic for this kind of questions. No Google spam to filter out, and you can ask follow-up questions on topics such as configuration.

Hallucination and inaccuracy is a downside, but even so it is often still better.

oh? I haven't actually done much with all the A.I. except co-pilot. I'll have to give it a try.

Thanks!

I skim from the bottom up now

> We find that only a small portion of product reviews on the web uses affiliate marketing, but the majority of all search results do. [...] We further observe an inverse relationship between affiliate marketing use and content complexity, and that all search engines fall victim to large-scale affiliate link spam campaigns.

I think this is an excellent methodology for testing the quality of search results. I would love to see a standard search engine test and scoring system based on this, maybe similar to some of the LLM scoring systems.

This doesn't apply to the content complexity finding, but the finding that "product reviews which are in top search results are more likely to contain affiliate links than product reviews which are not" can also be explained by the fact that if and only I am getting a bunch of hits on my product reviews, I'm incentivized to monetize that with affiliate links.

We should also ask ourselves if affiliate links are really that bad. Someone could be making honest complete reviews and monetizing those with affiliate links, does that inherently mean that the search results are lower quality?

That approach also misses all the copied-a-github-issue low-effort content that seem to crop up on Google.

Anecdotally, I can say that I’ve definitely noticed a negative correlation between affiliate marketing and content quality, though I would be very interested in seeing a formal study of it.

Affiliate links create misaligned incentives between content creators and consumers. This includes on the products listed themselves, where the affiliate is incentivized to select the products that give the most kickback rather than the ‘best’ for the consumer. But it also includes the content itself. Affiliate links create incentive to (a) write reviews, where a site wouldn’t have bothered before; (b) churn out lots of content with lots of links, that can be picked up by search engines; and (c) to not invest much in the actual reviews, and instead generate quick, low-quality content, since the content creator doesn’t actually care about finding the “best” product (which takes lots of time and money to do correctly), only on having readers click their links.

This is why sites with names like Celeb Rumor Central have dozens of articles like “top 10 coffee makers 2024”. They just hire a freelancer to do a few quick Google searches for coffee makers, then churn out 10,000 word articles with sections like “the history of coffee” and “why do people drink coffee?” Then the actual review is “this machine is purported to boil water and drip it through coffee grounds, and has high ratings on Amazon (we may earn a small commission when you click a link on our site)”. Increasingly, they’re just using AI to do it, cutting out the cost of a freelancer.

In my personal opinion, affiliate marketing is one of the worst things to happen to the modern web, and the source of a ton of content farm SEO spam pages.

There is a lot of money to be captured in paid search with these sites. There are some pretty large players in the space who just built these types of top 5 or top 10 pages for tons of verticals and then bid on keywords related to those verticals. They can go to brands and demand significantly higher fees than most affiliates would get and the brands will usually pay because that acquisition is usually more efficient than try to bid on the terms the affiliate is bidding on.

Back when I was still involved in acquisition marketing, I did a test where I killed all paid search and built relationships with the affiliates to negotiate our "ranking" higher up the page. It was a huge boom for business. We scaled paid acquisition at a profitable CAC, which was very difficult to maintain, let alone scale, in bidding directly and it was significantly less work to manage.

Consumers value this type of content quite a bit, even if many are skeptical of the quality. Sometimes it's nice to just see a pared down list of things with even a cursory rundown of features/differences.

Sure but I feel like you are taking game theory and applying it to real life. See for example https://www.rtings.com/monitor/reviews/best/by-usage/gaming they do a lot of actual reviews with tests and all their links are affiliated. If as a Google user I want a review of Gaming Monitor, having this as the first link would likely be an acceptable-to-good scenario.

It's bad because most of the reviews will be for items you can only get on Amazon or another retailer with an affiliate link program and not for items available elsewhere.

I've seen so many "Best $WHATEVER of $YEAR" and it's really "Best $WHATEVER $PRODUCED_BY_WEIRDLY_NAME_FLY_BY_NIGHT_COMPANY_ON_AMAZON $YEAR"

In theory, if we had truly open marketplaces that offered equal monetization of all links to all similar products in the same category regardless of the price offered to the consumer, sure.

In practice, that is never the case. At best you're getting reviews that only compare products available in the same marketplace (say, for example, all the table saws you can buy on Amazon). At worst, you're getting reviews from vendors who offer the highest payout.

Also, a heck of a lot of consumer products are absolute garbage. (IMHO, most of them, but others may feel differently.) Who is going to write an honest, scathing review of a product and then monetize the link to it? Why even bother?

How can an affiliate link not be in conflict with an unbiased review? They don't get paid if they don't recommend the thing.

> that inherently mean that the search results are lower quality?

Almost? There doesn't need to be anything nefarious going on, but human beings align to incentives, basically always. So the trick with reviews is going to be to avoid any incentives coming from the product side, and embrace those coming from the consumer side.

Forgive my naivety, but wouldn't a simple way for a search engine (like Kagi) to avoid falling victim here to detect affiliate link programs? There's got to be a small handful of patterns for affiliate link tracking:

1. Domain Interception & HTTP redirects 2. Tracking codes embedded in the URL directly

It looks like Kagi does this[1]:

> Kagi surfaces shopping results featuring unbiased reviews and no affiliate links to help you identify the best product across categories. Top results include discussions focused on helping you find the best item to purchase - you are not bombarded with affiliate links and ads. Continue to scroll and you will see product comparisons across multiple vendors so you can pick what best suites you. Kagi's shopping search will always return a detailed discussion of which product to buy not a competition amongst advertisers promoting where you should buy. Kagi is focused on providing you the best results to make an informed decision not polluted by affiliate links and advertisements.

1. https://help.kagi.com/kagi/features/shopping.html

I tried a search and found a site with amazon affiliate links in their featured "Shopping" snippet

https://i.imgur.com/vPRpBYJ.mp4 (yes the recording is very broken but you can still see)

Maybe the affiliate stuff is still there but ranked lower?

Sure but it sounds to me like it shouldn't be featured in the "Shopping" widget when it has 9 affiliate links?

When I hover over the "i" icon it says:

> Information provided by Looria, Amazon, Reddit and other sources. Shop confidently with Kagi - our results have no affiliate links.

The sites featured in the "Shopping" widget were:

- Reddit

- RTINGS: has 8 affiliate links (just from one article I picked, it was featured several times)

- NYTimes: has 47 affiliate links (just from one article I picked, it was featured several times)

- Techgearlab: has 9 affiliate links

Even if you ignored the widget, the first result was a site with 7 amazon affiliate links.

On second read, I think what the Kagi doc means is that they don't provide first-party affiliate links (i.e. if you click on an Amazon.com search result it's not using their affiliate). They will still show you search results that have affiliate links embedded (but presumably don't affect or negatively affect ranking).

Kagi is better than Google, but just yesterday it returned the top link to Amazon-linking “check the price for this item on Amazon right now” affiliate spam. So it’s not perfect.

Oh, OK. I thought it was perfect.

I think it’s pretty good. I understand sometimes we root for the good guys like Kagi and look at their work with rose tinted glasses and even think they’re perfect, like you did, but it’s not really feasible.

A pattern I’ve noticed from people not working in tech, or more junior people in tech, is that they have a thinking that software can ever be perfect. If either of those categories describe you, I can tell you from experience no software is ever perfect!

In adversarial environments, this is true. In non-adversarial environments, software of a certain defined scope can at least get very close to perfection.

Ultimately a site can get around affiliate / referral detection with internal links that redirect though I suppose. So the crawler would need to follow and track all links to detect this. Also sites will surface different links / content based on user agents, so I don't know how effective this is.

Maybe there is room for one search engine that does this - for paying customers or as an investment to be better

Kagi is a paid search engine to avoid the manipulation that Google welcomes. To avoid every trick and targeted tricks it may be cost prohibitive. Fortunately Kagi isn't popular enough yet for targeted manipulations to have a noticeable impact.

Currently, Kagi has (if you hover/click the shield icon to the right of a result) an indication of the information it knows about a website (as well as a way for you to rank it higher or lower for yourself).

One of these is "ads/trackers". I imagine that it would be feasible for this to include some of the more common affiliate URL types, or third party lead/affiliate tracking bounce hops like awin.

Clearly there will always be some amount of ability to "defeat" this kind of measure by obfuscating links, but eventually the user needs forwarded to a link that has a referral parameter or a site that sets an affiliate cookie or similar.

The "tracker category" also can give a bit of extra information - things like "invasive fingerprinting, advertising"

I am working on a search engine that checks a site for affiliate links from known providers and demotes sites that have a large amount of them.

In addition:

* it demotes sites with popups (think newsletter sign ups)

* it demotes sites that block (or complain about) ad blockers

* it demotes sites with a high number of ads and favors sites with no ads

* it demotes sites using certain sketchy ad companies.

* It demotes sites that have paywalls

* It detects possible link networks and flags them for human review/removal.

* sites with RSS feeds get promoted.

* There is a toggle to hide all sites with ads or external trackers, but it is still WIP (The whole project is).

There are many other features. No idea if I am going to make it public, I created it to update my skillset. I actually thought about setting up a nonprofit and making it open source, but I haven’t decided.

Very interested, any way to follow you?

Me too.

I don't know if this would be a very long-term solution if the big ones (ok, google) did it. Advertisers would catch on very quickly, and some legitimate review sites which might get funding through affiliate links unrelated to the product being reviewed would lose out to straight-up paid for "reviews" that are funded wholly by the manufacturer and just don't use affiliate links.

What would be the difference between an affiliate link program and a web ring?

I don't know if discovery is actually a bottleneck to be automated away. It might be the fun part. I'm thinking back to the Napster approach where you could browse other people's libraries for music ideas.

this is one way to do it, but I wouldn't say its sufficient. If I search for 'things to do in Seattle', you get many 'blogs' and such that a writer gets paid by sources to insert their place into the things to do list. I didn't word that well, so for example: I own a coffee shop, I pay them moneys, and the '25 things to do in Seattle' writer puts my coffee shop in the list.

If I do an image search for the word 'strawberry', how many of those results are not stock images, images from a store, etc. of a strawberry? can you find an actual picture of a strawberry sitting in the wild? or just some picture of a strawberry a person uploaded without trying to sell you something?

Then the search engines will game the benchmarks. A different definition of "Search Engine Optimization" I suppose.

I know that tends to happen for LLMs but I don't know if Google/MS would care enough about some obscure benchmark system to try to game it.

Their entire business is gaming metrics. Most of those metrics are ones they invented for themselves, but if there's any advantage at all, they'll game it

I don't know, feels like a paper titled "Is Google Getting Worse" could have benefited from actually looking at Google results rather than only results of other search engines.

Edit: This got downvoted to hell, so let me be more explicit. This study did not look at Google results, the title is pure clickbait. They used Startpage results as a proxy for Google results. I don't think that's a valid assumption, even if Startpage is using Google's index.

Using a reasonable proxy is not "pure clickbait", though it may be misleading. It adds an additional assumption about the Google results not being tampered with by Startpage, but it seems like a reasonable one, compared to the alternative of Google doctoring the search results when detecting a scraper and/or doing personalization.

If the authors have done their due diligence and confirmed the results from Startpage are actually Google results, then I don't see why they couldn't claim so in their title.

The authors did not do any work to establish it as a reasonable proxy. We know that's the case, because they don't describe any of that work in the paper. they just state that they're using this proxy measure, and then seem to be using Startpage and Google interchangably for the rest of the paper. Sure, if they'd actually done the searches on Google, the title would not be outright dishonest. But they didn't, so this defense of yours is just absurd.

It's pretty obvious why the results won't be the same: the feature sets of the search engines are different despite being based on the same index. Startpage's own documentation even has a page on how and why the results are different!

(Personalization is a part of the Google feature set, and has to be taken into account when considering the results. It's also a part of the Bing feature set. This didn't stop them from reporting the results of their testing on Bing in detail.)

Hi, I'm lead author of the study, so I can give some background on this. This is certainly valid criticism and we could have addressed our reasoning for using Startpage in more detail, but it just didn't fit in the paper anymore.

The reason we used Startpage is simply that it's much easier to scrape. We started off checking only Startpage (as proxy for Google) and DuckDuckGo (as proxy for Bing), since they are both simple to scrape and produce stable rankings. Plain Bing, on the other hand, is a lot trickier. You often get different results for the same query and you get blocked much more easily if you send too many. The only stable way to use Bing is via the (certainly not very cheap) API, though even that wouldn't necessarily guarantee the same user experience as the web frontend.

We did notice, however, that DDG (despite being mostly Bing) did deviate quite a bit in their results, so we started scraping Bing as well for a fairer comparison. As for Startpage, we did check it initially and we found the results to be virtually identical, except for a few minor rank differences here and there (which are probably just geo personalisation). The differences may have become larger now that Startpage also taps Bing to a certain amount, though when I do spot checks, the results are still sufficiently similar and they are also sufficiently different from what actual Bing gives you. Most prominently, Startpage/Google give you a lot more YouTube results, which we did another small spin-off study on (to be published at CHIIR this year https://downloads.webis.de/publications/papers/bevendorff_20...). Moreover, we could also measure certain immediate effects of Google's ranker updates in Startpage, which weren't as apparent in Bing. So we are confident that Startpage is a reasonable proxy for Google, though it's certainly something we will keep in mind for follow-up studies.

Hey,

I'm sure that using these sites was the thing for your research for all kinds of practical reasons, and you don't need to justify that :) My actual complaint was about the misleading title, which you did not address.

Note that the clickbait title actually hijacked your research; there's like three comments out of 240 here that are about your paper. So while it probably feels good to have received all this attention, it's just bogus engagement. I guess it's pretty meta for a paper about search result spam to do blackhat social media engagement optimization though.

I wouldn't say it's misleading or wrong. It's certainly a bit catchy, but it also aligns with a greater discussion that's going on at the moment. Regardless of the title, most people probably wouldn't or couldn't discuss scientific literature in-depth anyway, but search engine spam is something many can relate to, be it from Google or otherwise. It's certainly a topic we will be hearing about a lot more in the future. Besides, we didn't even anticipate this to be picked up by non-scientific journals that quickly, though the title may have helped. ;-)

Yes. Google used to be amazing, then it turned into an advertisement company. Slowly at first, then about a decade ago the pace increased.

But the worst part is, Google SEO has infected the entire web and made it into complete garbage. Hopefully, this last decade or so will just be a blip before we return to baseline, where it can be wild and free again.

Are there stats* of Google Search across the years? I felt I don't use Google as much as I used to. And it isn't because "I know more stuff", but mainly because the way we use internet has changed. I wonder if kids or teens (who most of them don't know how to use an email inbox) would use Google... (I guess yeah?)

* Of course, the stats should include the total amount of internet users globally, or normalize the amount of searches based on that...

I've been in web development and SEO for almost 20 years now.

When I first started out all the veterans of SEO kept telling me not to do this, don't do that with things that could get your site buried in the SERP's. At the time Google's algorithm was really good at ferreting out affiliate links, link farms and other nefarious black hat techniques SEO's used to game Google.

Now? Complete opposite. I have several freelance clients and I've used every dirty SEO trick in the book and all of them have worked like magic to get my clients sites ranked on page 1 or 2 of the SERP's.

I have no idea what changed, but Google is super easy to manipulate now to get your site or specific pages ranking really high. I haven't heard or seen any of the horror stories I read and people blogged about constantly when I first started out for years - which tells me they're all probably doing the same thing I am and not seeing any repercussions.

Maybe Google doesn't care because users have become so savvy, they can filter through a ton of garbage in minutes to find what they really want?

Do you have any writing on this? Would love to learn more.

I recently googled "rubbish removal mysuburb" and found a pretty good looking website. Let's say it was myrubbishremoval.com/mysuburb. The page text repeatedly named the suburb.

It turns out they appear to have a folder for basically every suburb in the country, thousands of subdirectories yielding the same looking page with that suburb name inserted into the text, making them look local no matter where you are. I can similarly find this with many other industries.

This sort of thing twenty years ago was a huge no no. Google talked about detecting mostly duplicate text and would bury you for it. Like a very basic rule was that if two pages were mostly the same only one would really rank.

Nowadays the "lead funnel" is an entire industry on websites following this pattern, and it clearly works.

Interesting because I've been under the assumption that this kind of "keyword stuffing" was cause issues with ranking so I've always wondered what some of these sites were doing "right"

Definitely I think the way we use the internet has changed profoundly. There are a lot of apps that provide useful information but they may not be made indexable by search engines. Much less useful information is simply out there on the open web, and much of it are locked behind logins. There were previous deals like Twitter sending a completely copy of all new tweets to Google, but these are basically dying.

It's especially interesting since you mentioned normalizing searches by the number of internet users. The country with the largest number of internet users is China, with more than 1 billion of them. And they don't have access to Google. And their local copycat, Baidu, is years behind Google in terms of technological sophistication and simultaneously years ahead of Google in terms of user hostility. So what do Internet users in China do in a post-search world? They simply open various apps and use the full text search feature of different apps. For general knowledge they might open ZhiHu and search there; for something resembling the old-time personal blogs by individual users they might open XiaoHongShu and search there; for short videos they might open Douyin and for long ones Bilibili. For reaching an organization be it a store or a museum or a hospital or a government department they might open WeChat and search there for an official account or mini program (a mini program is a website that uses WeChat APIs and can only be opened in WeChat).

I made these observations on a recent trip to China and it's clear to me what a post-search world looks like because China is already there.

> And their local copycat, Baidu, is years behind Google in terms of technological sophistication and simultaneously years ahead of Google in terms of user hostility.

Baidu search was fine during early days, issue as you hinted was PRC internet went mobile first and content got locked behind various platforms and made deliberately hard to scrape - crawling/indexing got locked much earlier than west. Hence now as more gets locked in west behind logins, western behaviours also shifting towards that model, how much of default search is query + wiki/reddit/youtube or straight into short video services like looking up recipes on XiaoHongShu. Reddit especially, simply because reddit app has horrible search experience. Also technically, Baidu rankdex predated Google PageRank which Larry Page referenced for Pagerank patents. Eitherway, depending on how ChatGPT copyright drama plays out, imo more people will just go the lazy route and have AI generate good enough summaries for most queries.

>I made these observations on a recent trip to China and it's clear to me what a post-search world looks like because China is already there.

You are talking like open web is dead but it's not. There are millions of blogs and personal sites out there. Walled gardens are user hostile and hungry for money, that's why enshittification[0] happens.

[0] https://en.wikipedia.org/wiki/Enshittification

The open web is not dead. Unfortunately it will be soon, where "soon" is roughly a decade or so. If you haven't seen the signs of it dying a slow death, you've been hiding under a rock. I totally agree with you about walled gardens and enshittification; but I see no way to stop them.

You have been using Google less.

Quick: If you want to search on what is the best toaster to buy, what URL do you type?

How do you find out the weather?

How do you find a local brewery in your area?

For me the first answer is Amazon. And the second is ask Siri. The third is Apple Maps.

Google is now below 50% of my search terms. Heck, I’m more likely to search Reddit for some queries.

Long-term stats are tricky because of how much of the landscape has changed. There's the desktop/mobile split, developing countries increasing their internet use, heavier use of apps, growth and decline of results getting indexed, and change in what we search for.

It's not worse when you append site:reddit.com to every single search but this is only a function of the fact that reddit can't figure out how to build their own search. Outside of maybe programming stuff where I'll still click on links, I don't think google has driven me organically to a new site in years.

> It's not worse when you append site:reddit.com to every single search

Could you give some examples of search queries that would benefit from filtering by reddit?

(My own example: I've been looking for recommendations for a solid Linux laptop. A good result would be a list of reviews written from personal experience of owning such laptops. Reddit was useless for that.)

"Armoured core 6 can't beat pca enforcer walton"

People actually discussing how they did this is the only real answer, generic websites with pictures and descriptions of his weapons never help.

Kagi allows users to adjust the ranking [0] of certain domains. I'm not sure if that's shared among all users but it would certainly help.

[0] https://blog.kagi.com/kagi-features

Fellow kagi user here. I have been very happy with the service so far. Switched over to it when they announced their new pricing model a little while ago. It is more than worth the cost. Even without tweaking, the quality of results is far higher than what I had gotten used to getting from DDG or DDG with !g. The ability to rank domains is amazing though.

Kagi has improved immensely and I no longer ever feel the need to check Google against it. It's a great product that I am more than happy to pay $10/mo

I'm so happy that their new pricing scheme has done away with search quotas. Their claim that the average user does 100 searches a month seems absurdly low. I would blow through my 300 search quota in a few days.

I average about 60/day since the pricing update. Fewer than I expected, but enough to hit the quota in under a week.

Every time I see someone talk about this feature on HN (almost every day) I get jealous, it's such a good idea. When I'm searching (on google) I'm always thinking; when I get Kagi I'm gonna downrank this site, uprank that one etc.

I'm curious, why haven't you switched to Kagi yet?

For me… I don’t want my search tied to my credit card. I go through lengths to keep my digital and analog domains as separate as possible.

> don’t want my search tied to my credit card

Pay with Bitcoin [1].

[1] https://blog.kagi.com/accepting-paypal-bitcoin

Bitcoin is such a bad payment method, even with Lightning network, because you need to buy a volatile asset and payment processors typically charge huge fees to receive BTC payments.

If Kagi supported USDC I would definitely use it.

Could you use a prepaid debit card or is that? And what other search engine do you use and what makes you think they don't know who you are?

I primarily use DuckDuckGo go. I use account containers. I use a VPN.

I’m not letting perfect be the enemy of good.

They don’t keep a record of your searches, so technically your searches aren’t connected to your card.

How subtle is the wink supposed to be in their or your statement?

I didn't mean it in an ironic way if that's what you're referring to. I interpreted your statement to say "I don't want my search history connected to my credit card.", sorry if I misunderstood what you said.

If what you mean is you don't believe they are not connected to your account, I can point you to the FAQ entry on their website.

https://help.kagi.com/kagi/faq/faq.html#what-data-does-kagi-...

Ah, right… Well, if it’s on their website!

Good idea indeed, though you could block some sites (on any search engine) with a browser script

It's not shared, but they have a list of most blocked etc. domains: https://kagi.com/stats

That is an interesting list; I'm not at all surprised to see pinterest holding the top ~10 spots, but things like facebook, instagram and twitter being so high does surprise me; perhaps it's indicative of the type of user they have (I too block those domains).

I could take that list, as-is and use it as a block list on my entire network.

I mean when you look at all the raised and pinned domains it's almost entirely programming related topics so "programmers dislike sites that have terrible browser experiences for unauth'd users" is like the least surprising thing.

That's a very good list. As a programmer I wish many of those sites, like geeks for geeks, would disappear forever.

That is a brutally small user base given how much I hear about them on HN.

It's a paid search engine, and not a cheap one. Not exactly mass market material ;) and I'd assume the percentage of HN users is rather high, hence a lot of comments from us, here.

@vlad I got 429’d here

Also a kagi user since they changed their pricing. The results have been decent for me for the most part, except for shopping which I still use google for. I do wish they’d step up their css game though. Maybe it’s just safari on iOS, but I get style problems constantly. The address bar doesn’t behave properly with the cursor if you type in a decently long query, and so often the search results will overflow the container horizontally. It’s a small price to pay for good search results, but stuff like that should be table stakes for a tool like this.

Can you open an issue on kagifeedback.org with details?

I get the reason why Google does not do domain level filtering due to antitrust concerns, but at least they should be able to adopt this kind of user preference approach. Why aren't they doing this years ago?

They were doing it years (more than a decade) ago, but the feature was eventually removed.

If it’s purely user generated and not something with any defaults, why would they be subject to antitrust concerns?

ublacklist does this for Google.

Quite likely underreporting affiliate links due to obfuscation like cloaking, hiding redirects behind javascript (they mention in the paper not rendering the page), using JS and a POST, other URL minifiers etc.

One interesting solution to the problem is to have more than one dominant search engine and its algorithmic choices, having half a dozen web-scale engines with some variation at least gives the user a choice into other avenues of information discovery. (There isn't really much point in using Startpage and DDG here since they're effectively meta search engines of Google and Bing). For SEOs in English speaking countries there is not much point in thinking beyond pleasing Google.

Clearly AI and whack-a-mole spam sites have been a problem for a while due to the prevalance of people tacking on 'reddit' to their query to find other humans talking about stuff.

It seems that the answer from their study is mostly no (reading the conclusion), but they seem reluctant to admit it, so they focus mostly on results being mediocre and spammy.

If the answer was yes, the title wouldn't be phrased as a question

I visited a small country in my last vacation. I ended up bringing a bit of money back because I was in a hurry I thought it would be easy to exchange back at home, even at worse rates. I live in a big city after all. Of course, I was very wrong.

I spent an afternoon Googling every possible incantation only to get useless AI generated text, travel agency sites or simply unrelated content.

I was about to accept my loss when I tried Kagi. The first page showed an exchange that accepted the currency. Very far from me and with terrible rates, but still.

Anecdata and all, but the fact is that I'm using Kagi more and more and it's winning my trust and good will fast.

As an aside, many full service banks can exchange foreign currency for you. For example, Bank of America does (https://www.bankofamerica.com/foreign-exchange/exchange-rate...). You can also order foreign currency prior to your trip and they'll mail it to you.

Interestingly in my country many full service banks have dropped foriegn currency services. They did that during the panademic but never returned back to offering that again. I suspect they make so much money from Visa/Mastercard commissions that they'd prefer people paid with card rather than cash.

Yeah, a decade or so ago I’m sure I could have found one much easier. The only thing widely accepted is dollars and euros, now.

Right, it’s the same here. I learned that the hard way. If I ever need cash for traveling again, this is the way. I much rather use credit card if I can, however. But this place didn’t even had cell phone signal, so accepting anything other than cash was impossible.

What many people do in this case is to keep the money for the next time you visit the country. (or as souvenirs)

It would have been an expensive souvenir. And I don’t travel all that much. Also, inflation and even outright currency replacements are somewhat common in smaller economies.

This sounds like an ad for Kagi, though.

Why would Kagi pay someone to promote them on HN when there's already dozens of praise everyday here?

Even as a happy customer, I can see how suspect this praise can be, but remember that HN is exactly who Kagi was created for. And I don't remember having spend $10 a month on something that had that much impact on my daily life. It's _just_ a search engine, but I do about 50 search a day, and having great results each time while the other one is giving me more and more garbage results is huge. So I completely understand why people satisfied with Kagi can be noisy about how great they think it is.

You mean my comment? It’s not. I have no relation with them other than a free account.

I think 2005-2010 Google was peak web search. It would have given me an obscure blog with the top 5 currency exchange offices. Kagi gave me one single crappy result, but it was still, sadly, better than what 2024 Google can do.

Anecdotally, yes.

Google search for topics I'm unfamiliar with/wanting to learn about all lead to low quality, SEO-optimized to hell, clik-baity sites that are just riddled with ads. I have to add "reddit" to most searches just to find semi-relevant content.

But Google search for topics i'm super familiar with and just need a transactional search to look something up tend to be much better and generally the fastest way to accomplish a task.

Right, adding "discussion", "forum" and even "reddit" to the search term increases quality dramatically. Also, Google is still great to search websites. I gave up on Stackoverflow's own search but add "site:stackoverflow.com" to my searches on google.

BTW, maybe someone wants to create a very simple webpage with a search mask that allows adding a few (customizable) terms and options and simply forwards that to Google's search when pressing enter.

I've started using uBlockList to block those sites and it does clean it up quite a bit. I assume those sites also work for some, but they're usually of low quality to me so I block them.

The second type of searches you describe seem to be better so far, but I've stumbled upon a bunch of obviously generated garbage recently. So not sure how long it'll hold.

I've tried things like DDG, YaCy, Bing and others, but often Google is just significantly better (but not necessarily good).

Yes, Google is getting worse, and was already abysmal, from some years back.

In longitude, latitude, and by many other measures :)

Plenty of earlier posts and comments about that on HN, for many years now. What's so surprising or new about that, then?

As the saying goes, it's news if a man bites a dog, but not the other way around - doggone it if I know why, man ...

Consider using ublacklist[1]. I don't see SEO junk like Pinterest, Tutorialspoint, Javatpoint, geeksforgeeks, etc at all in my searches.

[1]: https://github.com/iorate/ublacklist

I didn't know this exists, thanks. Every time I google anything programming-related looking for the docs, and the official docs are buried under a pile of SEO shit, I die inside a little.

This is particularly egregious with Python, and I suppose it must be just as bad or worse in the JS ecosystem.

And here are some lists for ublacklist that can help you use it easily [1]

[1] https://github.com/rjaus/awesome-ublacklist

We're back to the pre-Google era and suddenly I'm aggressively bookmarking stuff like it's 1995.

I recommend not just bookmarking, but downloading a copy. Either a webarchive file or manually extract the main content from there (Reader mode in browsers is great help). Link rots happen. More than half of my bookmarks don't work.

Yes!!! I've been playing around with SingleFile and other similar software to try and hoard as much useful data as I can. And I've been mocking preppers for all these years.

An acquaintance of my second cousin has experimented with SingleFile CLI (https://github.com/gildas-lormeau/single-file-cli) and she reports it works great (it is rather heavyweight because it does invoke headless Chromium or even Firefox if you set it up right).

it also never hurts to throw the URL into archive.org. This will archive it for everyone else too

On the other hand it is not 1995. Time has moved on. I wrote a Simple RSS feed, that also serves as search engine for bookmarks.

I am able to run it in attick on raspberry pi. We do not have to rely so heavily on google.

https://github.com/rumca-js/Django-link-archive

It is true that it does not serve me as google, or kagi replacement. It is a very nice addition though.

With a little bit off determination I do not have to be so dependent on google.

Here is also a dump of known domains. Some are personal.

https://github.com/rumca-js/Internet-Places-Database

...and my bookmarks

https://github.com/rumca-js/RSS-Link-Database

Some more years, and google can go to hell.

I wonder if "affiliate links" is a reasonable proxy for page quality? I wouldn't use it directly in a ranking function but it might be a nice automated way to estimate whether results are good.

In a good way or bad way? Affiliate links might bias reviews, but their presence might mean the review was high enough quality that enough people see it and click on affiliate links.

I mean the presence of affiliate links indicates the page is likely spam. I bet that's a good heuristic more often than it's a bad one.

I don't know if the overall search quality has degraded or not, but SEO has been become definitely a much more severe issue than ever before. Google is the main target for this attack for obvious reasons but no other search engines are really immune to this. I'm skeptical if this can be tackled by any technical solutions; the problem is not just a specific type of SEO spamming but the structure where the enemies are constantly optimizing against your fundamental goal.

    but the structure where the enemies are constantly optimizing against your fundamental goal.

Like companies avoiding taxes.

Google is better than Bing and anything else for my day today work. Few things I miss though :

1. Spam is more than in past. The outrage porn, clickbait headlines etc. are lot more than in past.

2. Dominance of few domains despite poor quality content. For lot of coding related queries, dev.to, hashnode etc. appear in top results despite being clearly spammy.

3. Paywalled content. Most irritating part is sites like medium which appear in top results, have high value content and yet are behind paywall.

Internet is growing and so are Google's problems but I think they are still on top of things.

This paper does not make sense to me. They used Startpage (https://www.startpage.com) as a proxy for Google results ("result pages of Google (by proxy of Startpage)" in a discussion of Google results?

Except, if you put the same search in Startpage and Google, you get different results. Image results especially are quite different. Text results were mostly just a reorganization on my quick tests. (Tried the title of the paper as its own search "Is Google Getting Worse? A Longitudinal Investigation of SEO Spam in Search")

Edit: One other notable result is from Figure 3, that a huge percentage of the results now are Amazon and Youtube. Many orders of magnitude in most cases. Amazon (3000-4000), NYTimes (1000), Walmart (~500?), Insider/PCMag/Tomsguide (~50?)

Since google search is highly personalized and localized, this may not matter, and may be why you see different results.

> Startpage delivers Google search results via our proprietary personal data protection technology.

I knew I wasn't the only one thinking this lol - glad it's proven by an investigation How does one think Google is planning to change it's ranking system?

I have a few questions for Google employees..

Are you allowed to search the internet with an ad blocker installed?

Do you use a special search interface that doesn't return results with ads?

It is. My suspicion is that they are using ai for search results considering how inreliable it's become. That's why searching for "sinus inflammation" used to yield results for penis inflammation for a while. That one seems to have been resolved. However I am still getting completely unrelated results every now and then, indicating they are having serious unresolved search engine issues.

Google's search quality has been going down the drain for years. I get more and more fake wikipedia and stack overflow SEO results and I use it less and less. Can't wait until some search oriented LLM company replaces it.

I wonder what percent of wiki entries are being taken over by AI writing. I feel like I wouldn't know with wiki style language. There was at least a few TV tropes entries that I suspected, only because the humor and observation felt off.

To me it feels like they prepend every search query with “where to buy”. Or when I ask for more specific things, like “cheap used Linux laptop”, it goes like naaaah, surely you mean “best laptops you can buy in 2024”.

As long as Pinterest keeps coming up on my search results, the answer is yes!

well Google NEWS certainly seems to be absolutely OWNED by spammers of late.

eg just yesterday - a search for the NYSE listed company "betmgm" on google news [US] yielded 100 spam results [affiliate offers + bonus codes for BETMGM sportsbook]- and not one real non-commercial, non-spam news post concerning the company.

[data: https://gamblingindustrynews.com/news/affiliates/google-news... ]

I am not sure if Google has empirically gotten worse or not, but I do know that I very often have to use multiple search engines more often than not now in order to find the information I am looking for.

For me, the biggest quality drop literally happened overnight. It was when they removed the PLUS operator and any other changes that they made that same day. There was many other small changes that made it worst, but that's nothing in comparison. I think that was the day Google became evil.

The worst thing Google allow phishing sites in sponsors. There are many cases of people searching for a website and lands on the phishing site instead, getting credit card details stolen.

Okay. That's it. I've f'n had it with search sucking. Reading these comments is just demoralizing. We don't have to live like this, and we won't.

Stay tuned here on HN.

>The study finds an inverse relationship between a page’s optimization level and its perceived expertise

What if Google flipped its SEO weights from positive to negative?

I wish I could access old Google again, where the results made sense and searching wasn’t infiltrated with AI.

Google is a clear case for why you should pick a service because of it's business model.

Oh, what a methodology. They ran with the meme of "best pants" having bad results, when the problem with "best pants" is that it's a nonsense query. I personally cannot recall ever doing a "best pants" query, and the results of such queries don't factor into whether I think a search engine is good.

Today I searched for a product, google found a local site with the item, I clicked on the link and I was redirected to aliexpress. How they can not catch this?

Try going to the URL directly, and try curling the URL with a referrer of Google.

There's a thing that's been happening the last year or two where compromised web servers serve the intended content unless there's a referrer of Google (and maybe Bing?). The site looks normal to the owner, or to crawlers, but when a user clicks through from Google it redirects them to a spam site.

I mostly see it with restaurants getting hijacked by herbal quackery.

You need to learn more about cloaking. This is a fascinating topic. There are many techniques to show different results to Google and a real web user, like checking the user agent, checking the ASN for the originating IP, checking for indications of a real browser, etc.

Internet-scale web crawlers are hard to hide. You can just set up a bunch of domains, and flag any IPs that repeatedly visit them as likely crawlers; then serve malicious content to the rest.

The TL;DR = Not really, according to our methods & analysis. But maybe.

I’ m worried about Gmail.

I switched to Duckduckgo as a default search engine for a year and I never needed to come back. More diverse results, fewer ads, and less SEO hacking. I think Google dropped the ball. This topic is becoming common in HN.

Tried using Google Bard. It gave me complete nonsense. It cited the sources for the nonsense, so I checked them: AI generated SEO spam pages. Gave me a good laugh.

(By comparison, Phind gave the correct answer, and high quality sources.)

Yes.

Google Images is filling up with AI slop to the point of unusability for some purposes, even content that isn't actively engaging in SEO shenanigans is getting pushed to the top. In some cases even the key image that Google displays first on a normal search is fake, like when searching for the "tank man" prioritized showing a fake AI selfie from the mans perspective. It's still there in fact, just not the first result anymore, but Google might have demoted it manually after the backlash.

Looking to browse works by a specific artist? Good luck, Google evidently can't tell the difference between a genuine J. C. Leyendecker piece and anything shat out by a "Leyendecker style" image generation model. Search for "Yoji Shinkawa" and this https://i.imgur.com/RYghaoY.png is currently the first thing you see, which isn't even close to his style, but Google has somehow determined it's the image that best represents him. The full images page shows his actual work but interspersed about half-and-half with AI imitations.

My speculation is that Google prioritizes showing recent results, presumed to be the freshest most up-to-date information, but of course for a historical event like Tiananmen Square or an artist who died in 1951, nearly all of the fresh results past a certain point are AI simulacra.

I find DuckDuckGo's[0] (DDG) images feature to be a brilliant replacement for Google Images/Lens. Google Images doesn't let you view the actual resource anymore and disabled that feature years ago. DDG's image feature is crucial since it doesn't bring you to the site itself, it just points to the raw URL of where the asset resides and you can easily download it, without hunting the graphics down on the site itself, although DDG does offer a link to the page where the image was found. The important bit is pressing the 'View File' button.

[0] https://duckduckgo.com/

I think google got into legal trouble for doing this (linking directly to the source image, instead of to the site)

Or they were at least threatened with legal action. I remember when it happened many of my friends were annoyed overnight

I doubt a legal case would succeed, and Google would know that. But the types of companies who threaten to sue over deep image linking are also the most likely to threaten not to sign contracts for content sharing with YouTube etc or for ad services. In other words, Google's other businesses create a liability for their search effectiveness.

If I recall right, it was over Getty Images, who were upset that people were right-clicking their images to save them from the image search (with the watermark) instead of clicking through to the page to be prompted to buy it

So rather than just delist Getty and solve the problem, they decided to make their product objectively worse

It's a trifecta of lies:

1. Domain-specific engines is too complicated. Let's have a general purpose search engine.

2. Too many buttons is too complicated. Let's have a single search box.

3. Guessing what you want when you are anonymous is too complicated. We need your search history to tailor your search for you.

Please, search engines, hear me: you can not simplify the needs and wants of 7 billion people and zettabytes of online content into a single little shitty search box! It's too complicated!

At this point the only thing these search engines are good for is for finding the URL of something you already know the name of. It's a phone book. I type Hacker News on google, it tells me its phone number: https://news.ycombinator.com/

Comically, when I google hackernews or even "Hacker News" I often get the other Hacker news first.

Finding my preferred UI https://hckrnews.com/ that's not even in the first page of results. Rajat's version https://hckrnws.com/... nowhere to be seen.

Google is becoming a mediocre phone book.

#3 is certainly bullshit, but Yahoo failed because of #1. Forcing a user to drill down through hundreds or thousands of categories to do a search is absurd, don't you think?

What other buttons do you want?

But for sure Google has crippled its core product. First they removed the + option, which forced the inclusion of words (their excuse was that it "interfered" with their stupid "Google Plus" product which is now gone). Yes you can use "allintext:" but come on. I'm not even sure that's honored anyway.

And the removal of the ability to exclude certain sites from results.

> Forcing a user to drill down through hundreds or thousands of categories to do a search is absurd, don't you think

Are you saying that's how Yahoo worked? IIRC, that was optional, they always had the search box:

https://web.archive.org/web/19961023235123/http://www10.yaho...

I think the search box was always there, but for some reason it seemed that you needed to select a category...

OK, I just looked up their old layout, and they buried the search bar amongst so much crap that it wasn't clear whether it pertained to the ad or the thing above it, or what... if you even noticed it: https://helios-i.mashable.com/imagery/longforms/04ILIeAX3JAF...

For output to be accurate, input has to be precise. But there's a trend of dumbing down and minifying interfaces and then tacking AI to guess what the user really means from the little input he is allowed.

It will never work. It can't work. Anyone who thinks this can work is delusional. And with Google it's particularly obvious how stupid the trend is.

Look at the "tabs" Google has for search: all, images, videos, shopping, news, etc. This is things users can input. But wait! What if an user wants news about something? And they have to reach all the way out to the news tab. That's too much for our bubbling moronic users to manage! They can't into computers. They have room temperature IQ. They have never used Google before, so they don't know where the tab is. They probably don't know what tabs or links are either. I know what I'll do. I'll put an AI to reorder the TABS of my users based on their search history, query input, season of the year, and their zodiac sign based on what birth day they used when they signed up for a Google account. That should solve it.

And now the order of the tabs is all over the place and when you want to click the "images" tab it's sometimes not the second tab and when you want to click the "videos" tab it's sometimes not the third tab.

I think this is very interesting because you have to think. If Google can fail this hard at tabs, which is not really a complicated thing to program, imagine how hard they are failing at indexing the entire interweb. Imagine if they are doing to search results the same nonsense bullshit they are doing to the tabs. Just imagine it. It's clear they have absolutely no idea what they're doing with the tabs.

>And now the order of the tabs is all over the place and when you want to click the "images" tab it's sometimes not the second tab and when you want to click the "videos" tab it's sometimes not the third tab.

I also noticed this; they redesigned their search tabs few months ago and now it's sort of bad and user unfriendly. Sometimes there is "News" tab, sometimes there is no news tab and tabs look all the same and generic(white rectangles with black text).

>I think this is very interesting because you have to think. If Google can fail this hard at tabs, which is not really a complicated thing to program, imagine how hard they are failing at indexing the entire interweb. Imagine if they are doing to search results the same nonsense bullshit they are doing to the tabs.

God knows what is in their index and what is not in their index. I think every search engine needs to make its index open and transparent. And yea Google can fail with all sort of things, they are not ubermensch or something like that.

Google's ranking algorithms and search technology are millions of LOC and not even Google engineers know how exactly Google Search works.

I dislike the recency bias so much.

I can't count how many times I needed a "until X" button, but it's an ever increasing trend.

Yeah that is a huge problem.

Another search to try is "hand reference", tons of AI garbage

This one was fun:

> like when searching for the "tank man" prioritized showing a fake AI selfie from the mans perspective

In this case the label is that it's a AI-generated and the source is an article talking about that AI-generated picture.

What's Google's incentive to be "good" now?

They own the search market anyway, and the more time you waste on their platform searching for what you want to find, the more ads you see and the more money Google makes.

What else are those 97% people gonna do, "Google on Bing" instead?

So for them, being bad is actually more profitable than being good, meaning there's a conflict of interest between what Google provides and what their users want, but since there's basically no equivalent competition, they get away with it laughing all the way to the bank.

> the more time you waste on their platform searching for what you want to find, the more ads you see and the more money Google makes.

That's not the only way - it's in their best interest to rank ad-laden pages higher. That's more ad-impressions, which is how they really make their money.

Lets say a user searches for $FOO. Why on earth would google return the most relevant result if that result is ad-free? They can return the second-most relevant result, and get impressions on both the search-result page and the page that the user sees when they navigate to the first result.

Google's Ads (in search) are more profitable for them than AdSense, their RTB platform ;-)

I would also assume they're largely not competitive with each other. Adwords ads are mostly valuable when they've got intent. I.E. I search "5g home internet" and it passes me off to Verizon or AT&T.

Searches that end you up on AdSense (or pages that could have AdSense) probably are much less likely to have intent. So instead they pass you off to something you will read so they can hit you with retargeted ads from the last time you searched with the intent of buying you something.

I am sure there's overlap, but probably not a lot.

You have to balance being good with maximizing profit. If you aren't good, your userbase will go elsewhere.

> You have to balance being good with maximizing profit. If you aren't good, your userbase will go elsewhere.

When you have +95% of the market, you can merely be good enough that users don't leave.

In this case, until Google literally starts serving ads-only and links to other google products (Hi Chrome!), they are not going to lose any money.

I get what you're saying, but why hasn't a better search engine emerged yet?

Bing's AI shows some promise, but I can't find anything better than Google. DuckDuckGo is shit. Classic Bing is shit. Brave Search showed promise, but it's also full of spam and with a smaller index. Marginalia showed some promise for smaller websites, but it's small, too.

All of them are unusable for local searches, except for Google, which is where I need search most. That and searching for obscure programming-related error messages.

Kagi doesn't even use it's own index, you're basically paying for a UI making API calls to Bing and Google. How well does changing the ranking work anyway, if you don't have your own index?

I want to believe that if Google were more aligned with users, search would be better. But where is that better search engine to showcase it?

Do a poll on HN. I bet that the vast majority of users here still use Google's search, and you can't blame HN users of not knowing of alternatives.

If Google really keeps its market share due to their monopoly, where is that better option that's being ignored? Tell me and I'll jump on it.

I know it's been mentioned in this thread but Kagi is a gods damn breath of fresh air compared to the current public search engines. They had a short outage recently and having to go back to Google for 30 minutes was painful. I have no idea what their secret sauce is (likely just being different from Google's algo that's SSO'd to death is enough) but it's now the easiest $10 I spend a month.

>I get what you're saying, but why hasn't a better search engine emerged yet?

Although Moore's law brought down the price of hardware and information processing dramatically in the last 20 years, it is still fairly expensive to crawl the Web, index it and rank it. Hardware cost + engineering cost can escalate pretty quickly, unless you decide to have smaller index than major search engines but then users will complain that search results are not good enough.

Bing Deep Search is interesting.

That's what Microsoft was thinking when they, in practice, stopped developing Internet Explorer (after version 6). That didn't work out too well for them...

Where will they go when Android, Chrome, Safari and Firefox all default to Google and plenty of Edge users also switch to Google?

Due to the defaults it's the only search engine most people ever heard of. Think of Plato's cave allegory.

If all your life you've only used Google and never anything else, making it the ground truth for you, how would you know it's bad in order to motivate you to look elsewhere?

> Where will they go when Android, Chrome, Safari and Firefox all default to Google and plenty of Edge users also switch to Google?

LLMs

Honestly, I hate to jump on this bandwagon, but I was after some information to help my wife put together some notes on a topic; at first I was coming up with questions we could answer and I'd search for the answers/sources online;

I decided to first try ask an LLM, so I asked Bard some targeted questions and asked for sources, and had all of the answers I needed, conveniently bullet pointed within 3-4 minutes, and all I had to do was go and verify the sources, job done.

I've done the same for a handful of things I use to default to a search engine for.

My favorite is recipes.

The blog spam around recipes is notoriously bad and is a direct result of Google Search.

Using these LLMs, all I need to do is tell it what I'm looking for in general and it will give me an entire recipe with no extraneous information.

It can handle adding or removing ingredients, substitutions. It can adjust servings. It can flip the recipe to work in a slow cooker or pressure cooker.

I've even had some limited success where I list restrictions based on picky eaters in the house for creating longer meal plans. No search engine can compete with that.

There were apps designed around these use cases. I foresee these sort of nuanced and personalized interactions being a key to drawing people away from search engines.

Absolutely, these sort of things which have varied but somewhat straightforward solutions will excel.

I'm going to give it a go for recipes, because as you say, blog spam is a nightmare. My wife has a lot of cookery books which she likes for her style of cooking, but I'm a little more "ad-hoc".

GPT4 is the only one I've used much but it really can be good as a Google alternative for certain kinds of questions. I wouldn't trust it for anything obscure and complicated though - if you ask it about something like competitive Pokemon, you'll just get a stream of confidently incorrect junk that any beginner could disprove. (Asking for sources does help this sometimes though.)

For sure, given Google being who they are, Bard is quite good at things like Kubernetes, Golang, things you'd expect a search engine to have indexed, etc.

I haven't used ChatGPT extensively so I can't make a comparison, but with the right questions, I've been able to get great answers to technical things I can't be bothered to look up by hand.

LLMs are still far away from the daily mainstream usage Google gets, and since all devices default to Google, and people never change those, they'll stay on Google and only occasionally switch to LLMs for more advanced tasks.

I thought the same thing and recently ran across perplexity mentioned in some random tweet. (I know I am late to the party) It blew me away, I use it as my primary search engine now for most things.

They're going to use ChatGPT, and guess what, Bing hosts that too, so yes, they are gonna Google on Bing. Bada bing, bada boom.

I recently caught myself that I unconsciously retry my google searches by adding more keywords. I also noticed that small sites, blogs, forums are gone from google searches.

Now I am using 3 search engines.

Bing for Normal searches. Google for local(country) searches. Yandex for small sites, blogs, forums.

BTW, Kagi has a dedicated "Forums" toggle at the top of every search result page. It's very useful. Google used to have this but removed it.

Unfortunately Kagi seems to have a US based pricing. I will ask them if they plan to add region based pricing.

Edit: They aren't likely to add it. https://kagifeedback.org/d/687-implement-regional-pricing

I've found adding "forum" to my query on Google works decently

Have you tried DuckDuckGo? I find it fantastic -- it does both general queries and local country queries well. Anytime I go back to Google (e.g. when I'm setting up a new computer) I am surprised that anyone still uses Google -- the result I am looking for is almost always half way down the page on Google yet it is #1 or nearby on DuckDuckGo. Apparently it's based on Bing's index but with their own tech on top.

I simply add the word forum to most of my Google searches.

One of my biggest complaints about Google is the prioritization of pop culture slop over everything else. I feel like I'm frequently getting auto-corrected because some Netflix show is titled with a pun of a phrase or there is a dramatization of some historical figure that Google considers more relevant than the real person.

Google "Queen Cleopatra" is a good example for me. 95% of the results are about the Netflix documentary. There is bodies of work dedicated to the history and legacy of this queen. Quality content, produced by leading academics, historians and universities. And yet Google insists I should know everything about this poor production (a bit of an understatement) and nothing else, except for a wiki article as one of the results (Good job Google, you at least got this bit right).

Google knows so much about me. And yet it continues to act as though it doesn't.

I think it’s the word ”Queen” (which is not normally used to refer to the ancient ruler). If I just google “Cleopatra” I only see results about the historical figure.

I've noticed a pattern with 'free' services like Facebook and now Google. With simple tasks like checking emails or finding phone numbers in contacts, there's no friction. However, Facebook lost my trust long ago as it seemed more focused on capturing my attention for ad revenue, leading to time wastage.

I initially trusted Google for its efficient and seemingly fair service, where smart ad-targeting was the price for speed. But now, Google feels similar to Facebook; it's harder to switch to alternatives like Kagi on iPhone due to financial ties with Apple.

This shift in Google's approach, prioritizing trapping attention over genuine service, is frustrating. I'd rather pay for a search service that values my time and provides real utility, than endure the hidden costs of 'free' services.

In last 5 years I have noticed a clear decline in the usability of Google Maps as ... a Map.

Increasingly you have to pick a destination and blindly follow directions, your ability to use it as an informational tool, but exercise your own judgement seems to have been intentionally crippled.

I can’t even blindly follow it. The last two or three times I’ve tried to use Google maps for directions, it has redirected me through walking trails and bike paths. Each time I closed it and just used Apple Maps to get where I needed to go.

> I'd rather pay for a search service that values my time and provides real utility, than endure the hidden costs of 'free' services.

How much would you pay for Google search without ads? Or more importantly, how much would the average Google user be willing to pay?

If you live in the US and click 3-4 ads per month, you're generating ~$10-20/mo in revenue for Google from advertisers.

(Like you, I would also love to pay for an ad free Google, but sadly advertisers are willing to pay Google a lot more money than consumers are willing to)

I do pay $10/mo for search. Also lol "and click 3-4 ads per month" what? This is HN, who clicks ads? I've never clicked an ad. I spend all day trying to teach people to never click ads. At this point I actually couldn't click ads even if I wanted to because uBlock hard-blocks all the domains they redirect through for conversion tracking.

I occasionally click on Sponsored search results, if they're what I'm looking for and I don't much care for the company on the other end of the link. Take that, Big Corporate!

10 years ago I'd pay for ad-free Google if I had the budget.

Today? Nah. Search has become bad enough that I don't believe it's worth any cash.

Google, YouTube, etc., they're all getting worse. I'm finding it ever more difficult to learn anything anymore from the internet. Whether it's learning how to play the guitar, how to fish, weight lifting or how to do mostly anything, I get trash results. Google results are exclusively shallow content, most of which are trying to sell me on something. YouTube is full of short videos with click baity subjects entirely irrelevant to what I actually entered: "Don't make this {topic} mistake!", "This one trick will make you better at {topic}!", "Top 10 ways to become a pro at {topic} in two weeks!".

Due to the constant trash these algorithms insist on feeding me, which I believe is actively contributing to the dumbing-down of society, I've naturally gravitated towards books. Books are leagues better in terms of quality of content. They are more detailed and thorough. It's a richer experience so far.

Steel that was formed before we started making nuclear contamination, known as low-background steel or pre-war steel, often in the form of old naval vessels and other large, old, steel infrastructure, has a special value to us. It can be used for building instruments with high sensitivity to radiation, as the steel made since becomes polluted with what is now part of the world.

I see a strong metaphor for literature authored before a specific time, roughly when the web came to be, or certainly when two way discourse on a page, or aggregation, became prevalent. And certainly far before bespoke communications targeted at us, as individuals or interest groups.

If the modern web were air it would taste of metal; I have a fear that I will become biased that older texts are superior for the sole reason that modern texts can be assumed inferior.

It did not matter how many times I read the Haynes Manual to bleed the clutch on my 1989 pickup truck, having someone show me on video was the only way it was going into my thick skull. However it worries me that the way google works anything useful like this with only a few hundred views will eventually be deleted to make room for the next funny cat video or 'influencer' pulling shit out of their asses.

Did you pay for the video proportional to the value that you received? Most people don’t, so why expect the services to not get worse, and be forever subsidized by another cash cow?

A lot of enthusiasts simply want to create content for other enthusiasts. You pay with advertising. It was working well until recently.

>Due to the constant trash these algorithms insist on feeding me, which I believe is actively contributing to the dumbing-down of society

This part really worries me right here. We don't know the long term ramifications of the modern day internet, but so far they do not look good.

And the people making the algorithms & internet interfaces could not give a flying fork less about those long term external consequences . . .

SEO, including video SEO, is constantly in tension with people who want to make money vs. people who want information.

Even if Google were incentivized to clean up their algorithms, the people who want to make money will necessarily always be one step ahead in the cat-and-mouse.

In the case of YouTube, it's not trash videos that frustrate me, it's YouTube's UI.

First, they have been injecting unrelated recommendations into search results, under headers like "For you" and "People also watched". I don't want to see a pimple-popping video when I'm searching for something related to woodworking. (That's an extreme example, but I have actually seen those types of videos injected into completely unrelated search results. In fact, I just did a search for "hand cut dovetails" as a test and YT recommended two different disgusting pimple/pore videos in the first few dozen results.)

Second, they don't admit that they are out of results. Instead, when search results dry up they coughing up unrelated recommendations so that you can keep scrolling forever. This makes it look like there are more results than there actually are, which is completely unhelpful.

At some point YouTube broke it with an update, but for a while I had uBlock Origin configured to block the Home page of YouTube entirely, so I'd just see a black screen. I still have it blocking the recommended videos sidebar and the comments.

YouTube actually has a lot of decent content still, if you can find it. I've found that jumping over to look at channels that do collabs with channels that you already know is a good way to discover new content.

If you deactivate the video history, Youtube’s homepage stays empty

> Whether it's learning how to play the guitar

I don't know about other topics, but it should still be pretty easy to find quality content if you want to learn guitar. Spend a couple of hours to find the channel that suits you best and stick to it.

Overall, there's still lots of great content on youtube, but you need to look it up yourself. The recommendations are useless and make you lose time, and the suppression of the dislike count makes it much harder to filter bad stuff. Also most professional YouTubers don't have much to say. They repeat themselves over and over.

Maybe someone should come up with a custom recommendation algorithm. Don't know if that's doable.

I still feel like the Youtube recommendations are A+

I watch only certain channels and a lot of long form documentary and gaming content. So that is what I get recommended. The few times I use the "dont recommend" button it never pops up again.

I used some custom CSS to block out my YouTube home page, so I never clicked on anything on the home page. I was getting god awful "content" suggested to me on the sidebar of videos. Once I disabled the CSS and started watching some of the recommended videos my recommendations got instantly better. It seems the recommendation system gets better the more you interact with it.

It also is helpful to search for videos and click on those. If I search for long form podcasts or interviews, I will get more of those kinds of videos recommended to me.

How do you get rid of the trash Shorts?

I just tested now and

https://addons.mozilla.org/en-US/firefox/addon/hide-youtube-...

works.

Confirmed thank you!

You're right about the trash results.

Recently I was thinking about getting a dog and was trying to find as much information about certain breeds as I could. Everything was just the same stuff repeated over and over, very little substance.

It took me a few days to realise but a lot of the Youtube results were actually entirely created by AI. Not just the voice over and what it was saying but the actual video was as well.

You can get AI to spew out content on whatever topic and as long as there's money to be made from said content, this does not bode well at all.

Repeating a comment I read here a while back - Google has lost, perhaps even given up, the fight against spam content.

Funny you mention YouTube. We did another study on this as well. https://downloads.webis.de/publications/papers/bevendorff_20...

I gave up on Google a couple of years ago. I used DDG for a while but often had to jump back (!g) to Google for some things. Now I pay for Kagi (been a year or two) and there’s no going back for me. I hardly ever even need to scroll down to find what I need, it’s just there. Saves me so much time.

Google created this problem by slurping up all advertising money from open web activity for themselves.

The more market share of online display advertising they gained, the worse their results got.

Why? Because the only way to have a large volume of authentic content being produced at scale is to have a healthy ecosystem of independent sites that are profitable based on display ads.

As much as HN-types hate advertising, it was literally the only thing that made the web of yesteryear so special. Things like Adsense enabled blogs on tons of niche topics to be monetized and thus we had better open web content.

When Google decided there was more money to be made off ads before you even clicked on a search result, that ironically was what ended up killing search.

Now the only way to monetize content from search is via shill company blogs, affiliate marketing listicles (10 best dog toys), etc. So that’s what we get.

For example, if there was a passionate person creating authentic, amazing content about dogs, they wouldn’t even crack page 1 on any search for dog toys no matter how good their content is.

>For example, if there was a passionate person creating authentic, amazing content about dogs, they wouldn’t even crack page 1 on any search for dog toys no matter how good their content is.

So basically the problem of search engines and Google in particular is discovery. All early Google adapters say that they loved Google because it gave them relevant results and because they discovered new websites on Google. Nowadays they "discover" SEO spam, ads and the usual suspects like the most popular sites in that search category.

That's why we need "discovery engine" for Web, something like TikTok but adapted to Web. If search engines were the evolution of web directories, we need to think about how we can evolve search engines.

It's AI, people using it vs not. And the AI is ever changing, trained on the internet, so it will be a self fulfilling constructive interference thing. If people keep using AI to write articles based on their competitors websites, eventually it will just be ai talking to ai.

Google search went from critical for me to irrelevant overnight with ChatGPT 4.

Like, literally irrelevant. I'm still watching YouTube, using Gmail, and occasionally checking out something on Google Shopping if I want to find something locally instead of on Amazon. But I use Google search about 90% less now than I did a year ago.

Reply on news.ycombinator.com