A glitch in the SEO matrix | Hacker News Reader

A glitch in the SEO matrix

izzy.co

337 pointsbbischof237 comments

You either play the Organic SEO game or you Pay to Play (Ads). Google as a search engine is now useless when it comes to searching for a tool/software etc because everyone has gamed the "Best software for xyz" etc. But what's the alternative ? None. There are these "review" websites like Capterra/Software Advice/G2 and again you have to pay to play. You can technically get a review from a customer and get listed BUT if you want to be shown on the main page for that category, you need to pay crazy PPC.

Source: I play this game since I run a software business. Would love an alternative but there are none. You either Play the game or you Die.

How much of your time do you spend “playing the game”?

I do have a small team but I am constantly involved ensuring that Google is not being mean to us and the game is all about trying to stay on top 5-10 results.

Your bio welcomes strangers to chat, but, no method of contact! =]

How can I get in touch?

Sorry just added. Feel free to reach out.

It's been a couple of weeks, and I unfortunately still have no desire to engage with Twitter, ha.. — can you forgive this, is requesting an email a dealbreaker?

> But what's the alternative ? None.

And gob help you if you dare search for alternative to _____.

That page is certainly useful to find alternatives , I like I can filter by FOSS

Which of the ten Google result pages full of Alternative to___ pages are you talking about?

https://alternativeto.net/ Sorry I thought it was obvious its been there for 10y at least

Before the Great Reddit Strike, adding "reddit" or "inurl:reddit.com" helped in that regard pretty well.

Given that the Reddit strike was an almost complete failure it still works well.

Several niche subreddits are still offline.

I wouldn't say it was a failure per se.

Old content is still there, but new content is of noticeably lower quality. Many subreddits have moved to minimal moderation or stopped it altogether. Engagement in places I used to frequent like /r/Games (which didn't even participate in the strike) is both down considerably and of lower quality.

Reddit, like Twitter, is too big to fail overnight, but they are essentially zombies at this point. Their growth potential has been neutered and their most valuable users (the 1% making 60% of the quality content) are actively toying with alternatives.

Oh yea. We have seen that one gamed like crazy :).

I find the site AlternativeTo.net usually helps me find good options.

I'm always kinda surprised that this site is still ok, they even clearly provide links to the tools actual pages.

> But what's the alternative ?

Since ads are a red queen's race, wasting people's attention and assuming we don't want to outright ban them then the next best alternative is to tax them heavily. How about 0.00001 cent per pixel-second-view?

Then people will only put up ads when there's some real value in it, not just to keep up with the competition.

That's a really good idea. We need regulation to prevent prisoners dilemma situations to the detriment of all.

Cue the blurry ads lol.

From the point of view of someone who spent 10 years running an e-commerce store, and all its advertising, until a couple of years ago. Both SEO and search result ads on Google are dead. The hay day of being able to play the game and make a tidy profit is long gone.

Google are playing every trick in the book to extract every possible cent from advertisers, spying on their business and sales to maximise their own profits. Google's visibility of all transactions on every e-commerce website on the internet is insane. People complain about the tracking of users/visitors, but the tracking of businesses is just as bad.

They probably have better insights into the economy and market trends than most governments and banks.

The penny is dropping, advertisers are noticing, my long term expectation of Google's business are not what they were.

> Both SEO and search result ads on Google are dead

And then

> Google's visibility of all transactions on every e-commerce website on the internet is insane.

Ah, so they have all of the information and yet their ad products don't work. They don't know this?

They appear to work just well enough to keep you hooked while they increasingly eat your margin.

My 2 cents - in the past, there were bigger inefficiencies that advertisers could exploit. So they could buy a click for 10 cents, and profit 25 cents from it. Google has gotten better and closed that gap, so now you pay 15 cents (and due to increased competition, you might only profit 20 cents)

The opportunities are still there in local minima, places where it's too small for Google to optimize. But the trend is clear.

A perfectly optimized ad algorithm reflects the market, and offers little opportunity that isn't present in the business itself.

Costs go up because winners have been found. Like monopolies in capitalism.

Naive question, but if a winner had been found in a given niche, wouldn't there not be as much bidding competition and thus costs would go down?

Concretely: if I am the only pie baker in the world (I won), who else would bid against me for "place to buy pie"?

My unsubstantiated hypothesis is that google has gotten better at cross-promoting - so they could target people who like cake, increasing the market and competition for bids, making prices go up.

Incumbents often maintain large budgets to ward off competition. Competition wins enough to keep bidding, etc. If America is not full of monopolies, the bidding environment stays liquid. There is also the threat of using literal brand name targeting, or conquesting, which encourages your competition to run their own ads to keep you out.

I believe there are some agreements for larger corporations to continue to promote things even if it is not necessary. People with funny money can treat ad markets like billboards.

Another fucked up aspect is that the government often runs ads to promote its own policies.

Much like real markets there is sometimes unnatural activity that keeps it moving.

I've seen this narrative a lot in this thread. But if Googles ad-algorithm was actually good I'd expect to occasionally see useful ads. But Googles ad-algorithm is far from perfect. That's why people use reddit or other community aggregators to find useful products, the search engines just really suck at that task.

If you are like many of the commenters on here who is using adblockers and abstains from social networks, this is precisely why you see shitty ads.

Also if you aren't buying anything by clicking them, advertisers are ignoring you because you aren't a valuable user to them.

You are most likely reaping what you sew. Which shows that the ad networks are actually working.

It's not good enough that Google makes massive amounts of money for each click, the market demands they make more per each click every day. Gotta chase that growth.

Yes. Now the trick is to convince people that satisficing isn’t just laziness or somehow a road to business death.

> their ad products don't work.

It works for them.

They might need to make it work a tad more for their customers if they had to, but right now they apparently don't.

By "dead" the GP means "at the bottom of a race-to-the-bottom." People are paying $4 (to Google) per customer, to acquire customers with an LTV of $4, making them $0 per customer. That's the natural conclusion to competition under perfect information.

Yes. This is what I'm saying.

> They don't know this?

Of course they do, but people pay crazy money for this and it plays the important role in Google's market valuation, so it's in Google's best interests to continue chanting the Big Data Big Money mantra. It doesn't help that the myth/meme is strongly backed by the whole cyberpunk genre, as people love the dystopian themes of "big corporations know everything about you, down to your most secret desires you don't even realize yourself".

And it probably even work by some small but statistically significant margin, compared to some arbitrarily picked baseline, so they can even back this up if necessary.

The king is naked, though.

If online ad campaigns did not bring sales, a lot of people would stop doing them. There are other advertisement media; there is organic-looking product placement, there is overt product placement and "influencers", etc.

Still, people buy a lot of AdWords and similar, and occasionally I see a relevant ad I click. BTW ads within Facebook are usually of much better quality (for me as a reader), likely because they can correlate.more sources inside FB.

You occasionally click on "relevant" ads? I'm really curious why you would do that? Personally, I'll go out of my way to make sure that I don't buy anything referred to me by an ad, on the grounds that advertising is a vile and parasitic industry.

Not OP but presumably they click a relevant ad because it is relevant...

I relatively often click on ads for new electronic musical instruments, even though I rarely buy them. These are just interesting things to see, some actually novel.

I sometimes click on ads about things related to IT and programming, if they mention something novel and important, just to be aware of the landscape, and what potential competition are pushing.

I see ads relatively rarely though, because I run uBlock Origin. So the ads I see are usually placed in context with some care, not just randomly tucked on to an unrelated webpage. Most of the ads are really poor, as I can see if I use an unprotected browser.

If everyone in the world pays once for a product that doesn't work, you still get rich. They don't need repeat business, when they control they narrative. If they want to promote results touting the benefits of buying G-Ads, they can.

> If online ad campaigns did not bring sales, a lot of people would stop doing them.

I've done the math on this for a few small companies. I an thoroughly convinced that this is just not true for many companies. The vast amount of sales attributed to online ad campaigns are sales that a company would have gotten anyway even without the ads.

> The vast amount of sales attributed to online ad campaigns are sales that a company would have gotten anyway even without the ads.

This is a simple test. Turn off all of your ads for a year, compare against the previous. Write a blog post about it, destroy the industry.

This has been done, in a way.

Some time in 1980s Coca-Cola decided that they have enough of an iconic status, and they don't need as much advertising on expensive large surfaces. They prudently lowered such advertisement a bit, and checked. Shockingly, they noticed a decline in sales. They reverted to the old policy.

Even with all that data/information, Google still cannot find a more sustainable business model. Despite the countless number of sustinable businesses humanity has produced, so-called "tech" companies just cannot find one. These are not creative people. "We'll just keep on playing eavesdropping intermediary ("middleman") until we have undermined every business, every government and society along with it." Sounds great. Good luck.

That’s the case mostly for Twitter, meta and google that only/mainly live based on as revenue.

Amazon, Microsoft, Apple are more diversified.

With those companies, even with their non-data collection, non-advertising services business units, they continue to move more and more into personal data collection and advertising services.

Of course they do, soulless investors see monetisation options and go “free money when?”

They don’t care about good name and long term viability. As far as they’re concerned they can just sell when the stock tanks.

Advertising is in itself a sustainable business model, hovering around 1% of US GDP for the past century[1] (p.37 figure 3). Google really just needs to defend their position within the ad market to remain sustainable.

[1] https://www.bea.gov/index.php/system/files/papers/WP2017-9.p...

Id argue that advertising is am emergent side effect of money/commerce. Once you can produce goods for money you need people to be aware you are selling that product.

That being the case unless we stop using money their business is quite sustainable.

Right now we have a push-based ad model with a low signal to noise ratio. People are force-fed garbage in the off chance that in a moment of weakness they might accidentally make a purchase not in their best interests. The mutually beneficial aspect of what you're describing (informing people of goods and services at some Pareto optimum which they would actually benefit from) could just as easily be satisfied with a pull-based model, and the low SNR is an artifact of letting de facto scammers pay for the privilege of scamming people.

Similar background. Agree completely with your second paragraph.

What’s the best alternative for those adspend $$ in your view?

The alternative for users is Kagi, which is superior to Google for search results.

For businesses, there is no good alternative to Google. Many are investing a lot of effort and money in social media, but the returns are very low compared to Google.

I’ve been heavily considering the 1k Kagi but the limit thing has been giving me pause.

I’ve yet to try to trial tho so that should help give context how quickly I burn thru 100

I've noticed that there's a huge misunderstanding among people interested in Kagi: You're not banned from searching after passing the limit in your plan, instead subsequent searches will cost 1.5 cents each. So nothing to sweat.

Nothing to sweat, except money, of course. Not everyone has an open-ended budget.

Yes, I tend to forget about the plight of the commoners when sitting at my enormous hoard of several hundred cents.

You can put a hard budget limit so you're not financially ruined if you go over those $10. And there's ultimate $25 all-you-can-eat plan, too.

Oh I’m aware, but thanks. It’s more so the mental burden of second guessing if a search is “worth it” that would arise. Like I said tho I’ll use the trial to best figure out my process.

And also yea three cents for every two searches over the 1k monthly limit is pretty trivial lol. So nvm that shouldn’t b something to give me pause

Half the stuff I search for (old magazines) doesn't even appear in Google's SERPs any longer.

I resort to Duckduckgo and Brave's searches among many others. Even things like Yandex give much deeper dives now.

I just recently started using DDG is there any intuitive way to block the MSN links? Probably wouldn’t be too hard to get the original article url anyway tho.

And I’ve been pretty impressed with Brave search so far. Promising

DDG should introduce anti-bangs so you could do like "!!msn weather for my town" whatever and it would pull from anywhere that is not e.g. MSN.

I add "-best" to my searches. It helps.

So crowdsourced black hat SEO indeed beat Google’s ”thousands of the most intelligent and well-paid engineers of the planet”.

>But what's the alternative ?

https://metaphor.systems/ is pretty good for exactly this kind of thing. The way the search algorithm is designed also make it extremely difficult to game.

except it does not search that well...

There is no alternative. People talk about alternatives but everyone I've checked is even worse. Bing, DDG etc. All give complete garbage results. Perhaps not as gamified, but usually that just means you're going to a wordpress of a less competent admin.

Back to webrings! https://en.wikipedia.org/wiki/Webring

(This is a joke (or is it?))

I also think (hope), that reach/scope reduction is a possible counter-reaction as the winner-takes-all empires enshittify beyond repair.

From food to manufacturing to music and other cultural endeavours: making communities small(er) might be an interesting counter (r)evolution.

Without deciding whether your particular mention of webrings was a joke or not, the nostalgia for them that has been cropping up lately is a little bit puzzling to me as someone who remembers well those days. Because webrings themselves were often full of just pure junk, not to mention broken links, or websites that hadn’t been updated in years but we’re still “under construction” (with the de rigueur gif of a hardhat or something).

With its brokenness, those days were more enjoyable to many. The web was more interesting with the variety. Personal sites were a place for each mouth to speak. Today's sites are a place to be force fed pure junk.

Oh, no, I certainly agree that the old web was a better, more interesting place than today’s. By far. If you invoked webrings simply in order to invoke the old web, I totally agree (as opposed to invoking webrings per se). I just pushed back a little bit because even back in the day I didn’t like webrings. :)

I like webcomics, the scruffy sort, just one person a website and a dream. The best part of these little comics is where they will have a list of links to other comics they like.

This feels like one of the last bastions of the true web left, the small web that is actually interlinked, not the dead ends you find in your ecommerce or social networking sites.

But it too is disappearing fast, I for one will miss it.

I cant speak for any site other than G2. We have multiple unpaid products that appear appropriately in their lists and quadrants. The question one has to ask themselves, just like in SEO "is my profile optimized" and try work out what their automations expect rather than paying for placement.

Is there nothing like ConsumerReports.com for software? A service trying to just write impartial reviews for the sake of the reader?

I've been using reddit for this purpose

> You either Play the game or you Die.

And given the game is won by the highest bidder, all of the surplus created by technical progress goes to advertising which, as you point out, adds zero value and, in fact, detracts from rational agents making optimal decisions.

This is /exactly/ the kind of market failure where those of us who prefer smaller government want to start talking about appropriate regulation.

Hey @codegeek, have you tried SaaSHub? It's goal is to be more "objective" compared to Capterra & G2. Yet, it's questionable to what extent that is achieved.

The alternative is places where people congregate. Google used to be a great place to find relatively objective information, but being ad driven it’s all gone to commercial advertising. Word of mouth (IE searching with “[…] Reddit”) gives far better information.

> Word of mouth (IE searching with “[…] Reddit”) gives far better information

Used to. Now that advertising believe their salvation lies in forums, wadding into the big ones increasingly feels like entering the water at a British beach after the privatisation of Thames Water...

Not really. Thinking that Reddit is not gamed by marketers or bought/sponsored influencers in 2023 is just completely wrong. Any kind of aggregated rating is gamed, whether it’s Amazon or Reddit or Google or yelp

At best you’ll find one very opinionated guys recommendation which may or may not match yours

> Thinking that Reddit is not gamed by marketers or bought/sponsored influencers in 2023 is just completely wrong

I didn't claim its free of gaming, but I am claiming the quality of search results for "x" vs "x reddit" is notably higher for the latter in most cases that I personally use search for.

site:reddit.com is your friend, even with all the stuff that's happening over there

their ads are not worth it. they have become way too expensive and results are not good

Is there no search engine that outright bans all commercial content?

There are plenty of searches where I want to find what real people are thinking, not what some company's article says. This kind of thing could easily be crowd-sourced a la SponsorBlock.

> Would love an alternative but there are none

If you are trying to sell a product, I know one weird trick: Just make a product that people enjoy, and they will recommend it to others.

No need to pay for advertising, no need to play the SEO game. The only marketing you need for a decent product is a website with docs and a download button.

This works especially well if you have some sort of freemium product, because then the 90% of people who use the free version still act as multipliers.

I don't know if it works for all businesses, but it's worked for me over the last decade or so.

I use https://alternativeto.org to find tools, might be useful.

In contrast what is your top of funnel strategy for social proof?

What would a fair review site look like?

It's so weird that Google still hasn't created the ability to block spam domains.

I don't think it's a technical difficulty but rather a management decision to allow spam as long as they have their adsense ads and it baffles me they just don't care about their end users at all.

I think being a monopoly with most competitors lagging a lap behind can make you this way, but with chatgpt catching up quick, Google better start thinking about their users now as it's a growing sentiment that their quality is totally bad nowadays and their uncaring attitude towards spam domains like Pinterest and spam search results where sites are creating hundreds of pages with same content and different heading is just not gonna cut it anymore.

They actually did have it as an experimental feature for a little while, maybe 10 years ago. [0]

If I recall correctly, it was made available in response to another big wave of criticism directed towards Google about "Content Farms". It has been interesting to see the difference in response between the "Content Farm" debacle and what we are dealing with currently.

[0] https://googleblog.blogspot.com/2011/03/hide-sites-to-find-m...

Pretty sure 4chan is knocked down quite a bit. Sometimes when I search 4chan gif or b, I get a page of unrelated websites.

Most of these spam content farms are covered with ads supplied by Google. The incentive isn't necessarily there to remove them.

On top of that the worse the search results, the more likely the user will click an ad rather than an organic result. The Google of old wouldn't have been tempted by that incentive, but that Google is long dead.

Bad search result = more revenue from ads.

Obviously it's a fine balance, they don't want to loose users, but they will have the metrics to (religiously) work from.

As someone who used to run ad campaigns on Google until a couple of years ago, they are doing the same to advertisers. Users will happily click an ad, go back to the result and try another, many many times. Google have systematically made the advertising on search results worse, showing ads more regularly for poorer placements, removing control and visible auditing. It's all so they can extract more revenue from the advertisers.

This is such short term thinking imo but then what do i know. People who are paid millions of dollars are making these decisions at Google so maybe that's how the game is played. I still think treating your customers like the way they're doing it now eventually never always works out.

When you have a metric driven company, where peoples career progression and bonuses are tightly tied to revenue numbers in a database, you incentivise this sort of thinking on an employee level. The visibility of the impact isn't necessarily there at the top.

Not just a metric driven company, but one that's so large the left hand doesn't know what the right is doing and neither of them could optimize their jobs for the other even if the incentives were for them to do so.

> This is such short term thinking imo but then what do i know.

That's because it's completely inaccurate. It's just a meme propagated by HN and some others, with essentially zero correlation to how decision-making and prioritization actually happens over there.

The idea that clickbait is somehow good for Google's bottom line is absurd on its face, before we even tackle the idea that ads' interests are controlling search ranking

> The idea that clickbait is somehow good for Google's bottom line is absurd on its face

I guess I am simply unable to see so I will ask: why do you think the idea as outlined in the GP is so illogical?

>The incentive isn't necessarily there to remove them.

There is a reason people are using chatgpt as a replacement for google, or appending results with Reddit or Wiki

I'm waiting for a day there will be a capable LLM agent that would be able to crawl hundreds of results, read them all (working around all the obstacles, not spewing "reading content failed"), filter the marketing noise/SEO copypasta bullshit, find the meaningful bits, and present a nice summary.

Current LLMs can go crazy just by seeing a single well-chosen word, see https://www.lesswrong.com/posts/aPeJE8bSo6rAFoLqg/solidgoldm... for an amusing exanple. I think we need some breakthrough in adversarial training before LLMs can survive the very adversarial environment of SEO.

> I don't think it's a technical difficulty but rather a management decision to allow spam as long as they have their adsense ads

It is absolutely this, yes. They tell people where to go, and they also profit from the ads they put on the places they tell people to go, and they have no real competition[1]. It has made them one of the most valuable companies on the planet. There is no incentive for them to rank higher quality results over ad-filled pages.

We need to break up big tech.

[1] https://gs.statcounter.com/search-engine-market-share/all/un...

> they just don't care about their end users at all.

They do, a lot actually. We're just not the end users. The advertisers are. Thet don't need to care about us since they're a monopoly. Use Brave Search, DuckDyckGo or something elae to help change that. Get other people to too.

I tried using DuckDyckGo (lol) for a few weeks recently, but its results are just as bad or worse than Google's are usually. It's the same problem of endless worthless listicle noise.

Yeah, I used it for years and it used to be better but it turned pretty trashy, which is why I switch to Brave a couple of months ago and I'm pretty satisfied now. Still glad DDG exists though as an alternative protest vote.

If anyone wants this capability, the chrome extension uBlacklist ( https://chrome.google.com/webstore/detail/ublacklist/pncfbmi... ) provides it. I've found it very useful for removing github scraper sites from search results. Whenever you see a garbage result in a google search, you just click "Block this site" and it's gone forever.

> it baffles me they just don't care about their end users at all.

Just for you, especially for you in fact, I went and dug out this link to an interview with Corey Doctorow where he explains his theory of platform of “enshittification”.

https://podtail.com/en/podcast/future-tense-full-program-pod...

I think it will interest you to hear a reasonable theory as to why Google doesn’t care as much about their end users as you might think they should.

Interesting concept. Thanks for sharing!

> It's so weird that Google still hasn't created the ability to block spam domains

by who??? Google itself "blocks" or downgrades spam websites all the time. If they open it up to "the people", the same SEO crowd will rush in with scrapers to decimate competition, including any legit websites. So strange to see people jumping into conspiracy theories here without thinking of the consequences for a second.

I am pretty sure OP is referring to the ability for an individual, logged in user, to maintain a list of "don't show me results from these domains" on their Google account.

ok. didn't even think about that use case because I'm never logged on while searching. More bizzarely however, with 300 million domains out there and more added every second, you are in for quite a chore! even more delusional

I always figured it's because there isn't a real definition of "spam domains". Google can't come up with a set of rules with the right error rate.

I'm talking out of my ass here and know nothing about how search actually works in 2023, but I feel like their search problems really are solvable with their scale.

Like you said, a first step would be blocking spam domains.

Another would be a return to how the algorithm used work, prioritizing results that are frequently linked to elsewhere using the relevant keywords. In the '00s, this was quickly gamed by companies setting up endless blogs to link to their own products, but I feel it could be mitigated by assigning some kind of trustworthiness score to the sites doing the linking. It shouldn't be hard to recognize that a site being recommended a lot on Reddit or some other well established repository of user-generated content is going to be more genuine than "recommendations" from some random blogspam site nobody's heard of.

https://news.ycombinator.com/item?id=36792737

How long until chatgpt injects an advertisement every other paragraph?

You’re making a few bad assumptions.

1) Google and you are defining spam differently.

2) Spam is adversarial. They have the ability to block spam domains from last week, but it’s an ongoing investment as the adversaries adapt.

That said I do think Google depends too much on human raters who are not customers. It’s caused a lot of drift between the customer’s expectation of quality and Google’s.

After having used computers for a few years (oh, about 30), and having grown up with the internet (first dialup at age 16 in 1996), I gotta say,

Access to information should not have (ranked, optimizable, commercially-bent) search as its base interface. This is a cosmic, civilization-level screwup. Not only do you get the decades of SEO and advertising, but the whole system becomes lossy over time. You can't find what you used to anymore.

Google was absolutely incredible when it first showed up. Now it is not great. When I want information all it gives me is shopping or listicles one step removed from shopping.

What's the alternative? Manually curated directories lost to search. Anything completely manual isn't going to be "web-scale". Anything algorithmic is ranked and optimizable.

Alphabetic? AAAPlumbing (btw my wife gets tons of business because her name starts with an A, and the insurance website is sorted by alphabet)

By date?

Manual curation, the way libraries work. (Or at least, good libraries) And search functions within those curated datasets.

Who is going to be the neutral arbitrator who assigns Dewey Decimal numbers or Library of Congress codes to web pages? Can’t trust the page authors that’s for certain.

No one. Remember the big list of cool blogs that were generated off HN a couple weeks ago? That's one curated dataset. People can start their own curated databases and then see if others find them useful. Big ones become well-known, like big libraries. Small ones might become known for having the best resources for niche subjects.

Wikipedia is a pretty decent, manually curated repository of encyclopedic information. Google search gets worse and worse every year but Wikipedia is pretty stable over time.

Sure, it has its own problems with edit wars and editor politics, but compared to the cesspool of SEO spam on Google, Wikipedia is an information paradise.

Wikipedia is a barest, insignificant smidge of the knowledge that's available on the internet. Yes, it's incredibly useful, but no, you can't stay within its bounds of knowledge for any appreciable time if you're actually trying to accomplish something beyond falling into a Wikipedia hole.

For manually curated content, Google and other search companies were solving that in 1998 because manual curation wasn't feasible for the amount of content on the internet. It's been 25 years since then, and we're not exactly producing less content online.

To further support my argument in favour of manual curation, I would like to ask: why, for the past few years, have people ‘in the know’ been appending “Reddit” to their search strings on Google? Because Reddit is the largest repository of manually curated links, discussions, and original non-encyclopedic information on the internet.

People want manually curated information. They don’t want automation which by its very nature, as a fixed set of parameters, can and will be gamed.

Because they want a bias-filtered selection of the internet. It is very obvious looking at the reddit comments that most are made by bots and not humans, the difference is that most behave similarly and the upvoting favors sameness.

That’s not at all obvious to me. What subreddits do you visit? The ones I frequented had maybe 5% or less bot comments. I think mods generally banned the bots from those subreddits too.

I feel like Reddit is a bad example for an argument against search, because it still relies heavily on search. People aren't finding a specific subreddit for their query and then browsing it until they find what they're looking for, and they're usually not even searching directly on Reddit (because of the poor search quality), they're searching on Google.

What does "manual curation" mean in this context? I think it really just means "absence of spam/low quality content", which manual curation is not strictly necessary for.

Yahoo, back in the days before Google, was a manually curated search engine. They had a team of people doing the indexing work and creating the database from which the search engine pulled results. In principle, this avoids SEO spam because the manual curators aren't going to add spam sites to the index. Reddit functions similarly because, in principle, users aren't going to upvote spam posts.

Ah. But we're still using some algorithm that's doing curation to even look at reddit. I see you're point about reddit being manually curated via upvotes, interaction, and moderation, but there still has to be something in between to even find the content and match it with what we're looking for. Even reddit's too big to just be a page of links, a la Yahoo.

Reddit has avoided

It's less about Wikipedia itself and more about its citations. Is Wikipedia sufficient to find nearly any important[0] information if you follow hyperlinks, or if you also look up physical references? What are the characteristics of the information that's missing? How long can you avoid Google for if you go along the chain of citations and links?

[0] objectively quantifying importance is an entirely different can of worms, but people know what's important to them

Lists/"awesome lists" and lists of lists are great.

https://github.com/jnv/lists

> Manually curated directories lost to search.

I rather suspect that they'll regain some popularity. Probably not general-purpose ones, but a variety of directories each covering a specific kind of site.

> Anything completely manual isn't going to be "web-scale".

True, but even so, it might still be an improvement. Especially if it's hundreds of different directories, each with their own tight focus.

> What's the alternative?

Apple blocks all tracking in Safari by default and builds search engine without ads for iCloud+ subscribers.

Google will eat at it further with stuff like this from the article:

"Google knows this, so it adds a “Question and Answers” block before the first SERP result, probably diverting a tremendous amount of traffic away from that page."

They scrape answers from sites that they used to send traffic to and publish it themselves. In many cases, though, those sites won't exist without the incentive of organic search traffic.

I get why Google does it, but it does create a feedback loop where less of that info will be out there to scrape.

I've had good luck promoting my game on social media but when someone hears the name and Google's it they'll get results like pic related where someone has embedded an ad-wrapped knock-off of the game which contains more adverts. https://imgur.com/a/mz07uyR

Personally I can't find much on Google if I'm actually searching for something I don't know about. Finding the local mechanic is fine but querying a programming question, health problem or recipe is absolutely fucked. I sold all my Google stock recently but should have done so sooner. The value they have is in gmail and gsuite accounts but I can see the search business is circling the drain.

Conspiracy theory: they love showing shitty sites because those sites display adverts making Google money.

The majority of times when i search on google I'm looking for someone's personal take on a matter. Whether it's a software recommendation or alternative, a recipe, or experience with a tool. Google only has listicles, top x, or best in 202x.

I used to search reddit, but the hostile mobile page ruined that for me. Now i go to yandex and navigate through malware to find what i want.

It doesn't help that recent trend of developers building their content website with react and such, have no little to no concept of SEO.

I think it depends on the topic. On technical / software questions I do still find individual websites and blogs. For something like product reviews, my experience is exactly like yours and the only thing that shows up is the trash you mentioned.

I don't think this is a search problem, I think Google is legitimately surfacing what's available on the Internet for given query. I think this is the dead internet theory at play, and that there just isn't anything valuable to show!

People who would (in the past) run their own website or blog now post that same content on social media, which is why Reddit has become mandatory for people looking for genuine opinions.

> I think Google is legitimately surfacing what's available on the Internet for given query.

Google certainly omits quite a lot, though. Especially smaller sites run by actual people.

I've been looking recently for an opinion article on the Tailwind CSS framework, ideally listing its benefits, drawbacks, long-term maintenance considerations etc. There appears to be absolutely no way to find this type of information on Google these days.

Because of widespread spam, astroturfing, hidden shilling, and soon generative AI, there is no way to tell whether something written on the internet was actually someone's personal take or whether it is paid for. I'd argue there's no web forum or social media site anywhere where you can reliably say "this was posted by an actual human and was not paid for."

Sam Altman’s worldcoin and world ID would solve this

It wasn't written to answer someone's question or satisfy their search intent. It was just written to rank.

Anyone who’s searched for reviews or tech questions recently has experienced this. And it’s slowly eating everything.

The web is turning into nothing but potemkin content.

It's all parasitic bots on the surface stealing content and generating sludge. Humans find shelter underground in un-indexable private chats.

This is strangely where I've found the most personal benefit of LLMs. As long as it's not too current or depend on recent information (like the weather), searching for information in LLMs reminds me a lot of Google 10 years ago.

Yea sometimes LLMs hallucinate, but sometimes websites are just wrong. For a variety of topics LLMs are just a better form of search.

I just typed this into ChatGPT (so not even a question): "tar -x??? filename.tar.gz"

It gave me back "tar -xzvf filename.tar.gz" with a breakdown of what each switch does. I don't think Google's ever been able to intuit what I'm asking for like that.

Totally. The internet is full of blog sites explaining how to do things that are impossible. Like how to cast from vlc to dlna on Android. They'll have a whole page with bullet points and everything that ultimately involves using a Chromecast. I once ended up at a page about an emulator that was hallucinating in GPT3 levels. Something like playing Gran Turismo on an xbox emulator or something. They have no shame.

I've seen this being very prevelant in technical support, searching for an answer about "Windows blue screen error code 0xwhatever" will yield an infinite list of results telling you to reboot, run sfc /scannow, and reinstall Windows

The root cause of all this madness:

Ads.

As long as there is even a marginal financial incentive to get human eyeballs on websites (as opposed to the viewers actually playing for services or, you know, just hosting websites for free because they are cool), this will keep getting worse.

A genuine desire to help incoming traffic doesn't really matter either. Anything helpful will be infinitely cloned, as long as there is incentive to clone it.

I dont really know a good solution either. Targeted ads are incredibly useful to small businesses, and that genie is out of the bottle.

No it's not just ads at all. SEO drives traffic. There are many ways to monetize such traffic, and ads are just one.

A lot of SEO is driven by local business owners who hire SEO people with dubious track records that make big promises. They invariably clutter up their websites with 10,000 articles along the lines of "best plumber in dallas".

It's funny you mention ads though. One reason that it's so hard to find an SEO person that's actually worth anything is that if they were really good at what they did, they could just throw up their own websites with ads and make a much better living that way than selling SEO services. They don't of course, which should tell you something about how good they really are.

The problem isn't that websites owners want to promote their website to the top of the search results, the problem is that Google's financial interests are aligned with those pages instead of its users.

If Google was putting significant resources towards combating SEO spam instead of encouraging it, the sites returned from a query (e.g. the sites that make Google the most money) would be the ones that the user most wanted, not the ones that maximally participate in Google's ad-extortion business.

> It's funny you mention ads though. One reason that it's so hard to find an SEO person that's actually worth anything is that if they were really good at what they did, they could just throw up their own websites with ads and make a much better living that way than selling SEO services. They don't of course, which should tell you something about how good they really are.

That doesn't even make sense. SEOs exist to make sure websites are more consumable by bots for search indexing. They likely have no backend experience or much experience working directly with css/html, and are more likely dealing with full stack developers who have no desire to understand how their code is functional for users but isn't consumable by bots. The most successful deal with brick and mortar companies which require actual capital to create and not just SEO skillz, and anyone with a brain would know dropshipping is so 2014 (there's like a million of these failed journeys documented on random boards like blackhatworld where people try to monetize search like you suggested, but it's a hella minority because we are deep into the enshittification of search). SEO is important to deal with things like the proliferation of angular 1, which was a hellhole abyss for startups in search because google wouldn't index their shit and developers needed to iterate fast. Hell, a large part of the react community is based around things like server side rendering just to get around google bot refusing to let JavaScript load and search results were polluted with handlebars variables for years because of that. Google bot still has problem rendering webpages while I do all of that shit all day with google's own puppeteer.

Tl;dr: SEOs are a cottage industry to help companies deal with the ongoing enshittification of search, and basing their value or success on whether they could launch their own site isn't a good benchmark.

> SEOs are a cottage industry to help companies deal with the ongoing enshittification of search

SEO is a $122 billion/year industry. That doesn't seem like a cottage industry to me. And it exists in order to get companies to rank higher in search results.

I think the SEO industry is a large force helping to make the web much worse.

That's a drop in the bucket compared to the rest of marketing at 5 trillion. SEO is surely making the internet worse, but google is also incentivizing the absolute hell out of it by dedicating resources and promoting personalities a la John Mueller and other Search Advocates within the space. Currently, Google's inability to assign value to pages in a coherent way is making it worse. The Google SEO guidelines, proliferation of tools like lighthouse, AMP pages, and an outdated search algorithm encourage the homogenizing of the web, too. It's a bit like if Google built a factory in the tropics where they organize pools of water. Their goal is to encourage more water, stagnant or not, because their job is to organize pools... but then folks complain about the mosquitos as if they weren't invited. Welcome to the jungle.

But you have heard of me!

-- Capt. Jack Sparrow, SEO guy

> One reason that it's so hard to find an SEO person that's actually worth anything is that if they were really good at what they did, they could just throw up their own websites with ads and make a much better living that way than selling SEO services. They don't of course, which should tell you something about how good they really are.

It's incredibly expensive to rank highly for something. There are tips and tricks to make your site amenable to the search engines, but it takes a lot of time or money to get it to actually notice you.

Education is the path out of this: advertised products are ALWAYS worse value. You are overpaying to fund convincing the next sucker.

The market is pretty efficient, and there are always alternatives. If you aren't lazy, you can save a huge amount of money by not buying advertised products.

On the rare occasions I see an ad for something cool that I want, I search for it on aliexpress and 90% of the time find it for 1/3 the price.

Whenever someone tells you that they bought something because of an ad, you should make them feel a little embarrassed about it. There should be mild social punishment for admitting an ad hijacked your brain and made you get out your wallet - it means you're a sucker, a rube, a feebleminded lazybones.

It's been fun watching the score of this post bounce up and down every time I refresh. People who earn their living off ads do NOT like being reminded that they are building a societal cancer.

It's not only that, I think. Traditional full page ads, for instance, had a signal value:

"We may not be best. But we are credible and big enough that we make a shit-ton of money on our product and can afford this incredibly expensive ad."

Today, an ad possibly just means "Google matched two suckers with each other and will pocket the difference."

How do you get eye balls to educate?

While i'm not naive anymore (i hope not, there's no way to tell lol), and i know most business is simply a scam/based on people's delusions or addictions or other vice without providing any value, the ad industry can't stop surprising me. In all my career i never seen a case of any paid ads actually getting people anywhere. But they continue to spend, and spend, and spend.

Retargeting may be the only exception, although it is annoying as hell, it won't let people forget about your stuff (and comes very cheaply because very few people will ever see it), so probably generates positive returns... But i'm not even sure about that one.

The internet as you know it would not exist without ads.

That would be awesome.

It would exist as we knew it back in the 90s / early 00s. It would consist of Wikipedia, McMaster Carr, BBS forums, and weird guys' hobby blogs. It would be one million percent better.

Webrings, the original Web 1.0 that many gush about on this site, were partially designed to farm adsense revenue.

AOL started putting ads on your dashboard less than 2 years after it launched.

There wasn't a very long period of ad-free internet.

Static banner ads with no (user) tracking aint such a bad thing IMHO.

I wonder how this could be implemented at a technical level. How do you avoid gaming the system?

tongue in cheek, I don't remember any ads in gopher sites, back in the days.

This river which has turned into a giant gutter would not exist without the filth.

True! That's the problem with ads. They changed the web for the worse.

Don't you go threatening me with a good time.

The web is still a zero-sum game without money involved. Search has 5-10 slots for whichever topic and if you're not in it, you effectively do not exist.

This was even true before search, in web directories. It has a limited amount of slots, you need to be on it at all and as high as possible.

> The web is still a zero-sum game without money involved. Search has 5-10 slots for whichever topic and if you're not in it, you effectively do not exist.

Well that is a ridiculous problem that only exists because of Google's UI.

What if, for instance, thet slightly randomized search results?

Right. If we didn't have ads, then nobody would illegitimately get traffic to go to their website LOL total crap who comes up with this shit

Have you seen a 419 scam? Doesn't have any ads. Just a direct intro to get scammed.

The thing about such scams is that they require input. Somewhere along the chain, a human user with a wallet has to send personal or financial info.

> The root cause of all this madness:

> Ads.

> As long as there is even a marginal financial incentive to get human eyeballs on websites (as opposed to the viewers actually playing for services or, you know, just hosting websites for free because they are cool), this will keep getting worse.

Search will only get worse for google users, not for google or google customers.

Google can easily produce only quality search results, by penalising a result by the number of ads it serves.

They don't do this.

They can identify the SEO people and companies, and heavily penalise their clients.

They don't do this either.

They can, because you're logged in, allow you to maintain a blacklist. They don't.

Trust me, for google this is an easy problem to solve; it's just that, for google, this is not a problem, but works as intended behaviour.

Meanwhile on Kagi:

The first five results for me was this SEO spam. I went in and blocked those domains. Now it's all good stuff again.

I remember how quickly Google removed the ability to block domains, once they added it. I think that was the peak of its usability.

It was removed because it was being gamed from the outset. The workable alternative would be a 'Social Credit' system where some domains can be blocked if enough of the 'right people' want them blocked.

Google running a social credit system may have unintended consequences though.

> It was removed because it was being gamed from the outset.

'Gamed' as in people would block the most ad-infested pages which just so happened to be the ones which made Google the most money?

No, 'gamed' as in mass blocking one's competitor's results. If you think ads are bad you should take a peep into the world of 'negative seo'.

I can assure you that any solution you can think of that will improve web search will be used in the exact opposite way you imagine.

You can allow individuals to customize their own search results without having to use that more broadly.

I do this now. There are plenty of chrome plugins to block domains from the results.

So Google was using the blocklists as a signal of site quality?

Seems the problem's not the blocking itself then, and the whole feature didn't need to be removed, just its use as a metric.

Was that feature not akin to the report button

I realize this is anathema to Google, but they could always pay a human being to investigate the domains that receive the most spam reports, and make a decision about it based on human judgment.

One wonders how much bullshit might have been solved through a .005% increase in Google's payroll...

Having worked at Google for almost 10 years, I can say that almost any solution proposed that includes "we'll have a person look at this with their eyes" would be instantly discounted as unworkable. The reality distortion field towards automatic everything with computers is intense.

It makes sense for Google, the company has learned that they can focus that intensely on reducing costs and get away with it. I think it is our fault as a society (or perhaps our government's fault) that we have not created an environment where the competition is stiff and Google needs to invest more heavily in quality to stay relevant.

That is literally how their search ranking team works. Here is a search strategy, here is another. Tell me which is better based on a bunch of random queries.

For spammy websites, you don’t review every one individually. But you have an algorithm that e.g. downranks them, and you AB test it.

They probably already have this strategy now, just not deployed for one reason or another. (For example, it might be expensive and unreliable to scan every website.) But if Google loses enough traffic to a competitor, they’ll be forced to do something.

How much review would it take? How many websites until you get to 99% of traffic?

Unless they used those blocklist to influence their normal search results, there is no way to game it. That choice was up to Google.

"Automated search engines that rely on keyword matching usually return too many low quality matches. To make matters worse, some advertisers attempt to gain people’s attention by taking measures meant to mislead automated search engines. We have built a large-scale search engine which addresses many of the problems of existing systems. It makes especially heavy use of the additional structure present in hypertext to provide much higher quality search results. We chose our system name, Google, because it is a common spelling of googol"

OP will soon be ranking for the exact phrase he discusses.

(If we discuss “a comprehensive ecosystem of open-source software for big data management” enough, perhaps we can get this thread to rank too.)

I am really hoping so. my last paragraph:

    PS: secretly, I hope that this post starts ranking for “a comprehensive ecosystem of open source software for big data management”, which is why I have said it verbatim so many times and added a helpful callout at the top for students. To be honest, I'd settle for the 19th spot: just above highadviser.com.

You're in the 14th spot on Duck Duck Go as I write this comment.

https://start.duckduckgo.com/?q=%22a+comprehensive+ecosystem...

Edit: DDG's results are not stable for me. I get different results every time I reload. Sometimes it comes up second:

https://imgur.com/a/YyqI2Z4

Position 11 in Google if I use a private tab.

Position 1 on Kagi (after a link to Apache Hadoop, and a box with a “discussion” link which happens to be SEO spam).

Especially as it's received a backlink on a high-content page from a high ranking domain like news.ycombinator.com!

Stole my thunder! Amazed your comment regarding “a comprehensive ecosystem of open source software for big data management” hasn't appeared on page 1 yet. It's been a whole 7 hours...

> this one of the 300 niche sites im building

Like bragging about throwing the most empty soda cans out of your car as you drive down the highway.

The whole Cisco courses, blog spam, SEO industry are some of the worst things to exist in technology. I hope all of them gets burned by a massive fire. Who even thinks of becoming a SEO consultant?

SEO is the player, ad-funded search is the game. Hate not.

Why not? It's totally valid to hate both the game and those who are willing to play it. They perpetuate the game, after all.

> Cisco courses

These Cisco certifications are such a scam tbh. Cisco commands and CLI are the most confusing, illogical designs for any feature. And they make sure to put in tens of thousands of features and hundreds of ways to configure every different feature. Then they create these overpriced courses and pay for their own employees to take every single one. Since Cisco employs a lot of people, and prefers people with Cisco certifications, this creates a lot of demand for the courses.

Most of the certifications could be replaced with a single 2 month networking course that explains the workings of features, provided they redesign their CLI to be slightly usable.

"SEO is also a zero-sum game."

One with external costs imposed upon others.

I had a go at fixing this with Sitetruth, which was an attempt to tie web sites to real-world companies rated using info from Hoovers, the SEC, DNB, etc. But the whole concept of tying web sites to real world companies now seems dated.

I am waiting for the era when SEO world is totally disrupted or destroyed by AIGC

The SEO world is gonna love AIGC. I'm sure it's already in wide usage there; it makes the convincing-but-fake review websites trivial to stand up in quantity.

It's the real content that's going to get disrupted/destroyed.

Yes, I know a few SEO/SEMs. They're ecstatic at the moment with how much easier and cheaper AI has made it for them to pump out shitty content.

This is already underway and it's already the case that Google is sending you to a bunch of mediocre articles that were written mostly by an AI. The real question is what happens once that trend compounds over the next 2-3 years. It is working right now so it is going to 10x.

AIGC is absolutely already here. Recently I was trying to find information about a miter slot on a piece of equipment. One of the more helpful articles had a quick shout-out to miter slots in the middle of a listicle about casino slot machines.

It's been in use since at least 2006 that I know of. That's when I was first let onto it by a friend running a big content farm. Now it's absolutely out of control. I use it myself generously to help me write articles, but you have to be incredibly careful because it so frequently hallucinates. You have to have some knowledge of the domain you are writing in to peep when it is going full HAL9000 on you.

The Internet now feels like this to me:

https://www.okcu.edu/admin/communications/web/wharrgarbl