Hacker News Reader

Comment by sph | Hacker News Reader

> On the other side, I've also noticed it appears to be aggressively pruning its index in the past few years, so the fact that it's crawled your site doesn't mean it's necessarily searchable either.

I've noticed this as well. I have a crappy website for my app I need to do better marketing for (not my priority just now), but I've noticed that, for however crap it is, I have received ZERO incoming hits from Google, apart from a couple people that have literally just googled my domain name.

I do not believe for a second there's not a single query done in the 2 months the page has been up, globally, for which my website wasn't a bit relevant. Either that, or the spam problem Google has is much bigger than anyone thinks.

Yet another data point in favour of the Dead Internet theory.

View on HN

You could try the google search console - it gives you a view on what hits/clicks have come in over time.

edit: Hah. I notice it suggests using it at the top of the page if you use 'site:..." - and I only get 5 results for my site when the console claims to have indexed 10 times that many!

edit2: Also duckduckgo returns more like 15 hits ...

Silly to see people complaining about search results and indexing without backing those claims with data from search console. It’s like devs turn off their brains when it comes to marketing because they don’t like it.

Google's bloody Search Console says I got 16 impressions in 2 months for literal searches of my domain name, and nothing else. Funny seeing people thinking I got those figures by reading tea leaves.

Who's the silly one now?

I have all sorts of things that I wrote years ago and I can never find them searching by title unless I put the specific name of the site in the query. I sure can find the slideshare though where some guy from Oracle stole not only my title but much of the content from my blog.

Dead Internet theory?

From Wikipedia:

"The dead Internet theory is an online conspiracy theory that asserts that the Internet now consists almost entirely of bot activity and automatically generated content, marginalizing human activity. The date given for this "death" is generally around 2016 or 2017."

Not sure I'd call it a conspiracy theory.

> Not sure I'd call it a conspiracy theory.

What I find funny about that framing is that, regardless of whether or not the theory has merit, a conspiracy theory by definition asserts that there exists two or more people conspiring with the intent to produce the alleged outcome. From what I understand, dead Internet theory alleges no such collusion or intent. I could be wrong but I believe that it merely suggests that the amount of bot-generated activity has come to dwarf human generated content to the point where the Internet is effectively "dead" from the perspective of its original purpose: humans sharing human knowledge.

10 or so years ago I wound up blocking everyone other than Google in my robots.txt because I was sick and tired of webcrawlers from China crawling my site twice a day and never sending me a single referrer. Same with Bing. Back when I was involved with SEO the joke was you could rank #1 for Viagra on Bing and get three hits a month.

At least so far according to Cloudflare bots consist of around 1/4 of all internet traffic. But that could be pretty far off depending on how they get those estimates.

This very link had a Cloudflare "prove you're human" screen that prevented me from reading it.

The figure I saw most recently was 42%. Weirdly my brain can remember the number but not where I saw it.

But what I'm curious about, whichever number is true, is whether people mean "malicious bots" when they say this, or just any kind of autonomous agent. And also whether they are counting volume of data or simply network requests.

Because if by "bot" they just mean "autonomous agent making a network request" then honestly I'm surprised the number isn't higher, and I don't think there's anything wrong with it. Every search crawler, every service detector, all the financial bots, every smart device (which is now every device) and a thousand other more or less legitimate uses.

I've got a script for parsing my web logs which removes all the lines which match persistent indexers/bots/scrapers and any obvious automatons. Logs generally shrink to 40-50% of their volume, so I'd at least double CF's estimate.

https://www.youtube.com/watch?v=kL8rHf_idt0

Thoughty2: The internet has died

In this video they rename it from 'theory' to 'prophecy'. As in the internet isn't quite dead yet, but its rot filled bloated body is near its dying breath.

Same issue for me. Message is clear: be relevant to be indexed. And everything indexed is relevant.

use intext:

Reply on news.ycombinator.com