The AIpocalypse is here for web sites as search referrals plunge

Are those passionate/nerdy people still running niche websites? A search engine that prioritizes that kind of content would be great.

Back Then (tm) you didn't have to fear getting hacked all the fucking time. Shodan didn't exist, Metasploit was in its infancy, you needed to know where to look to buy DDoS and even then, half the offerings were scams themselves. The biggest threat was people abusing your SMTP server.

Today? Run just a plain Apache without a domain name, just a plain old IPv4 and listen on ports 80/443. In seconds you'll get hit by a barrage of exploit attempts, scanners, god knows what. And manage to get something successful, you'll get a few emails that say "pay us X dollars in BTC and we'll leave you alone", and if not you'll get booted off the net if you don't hide behind Cloudflare.

Discovery is also not a thing any more. The old "rings" have long since passed, RSS feeds went down the drain with Google Reader (IMHO, that decision effectively killed off the blogosphere), and Google Search got completely poisoned with SEO.

The Internet has become the Wild West. Law makers have begun to notice but unfortunately they're so utterly braindead that fierce opposition to their regulation attempts is the only choice possible, so nothing happens. Rinse and repeat.

That web still exists and is still just as easy (read: hard) to find. I'm sure you can find hundreds of such sites by recursively asking people here. The commercial web didn't make it harder to spin up your own server after all, quite the opposite.

https://172.93.49.252

https://172.93.49.252/settings

https://172.93.49.252/submit/

This website has some forgotten www features that are becoming rare thanks to Big Tech, Big CDNs, and their followers:

1. It does not require SNI (SNI was added to TLS in 2003)

2. It publishes a stable IP address. See https://172.93.49.252/submit/

One can use an IP address _or_ a domain name (any domain name will work) to reach it

3. It is available via HTTP as well as HTTPS

http://172.93.49.252

For old time's sake, try requesting via HTTPS and then HTTP and see if you notice a speed difference

To get a starting list of potential web sites, freely available DNS zone files (public information) remain available

Every major www search engine has used these files at some point

https://czds.icann.org/

People often refer to DNS as the internet's "phone book"

If that is true, then DNS queries are "directory assistance"

The actual phone book is the zone files

Perhaps we need a wiby.me for DNS to select from all those NS and A records that go undiscovered

djbdns has a program called "random-ip"

   random-ip|sed 1q

Besides PTR queries, there is an adundance of freely accessible databases to fecth information about an IP address

This can often lead to a discovering random website

The type of website that is worthless to so-called "tech" companies and their online surveillance advertising "business model"

The new equilibrium might just be worse than the old one though.

Compare to "According to Prince, OpenAI's ratio of pages crawled to visitors referred has gone from 250:1 to 1,500:1 while Anthropic's ratio has gone from 6,000:1 to 60,000:1."

I’d be stunned if my sites have as much as 1% non-bot traffic, even outside of the ones that are vaguely straddling the border between bot and Ddos.

I think even Lemmy and BlueSky are mostly bots. As much as I loved the anonymous wild west of the internet in my youth, I have come to accept that that is dead, and AI/LLM/bots will finish off its carcass. I foresee a future where humans are verified using some kind of trusted mechanism so that we at least know when a website, content, or comments comes from a real person. I value this far more and I think so do most.

AI is still bad at giving enough information and won't preserve the author's tone. I'll still take searching on Perplexity but reading the details on (for example) https://ridiculousfish.com/blog/ over reading only the dry, possibly wrong summary. Blogs will be ok. Maybe the spam blogs will die, which I'm happy with.

Most countries have lacked the capability to manufacture basic infrastructure for ages. It may look scary to you now, but it has been looking scary for decades to some.

Face it, we depend on a global economy. Every now and then something changes, and we adapt.

Exactly my sentiment. The web being crummy is not because of AI. The mainstream web - Google search and ad-infested web sites - was already crummy. AI has a higher tolerance for the sea of detritus than I do, so I lean on it to avoid that garbage.

Yep. Nowadays even if you try to do your research on something like a foam mattress/bed in a box… be aware that the same company owns virtually all the ranking and review sites as well as all of the dozen brands they’re pretending to compare. Even the old method of appending Reddit to search queries hasn’t returned genuine results for the last few years since companies know to use aged and otherwise organic looking accounts to subtly name drop themselves and reply in agreement throughout threads alongside keywords that will bring in google traffic and upweight their odds of LLM mention.

But it won’t help the real useful sites either.

So we just get more losers and no winners

Exactly. Every time I follow a link to a link to a source I'm rewarded with a huge wall of text that finds ways to keep repeating the same phrases over and over to MAYBE reveal a little nugget that you need. Talk about having your dignity robbed.

I would add the search of an error message while developing. Mostly using the search engines for that.

But hey ! Now you can ask ChatGPT, Copilot or whatever for the same thing !

We are surounded

The only good outcome of LLM content generation for society so far it seems. I remember some recipe site folks were here on HN complaining about ad blockers and it was hilarious following their reasoning. It’s ok to manipulate the entire internet if you’re a business but you’re not allowed any control individually as a consumer I guess. I’m sorry, but filling the internet with 90% fluff garbage doesn’t entitle you to any sympathy.

This is a great observation. Google established rules that would increase your website's pagerank. In response to that SEO was created, and with that the experts. Tags, inbound links, and then more content.

A good example of how this spawned articles that are absolutely useless: https://www.partitionwizard.com/news/

All those articles have nothing to do with application they're trying to sell, but they do (did) bump the pagerank.

Many companies do this, which has polluted search results for issues to the most common denominator, making it impossible to find that 1 post that actually has a working solution.

And now we've reached the next stage, those pages are no longer written by humans, but entire websites are being generated for the purpose of selling you stuff.

I was searching for the name of the bitter part in a peanut, and stumbled upon this website:

    https://www etprotein com/parts-of-a-peanut-anatomy-of-a-nutty-snack/

(URL is missing a dot after the www and before the com)

We have entered a situation where there is no longer 'no answer', there is always some answer out there, simply to sell you stuff.

In the past if you searched for Brand X at Retailer Y, and they didn't carry it, you'd get no results. Now they autocomplete Brand X's name, and show you a whole bunch of products related to Brand X, with no indication that they don't carry it.

It also impacts normal people putting human content on the web. It's not just SEO "experts" whose businesses or projects are being buried.

"Web search was a waste of time long before the Apocalypse."

Difficult to find anyoone who will acknowledge this given the amount of mooney involved.

One search engine website pitched itself as having a mission to "organize the world's information". However search != organisation. The information the company was searching did not belong to them. It was outside their control. It generally remains disorganised.

The mission was actually to copy the world's information, intermediate access to it and sell advertising services.

The company did not sell "organising services" to information publishers. It sold online advertising services to advertisers.

Intermediating access to information was nothing more than a tactic to asemble audiences for advertising. The mission was to sell online advertising services.

Organising information for web publication is still an unsolved problem. (The problem of vast amount of disorganised information on the web remains unsolved.) Information that is organised is more easily searched. If organised well-enough, it can be generally be browsed without searching.

Utrustworthy, unethical coomputer nerds found ways to make money from the disorganisation. They became wealthy and were celebrated as being "successful". ^1 As such, it is difficult to find anyone who will acknowledge that relying on "search", where the algorithm is properietary, secret and can be still gamed by "SEO", etc., was deeply flawed. Flawed not in the sense of it offered no opportunity for making money. Flawed in the sense it ultimately does not work well for locating the information sought by the searcher.

If the wealth from intermediating access to information via "search" begins to dry up, then perhaps the fundamental flaws of web search will (again^2) be acknowledged.

1. NB. No one has been successful in organising the information published on the web.

2. As they were in the 1990s when one search project claimed (falsely) it wanted to counter the effects of advertising on "search" and remain in "the academic realm" only to then become the dominant web advertising services company.

It makes sense that product searches might be effective because product infomation tends to be organised.

And then the question is: why? Is it because the LLM is better, or because the search engine is worse?

I just purchased a monitor, from among the cheaper 4K 27" ones I could find. I’m in India, so they may not be the same models you’d get in places like the USA: maybe the situation is better there. When trying to compare them, or even find reviews of them, I could not find anything useful through a search engine. Exactly zero useful things. Not a single review of my four candidates. Oh, a couple of video “reviews” probably in Hindi… but I have learned to utterly scorn that category as consistently devoid of content; and a couple of “review” sites… with only copies of the first-party marketing material.

This part of the web seems to be just dead.

As for LLMs, well, I’m sceptical that they actually are all that much better: where is the information coming from? I’m deeply suspicious that they just look better. It’s well-established that they’re magnificent at conning.

For my own experience, I’ve tried DuckDuckGo’s Assist a few times, which is GPT-4o, and I haven’t been impressed with it. Almost everything it comes up with is superficial, or wrong.

I think you mean SEO not SSO. I know there is company that already is working in this space. I can't remember the name though.

How would LLMO and SEO strategies differ?

People need to corroborate facts you can't do that with just a box, you need someone else's UI to shape the trust you are talking about. That can't be done by showing a simple favicon next to a blob of text.

Temporary. AI will be enshitified just the same one day.