Where did the old web go? We followed 657,607 links to find out
0.mk
0.mk
It being harder to join has the effect of their being a smaller more engaged user base. Less likely that there will be pressure from corporate interests, more likely that every user has a vested interest in both production and consumption.
edit: And of course, you'll see the same in dedicated niche forms and spaces like HAM radio. I imagine the folks TXing EME have a particular flavor that is well maintained.
"Things used to be so good" is mostly rose-tinted glasses: What exactly WAS the "old web" anyway?
100s of half-baked sites hosted on Geocities, Yahoo, about pointless stuff? Epilepsy simulator websites splattered with gif-vomit? Everyone and their grandmother asking you to install their 10 toolbars?
Honest, serious question: What actually was of objective substance on the old internet that's nowhere to be found now?
You can find random pointless stuff now too, just that except Geocities/Yahoo it's Intsagram/TikTok/Twitter etc.
If you mean self-hosted websites, they're still here, and nothing's stopping you from creating your own.
If you loved all the Flash toons on Newgrounds etc there's unironically a lot more shorts and animations on YouTube or even Vimeo now, if you but search for them (the old gods like Weebl, David Firth are still there, and I suggest Sechi, DoodletmeGo, Nondescript Video Club, and からめる to start with then let the algorithm soak up the weirdness)
Just as the "internet" supplanted newspapers, magazines, and broadcast television for many people,
and how the newspapers replaced the town criers before them,
why shouldn't the "internet" be supplanted by a more accessible medium?
There's so much crap on the "old" and "middle age" internet: ads, scams, anal mods, trolls, bandwagons, vote wars, low-effort content and other repetitive noise that could be bypassed by just asking AI about something and make better use of your limited mortal lifetime, like watching that anime you've been putting off for years.
The old web was local forum created by gamers for gamers who hosted real life events. You could post asking for super specific advice to your locale. It was eaten by reddit.
There were cool IRC servers for all sorts of niche projects and topics. This still exists of course but a lot of it was eaten by Discord or Slack.
While I never participated, sites like LiveJournal provided both a unique outlet and view into other people's lives. I think a lot of social media like Instagram and Facebook eat this up.
I'm sure everyone has their own versions. And this is the old web that was gobbled up by mega sites which aim to commericalise the crap out of everything.
Btw I consider HN part of the old web.
But..what says something like Diaryland can't be brought back now?
Back then there was a forcing factor to making your own website. Not just personal sites, but bands, artists, interest groups, etc. Now everyone just uses facebook because (or whatever) because it's easy and everyone is there. Even during the transition was better as myspace allowed you to customize the crap out of it with CSS and, as we were just discussing, DiaryLand gave you full access to do almost whatever you wanted (in the confines of static pages).
It has a LOT to do with billion dollar companies throwing their weight around and influencing politics. Tom never did that shit! (although given time, who knows... he did sell out).
Pre-Google, specifically, pre-PageRank, and even more specifically, pre-SEO.
Your post is oddly hyper-focused on pointing out the negatives, probably due to the asshole hat you put on. There were lots of good sites with tons of good, free information, and because the main way people searched was using directories, we weren't inundated with 1,000s of results on the same topic, vying for a top-placement. The writing itself was much more laid back because SEO-optimization didn't exist. Also, those "half-baked GeoCities sites" were awesome! Of course there were a lot of junk ones, but again, it was just a bunch of people writing about themselves, their interests, and generally being creative. And again, NO ONE WAS TRYING TO MAKE MONEY, they were just doing it. No "like and subscribe." No gaming an algorithm. I used to love finding guitar tabs on those sites and I had a couple of sites with lots of tabs I did on my own for two different bands.
> There's so much crap on the "old" and "middle age" internet: ads, scams, anal mods, trolls, low-effort content, that could be bypassed by just asking AI about something and make better use of your limited mortal lifetime.
I feel like this was GP's point, no? Because people are just using AI for search now (this is the main reason I use it) that there will be no point in making these spamming sites anymore, which "opens up" the internet a bit for those of us who enjoyed the old way. Yes, it's all still out there, but it's kind of hard to find, and it would be kinda neat if more people discovered making their websites outside of walled gardens.
because people are hyper-focused on the negatives of the "current" web, or acting like the sites they used to love can't be recreated anymore..
> it was just a bunch of people writing about themselves, their interests, and generally being creative.
There still are. on YouTube, TikTok, Instagram, Reddit, 5ch and other Japanese subcommunities.. the fact that they're all "silo'ed" into these less-than-benign corpoctopuses is another matter, but people are still people'ing
this comment brings up a good concrete example: https://news.ycombinator.com/item?id=49293553
Basically: "everyone has a different idea about what the Old Web™ was depending on when they first started using the internet, and when the community they liked was discontinued or changed"
Fair!
> There still are. on YouTube, TikTok, Instagram, Reddit, 5ch and other Japanese subcommunities.. the fact that they're all "silo'ed" into these less-than-benign corpoctopuses is another matter, but people are still people'ing
I know there are, but as per my comment, it's just harder to find and you used to almost not be able to avoid it. I don't think platforms like YouTube and Instagram are great examples due to the monotization... everyone is trying to "make it." But ya, I mean, I do know it's out there. But I think the whole point wasn't that this stuff doesn't still exist, just that maybe it will become the more prevalent thing again. At least that's how I read GP's comment. But who knows.
Or: because people are just using AI for search now, the only websites left will be spamming sites trying to rank in/influence AI results.
Think about it: if human traffic is no longer optimized for, and the entire focus was catering to bot traffic, how amazing are you imagining that those bot-optimized sites will be?
On the old web, you would just stumble onto something deep but the barrier to get to it was fairly high. It could also just be total nonsense but at least you could see the passion behind it.
I agree with your general take. People talk about the old web as 'the good old days' but it really wasn't that great. I do have some fond memories of the mid 2000's when it was all starting to mature but also before corporate capture took hold. Even then, you would just end up stumbling around blindly most of the time.
There are so many piracy discords where you can't ask how to pirate something because that's against discord rules. I mean really? Why are you using it then?
Dont compare wanting a pro terrorism fourm with niche interests its so disingenuous.
and don't forget Big Billy's "no more tears" guarantee on all anal mods purchased from the official BBAM shop!
The old web culture can't exist because advertisers won't let it exist on modern platforms.
Additionally the spread of information is significantly less organic because of relentless pushes to use algorithmic feeds along with far worse distractions from content than there used to be.
Again, what's preventing you from buying a domain name and putting up your own website? bypass the advertisers?
"Not enough people will see it", you say? So you want the reach provided by "modern platforms" but you don't want their rules?
Yes you can still find roughly all the same stuff. No it's not the same.
No it's not the same.
When was the last time you actually visited any of the Old Web websites? I just found that Diaryland and Halfbakery are still around, over 20 years old, but it's been years since I visited them
Even though there was a lot of forced enshittification, the tech did not really force people to change: it's that fewer people feel the need to be the way they used to be.
and that can change again
I actually prefer a linear view with posts that quote what they want to reply to. This also allows a single post to reply and combine multiple remarks. So it allows more for convergence instead of divergence as with a tree.
Whoa, careful with your verb tenses! You just wrote a ghost story :)
its interesting because it sounds more correct to use have not yet because your life is still going on but its going on in the forward direction, not backwards in time to overlap an earlier HN state. So you could say its not right, but the sentence sounds/feels more right to preserve that your life is still going on.
We like that you are still here with us.
Wouldn't the proper tense be "I won't have lived long enough to see the entire lifetime of HN" then?
> We like that you are still here with us.
Same here, and it feels like a very "old web" kind of feeling, as we're expressing this empathy towards an anonymous stranger with a very deemphasized username.
I find it surprising how much empathy has seemingly been on a decline as the Web went from anonymity to pseudonymity to everyone's full name and face being attached to their online presence.
This one is almost gone already.
But I understand the OP was talking about how HN is today.
It gradually became a comedy website and at that point I stopped visiting it.
But now I think that downvotes have just kinda become a sort of digital tomato for people to throw, a means of burying a post without putting any thought into an argument as to why the post is illegitimate.
And that's the best case. In the worst case it's a really easy system for bots to game. You don't even need AI. All it takes is a bunch of accounts "legitimate" enough to not get flagged by whatever botting heuristics the site implemented and you can produce a false consensus.
HN is a certainly a smaller but more targeted market but there's no reason to assume we're communicating with other humans here necessarily especially given HN demographics.
Basically every area that is either forgotten by the mainstream, or completely unknown, and make sure the author is a person and not a company and you'll be on your way to way more interesting discussions, texts, content and interaction, that the B.S. of facebook, twitter and commercial web sites.
So when people construct versions of the old web today, they make a conscious decision to do something most people would not, and by doing so it sometimes comes off as indicating what they think of the other design decisions they chose not to make. It's not a bad thing, but it's not quite the same as it was anymore. That is why some of the spark is gone, in my eyes. It's kind of like counterculture being mistaken as the opposite of culture.
It is even more pronounced when people talk about "the old web" on old web websites all day.
TBH I do recall a lot of gloating over how AOL was full of clueless lusers who were inferior to the Internet
BBS / FidoNet, Usenet, IRC, and MUCKs would like a word!
Mind you, 200 years later, we still live in cities built for cars, not people. Rockefeller, Shell, BP and the like are still not dead, and still use everything they got to not go extinct while corrupting every politician that will accept their money. Just for a couple more years at a time, and a couple more, and a couple more...while destroying our planet like a cancer.
In 200 years we'll still have the web built for surveillance capitalism, not people.
There is no old web to go back to. Meanwhile state surveillance depends on its existence. Amazon, Apple, Google, Meta, Microsoft, three letter agencies, Palantir, and now the AI companies depend on it, and they're selling to nation states at a time. Again, it's just too juicy not to.
Back then we called them trusts, but now those big tech companies are seemingly worth more than actual countries. That's too much power without legislation, as they can effectively abandon the rule of law. Remember what happened with the Enron scandal? That was about billions, not trillions.
If we want a human web again, we need to build a new one, with privacy and secrecy first and not as an afterthought. No CA, and with mutual cryptography that acts like both a ledger and as a privacy shield for the users. No cookies, no tracking, no degenerate google controlling it all through owning the source code.
Blue skies mentality. The old web was good because it wasn't crowded or accessible to most, and everything was new. That, great filter, was unique and will never happen again; Pandora's box is open and cannot be shut.
Pretty much akin to one's childhood being yearned at old age...
One difference I see is that in the (so-called) "old web" era, if you were not nerdy enough to navigate certain technical issues, the path of least resistance was to not participate. As you say, those people stuck to phone books and newspapers and TV. That meant that entering the world of the internet was adding something; you could still read the newspaper and watch TV (and most people did), but you had the internet also.
The insidious achievement of the Facebooks and TikToks of the world is to make things smooth enough and make participation widespread enough that nowadays its harder to not participate, somehow, in some facet of online interaction, than it is to participate. By doing that they created a flood of junky stuff, and now AI has come along to amplify that approach many times.
This means that now, what the nerds are trying to do is not to add, but to subtract. In the old web, you went to the web to get something you couldn't get from traditional media or communications. But now, if a "new old web" is created, it will largely be to not get things you are effectively being forced to get from the flood of social media and AI content. That doesn't mean it can't be done, and in a way I hope it is done, but it means it's going to have to be qualitatively different. Before, the walls kept you in and you hopped over to get out, but now we're all out in the wilderness, and if people build walls, it will be to keep the wild world out and keep yourself safe inside.
The facebooks of today make it effectively impossible to self-curate attention. Social media is the publishers early internauts tried to get away from.
I disagree with your last paragraph, though. The "old web" was a small walled garden. The "post slop" web is going to be very similar small walled garden, that is chased to get away from uninspiring everyday.
Here's a free idea for anyone hoping to launch anything like this: implement a MLM style transitive web of trust. The forum is invite-only. x strikes - you are out, but the strikes on invitees transitively decay over the invitation chain. If you invite someone who then invites all the normies who get striked out - eventually that strikes you out too. Invite yourself to shitpost - eventually you will get punished for that. Yes, it's bad as it can exacerbate bubbles as much as the current facebooks do. Simple updoot mechanism suffers from brigading, but something can be worked out.
I don't think the old web was really a walled garden, though. It was just a sort of isolated garden that was harder to get to. There was nothing outside it threatening it. Now the garden needs walls because there are a lot of dangerous things outside.
The new Web is for the median human. The median human is very ordinary.
Today, if you own your own domain and fiddle with HTML then basically you're still part of the old web.
So basically, the old web required experts to help people get online, the new web negated them through homogenization.
The real experts from the old web basically became startup founders to monetize the homogenization and automation of their worldviews for the less tech-savvy (or specialized) folks.
People pine for the old web, but it was a churning, friggin' mess too. Search results were good though, but only when Google arrived. The standardization and ease-of-use from the new web has been a massive boon. I think things will change as people yearn for a wild-west, where there are new opportunities. The promise of a better life, seeing that the grass is greener, avoiding pain - those things always create golden ages.
Once AI favours people who can afford to pay for the best results.. or surveillance becomes too burdensome or dangerous, then watch this space.
The spirit of freedom always runs into trouble, when considering the notion of:
"Cut off the heads of others to make oneself appear taller."
Surveillance enables this. Decentralize again, but how?
ISPs - help or hindrance? Why/how can ISP charges be diverted to content creators?
https://gwhatchet.com/2005/10/13/professor-describes-influen...
Hit counters on the front page was mandatory of course.
I'd consider the facebook and mega era to be relatively new, the "old web" for me would be the one without centralization around a few giants, the era of random phpbb forums, private websites with "this site is under construction" banners and internet directories to find stuff.
2009-2014 is Web 2.0, back when people still thought social media was a good idea.
PNGs? We had (mostly static) GIFs and we liked them.
Two of my favorite sites [0][1] still online—Lurkers Guide to Babylon 5 and ex-astris-scientia—were started in ‘92 and ‘98 respectively.
Elegant content for a more civilized age.
With that said, its domain has changed a few times iirc… so maybe not the best example given the nature of the article.
But maybe that's more a measure of my own age and perceptions rather than an accurate representation of the various eras of the internet/web...
And now I wonder if Apache is even still common. I spent so much time fiddling with apache configs.
That random 20 year old video of some middle schoolers doing a flying kick and breaking a vending machine? Yeah, just popped up on my feed yesterday.
Now people don't give a shit about URLs. Webmasters casually move files around all the time because they feel like it, and if links get broken, who cares, that's the referring site's problem! Their beautiful file hierarchy is more important than the web staying connected!
I am not saying old content needs to be preserved forever, but so much content has factually been lost over time. Old logs from text-based MUDs for instance, even for MUDs that still exist today.
We hot-linked to all those image hosts because we couldn't imagine them disappearing.
Archive.org had incredible foresight and if it didn't already exist, I'd call such a project a pipe dream.
No, we hot-linked all those image hosts because we didn't want to pay to host it ourselves.
I enjoyed the emails I got from a couple people who were adamant I "hacked" their site because their "web developer" linked to images on my server... images that now said stuff like "I'm a loser bandwidth thief!", etc. (I never did use really nasty "shock" images-- just taunting stuff.)
This is a bit idealized. In practice copying data is not quite accurate (especially in bulk) and bit-rot is a very real phenomenon, both in flight and in storage.
You sometimes encounter it when dealing with files from the early '00s, it's very common to discover a few of them are corrupt, even if they've only ever been copied between harddrives.
It's copyright that causes anything to disappear from the web IMO -- torrents never die.
EDIT: I am aware that unseeded torrents do in fact die. But it really doesn't take much to seed a whole hard drive's worth of rarely requested data -- this also detects bitrot and so corrects errors automatically if you're not the only copy.
If you are, there's ECC, as well as making another copy.
There are many TinyMUD logs that were posted on Usenet, still to be found on Google Groups.
However, logging was controversial amongst mudders. It was almost always rude to log a private conversation without knowledge or consent; it was tacky to indiscriminately log while everyone was in the "hangout room" or Rec Room, as it were, and it was also bad form to post logs to Usenet or share them without redacting player names and other things.
But logging was built-in to most clients, and it was possible for server administrators to log (and hypothetically any malware-in-the-middle could log the cleartext, unencrypted TinyMUD TCP streams.) And many nefarious deeds by nasty players were exposed to the light when their logs were posted.
I'd say it's one of the biggest design flaws of the current web, what with more and more content hidden behind paywalls, increasingly restricted WAFs, and rendered client-side via convoluted JavaScript.
Archiving and mirroring of old-style websites, delivered as static HTML, is simple and straightforward. 20 years from now, most web content from ~1996 to ~2015 will still be accessible, but much of today's web content probably won't.
Let me introduce you to the alternatives: print media, film, stone engravings. That stuff tends to get burned and shattered and it takes FOREVER to make copies.
I'm being cheeky but I don't know what design change you could possible make to the web to make webpages not die.
0.mk, you had one job…
The plot features two American men who stumble upon Brigadoon, a mysterious Scottish village that appears for only one day every 100 years; one man soon falls in love with a young woman from Brigadoon. The show's song "Almost Like Being in Love" subsequently became a standard.Anyway Britney's 1998 debut album predates The Matrix, includes the song Email My Heart and came on "Enhanced CD" with tags and video. How about that
For me, this doesn't indicate that those young folks don't know what they are talking about. Rather, that missing the "old web" is more a cultural thing in itself. We got introduced to the web and discovered these communities we became part of, but something changed and our community died.
There had to be a cutoff that changed something! That killed our community. In reality the change, destruction of communities happened all the time. Some survived, some changed that returning to them wouldn't feel the same, some moved. If you don't feel part of that community anymore, it is destroyed, even if someone else thinks it is still alive and just moved to new-forum/teanspeak3/dig/reddit/discord/tumblr/facebook/tiktok/next thing.
The small Ultima online private server I used to be part of died, because a few people moved on with life, even though it was still there for a long time. Not because of WoW. My WoW guild suffered the same fate, despite Wow still being there. Now the few people I became close friends with are still left, wearing the guild name as a server name on discord, but that's a very different community.
Even if the exact people suddenly came back, it wouldn't be my community anymore.
My old web can't come back, because it is not a matter of technology.
I am sure the people that join the web today will lament the death of the old web of today in a decade.
Also:
> Reply to any 0.mk email and the message lands in a feedback queue the AI reads, triages, and acts on
Is this dangerous? What about jailbreaking AIs and having it delete everyone's account?
The article doesn't seem to describe the cost of the AI solution. It does imply that it is lower than the cost of maintaining and supporting their service manually.
The Old Web was, as most people here state, left behind in favor of the Walled Gardens of social media. The barrier to the watering hole was so low. I don't blame technical people for following their friends and family into these places. It kinda sucks that they let their marks on the web fade and expire.
That didn't have to happen. My personal site is going to live on the web as long as I do, and hopefully further if I can manage it. I don't care how much traffic I generate.
Also it's super easy to tell by looking at it, way too many LLM-isms. No need for a AI checker tool.
Did the zero ever mean anything? It's still a 3 character domain regardless if the first character is a zero.
0.mk started in 2009 as a passion project built by three of us. We worked on it for a few hours each week around our regular jobs. We eventually closed it in 2014 because the revenue (hint: no revenue) could not cover hosting, development, and the constant work of fighting spam and reviewing abuse.
The recovered historical corpus contains 657,607 links. For this analysis, we followed every one of them.
Of the 655,178 links with safe, crawlable targets, 76.7% no longer returned a loading page. After removing repeated destinations, 78.7% of the 492,620 distinct crawlable URLs still did not load. So duplicate links are not creating the result.
I use “did not load” rather than “gone” deliberately. Some URLs returned 403 or 429 and may have blocked the crawler. Pages that returned 2xx or 3xx count as loading even when they now lead to parked domains, login walls, or removed-content notices.
There is one large distortion in the yearly data. A single account created 83,398 URLs pointing to one hostname in 2011. At URL level, 92.5% of that year did not load. Count each hostname once and the result becomes 61.7%, almost identical to 2010 and 2012.
A few things I did not expect:
- 835 restored links point at Facebook’s old photo CDN. None loaded. - The first link ever shortened was a CSS stylesheet on a WordPress blog. - Someone shortened localhost on the second day. - The longest stored URL is 38,753 characters and repeatedly says TRYING_THE_MAXIMUM_URL.
Most users came from one regional online community, so this is not a census of the whole web. It is a record of what that community shared between 2009 and 2014.
I brought 0.mk back to test whether AI can now handle enough development, spam filtering, abuse review, monitoring, and support to make the service sustainable where the original economics failed.
Happy to answer questions about the crawl, the old data, or the rebuild.
“The name of the .mk domain consists of a minimum of 1 (one) and a maximum of 63 characters.”
Intriguingly, the current Republic of North Macedonia (and its neighbors/constituent parts) has undergone a protracted and contentious naming dispute of its own.
Have an LLM “guess” random URLs seeded with words from a dictionary, iterating over each word and guessing a URL.
It guesses a lot of correct URLs. This is one method of “URL hunting” that doesn’t involve a 3rd party list or index.
Then just scan those pages for other URLs, visit them, and add a tally every time you come across a URL (for page rank).
Then search anything, see what the results are. You have invented a dark web search engine.
Guessing random words and .onion at the end probably won't get you anywhere, although onion is also quite a random word...
The deep web,[1] also known as the invisible web[2] or the hidden web,[3] is the parts of the World Wide Web whose contents are not indexed by standard web search-engine programs.[4] This is in contrast to the surface web, which is accessible to anyone using the Internet.[5] Computer scientist Michael K. Bergman is credited with inventing the term in 2001 as a search-indexing term.[6][1] https://wiki.archiveteam.org/index.php/URLTeam#cite_note-1 [2] https://lwn.net/Articles/683880/
Like web pages that load slower with high speed fiber than the old web did on 56k and ISDN connections?
Those fruits are fermented enough to score a DUI.