The Small Website Discoverability Crisis (2021) | Hacker News Reader

The Small Website Discoverability Crisis (2021)

marginalia.nu

557 pointsggpsv253 comments

Neocities (disclosure: I work on it) has taken steps to try to improve small personal web site discoverability, which ends up being like a platform for people making web sites with a hybrid social component https://neocities.org

I like the idea of calling this the small web, I usually go with something like "personal web site" or "home pages" but it's never quite stuck for me. I hope they've added Neocities to the Kagi small web search because there's some pretty incredible sites available for that and our compiled sitemap will make importing easy: https://neocities.org/browse

The framing for this stuff is usually something like "wow remember the crazy 90s web" nostalgia pieces or "this is an active resistance against Facebook come join us in the lonely space nobody goes to." But really there's some incredible, magical content that requires the canvas the web provides, that isn't on the social media super-platforms and people very much still use the web to access them. Neocities alone serves hundreds of millions of views per month across all the sites, there's still a lot of web surfing going on.

I would actually argue that having a web site gives you more exposure for your content than an average social media account, because sans a few lucky accounts, most are being throttled and limited by weird algorithms to prevent people from seeing your content organically. Your google search ranking might not be great, but people share links all over the place, including in private channels (think Slack/Discord/IRC/IMs) and you can still get meaningful distribution of your content this way.

To paraphrase @izs "if you build it, they will come", is a misquote from a Kevin Costner movie about baseball ghosts, but if you build a good site with good content, people do just magically show up through mechanisms I don't myself quite understand yet. It's pretty cool to see new sites on Neocities that are unusually interesting and know they'll organically get view counts into the millions before it happens.

Re: Field of Dreams

If you look at this story from anyone else’s perspective, right up until the last few moments this is a story about a man with untreated schizophrenia or temporal lobe seizures escalating his illness to the point of kidnapping someone and transporting them across state lines.

Almost every company in the dot com boom was convinced the headlights at the end of their story would be vindication, not the ambulance coming to take them to a psychiatric ward. Almost all of them were wrong.

My mentor who inspired me to be an entrepreneur was diagnosed with schizophrenia and is basically spending the rest of his life in hospital…

I really think there’s something in schizophrenia beyond the illness. A lot People with it normally get symptoms around 27-28 but achieve insane amounts before then (same as my mentor)

> I really think there’s something in schizophrenia beyond the illness.

I don't think there needs to be any special association. "Predisposed to schizophrenia" necessarily implies "not neurotypical", and the outcome distribution for individuals who are not neurotypical is much, much broader than neurotypical.

The pinnacle of success in society has a pronounced overrepresention of neurodivergence, in the same way that pro athletes as a group have freak physical genetics.

But I would expect that there are equally many people predisposed to schizophrenia who, rather than overachieving prior to symptom onset, end up dysfunctional and battling a variety of substance addictions.

(and also I'd expect that the relative probability of these outcomes is highly affected by the strength of support networks and socioeconomic status)

I had an episode of delusional schizophrenia in my early 20s and luckily haven't relapsed. No hallucinations, just started to think everything was secretly talking about me or to me.

My pet theory is something like, my brain's dials for "avoid risk" and "recognize patterns" are turned up too high. So I breezed through a software engineering degree without ever partying, but I spend a lot of my time sitting in my house unable to motivate myself to go outside, and I'm not very empathetic (other people's words) and not very outgoing.

It's not that schizophrenia makes you smart, but that "smart" and "schizophrenic" are both functions of some high-dimensional space, and the same underlying differences can easily cause both.

On the other side, I have an elder relative who has paranoid schizophrenia and below normal IQ. Us in the tech industry are definitely going to get survivor bias from the "Beautiful Mind" cases around us.

And of course sometimes you meet those people who are smart, beautiful, rich, and friendly, with no downsides, and all you can think is ... "You son of a bitch" :P

"Recognize patterns" on high is usually an asset in our line of work.

After reading https://en.wikipedia.org/wiki/Logicomix I wondered how many of {Cantor, Frege, Gödel, Hilbert, Moore, Poincaré, Russel, Turing, Whitehead, Wittgenstein} would —given a modern DX— have been said to be "on the spectrum".

Some were wrong, but plenty were simply ahead of their time, at least from the perspective of the internet "fad" becoming a ubiquitous mainstream phenomenon.

Sure, looking back some of the ideas look silly. But when you look at where were are today and the wide range of what's popular and sustainable, some of that looks silly as well.

I just deployed to my neocities site then came and saw this comment :D

I almost got off of NeoCities recently because I thought I wanted to start adding dynamic parts to my website, but as history has shown me, whenever I start doing that I fall down a rabbit hole and get nothing done. So I buckled down and figured out how to overcome some stuff that was driving me nuts about Hugo and I'm back at it!

NeoCities definitely has a yonger-feeling crowd for the most part, but I quite like it. It's nice having the feed and discovering all the weird stuff people put on their sites. It does very well at bringing back the feeling of GeoCities. I also love how someone brought back the 88x31 buttons!

I also really appreciate the Sinatra + Sequel backend :)

I just love how Neocities has webrings. They were such a great way to find content related to the site you're currently viewing

I really like the stuff happening over at neocities :-)

Out of curiosity, do you make any metadata available? Would be a very interesting resource to have, working on making the rest of the web discoverable as well ...

> but if you build a good site with good content, people do just magically show up through mechanisms

This hasn't been true for a long time, thanks to social media downranking posts with external links, and Google downranking any site that doesn't post daily updates or heaven forbid, doesnt have SSL enabled.

A good site with good content takes time and effort to produce. And even then it will simply act as a feeder for people who will regurgitate the same information in simplified terms on a content blog (without a backlink of course) or social media. Worst case scenario, they'll try to productize that knowledge that was made available for free.

After this happens enough times, people simply stop maintaining those sites.

I'd love to see a curated awesome-list tagged github project "Awesome Small Web" to peruse.

https://github.com/topics/awesome-list

A awesome webring will be a great addition too

PS: I think shouting out neocities on HN just brought it down, lol. IT HAPPENS.

Is it possible to upload pages to neocities programmatically? I know you have a Ruby-based program to do so, but can i do it by ftp, http, or something similar?

The reason i ask is I've written (in Python) wiki software catwiki[1] that allows you to write wiki pages in Markdown. At some point I'd like to extend the program to generate a static site based on the contents of the wiki, and it would be nice to be able to automatically upload it to neocities.

[1]: https://github.com/cabalamat/catwiki_p3

> To paraphrase @izs "if you build it, they will come", is a misquote from a Kevin Costner movie about baseball ghosts

That movie was a remake of a much older movie.

What I put on facebook is the type of thing I don't want to get wide reach. I don't want just anyone to see pictures of my kids - that is semi private information that I only want my friends and family to see (and you don't want to see them anyway because you don't know me)

> What I put on facebook is the type of thing I don't want to get wide reach. I don't want just anyone to see pictures of my kids - that is semi private information that I only want my friends and family to see (and you don't want to see them anyway because you don't know me)

That's a different use case than what the GP is describing. Many people use social media, including Facebook, as a platform to build an audience of strangers.

I know, but Facebook is terrible for that purpose and so I block anyone trying.

Big fan of neocities, my personal website is on there! The neocities gem makes it very easy to update the site.

It's just two commands for me:

- jekyll build to build _site

- neocities push _site to recursively upload modified files in _site

Thanks for the link. I've been looking for something like that!

Not so long ago, every blog has a "blog roll", a list of author’s favorite blogs.

Nothing was better than have your blog in the blogroll of a "famous" blogger.

It is funny how people who didn’t live through this blog era are now reinventing it spontaneously. It’s a bit like bloggers were onto something 20 years ago, before being killed by the advertisements monopolies.

But there’s a big difference between old blogosphere and current blogosphere : old blogs had ads. Most bloggers were experimenting with it, one way or another. We were lured by monetization and killed ourselves in the process.

Younger bloggers seem to have learn about it: let’s do the same old blogs but, this time, without any ads and by actively preventing tracking.

That’s how evolution works, when you think about it. It’s beautiful.

Before the blogs, many websites had a links section as well.

+ the webrings.

Links sections were awesome, and made the web feel deeper than it really was. You could go on dives just clicking through and finding so much cool stuff. Plus if it was a hobby site, there was inherently some level of curation - I don't think anybody would be linking to any of the hundreds of lookalike SEO "blogs" nowadays if it weren't for search engines allowing themselves to be gamed.

Nowadays if it's not on the first "page" of Google (well, whatever the first group of infinite scrolling results is called) it might as well not exist. Makes the web feel flatter and less like a, well, web.

Webrings and some called them "Affiliates" (I dont know where the name came from, it makes more sense in Spanish, not sure in English), but they had this 82x32 buttons on the sidebar (sometimes anitmated GIFs) to similar websites, usually websites handled by friends.

Oh, internet was so muuch better.

Blogrolls could be either very freeform, or very topical, depending on the blog and the blog author, but they did the job--if you liked that blog, chances were pretty good you'd find something interesting in the blogroll.

I do still see "ads" on these sorts of blogs, but not at all the same type of ad as elsewhere. There are a few "ad networks" that are just free promotion for various other web revival sites, e.g. https://wsmz.gay/#misc-bannerlink

I quite love it, especially when it fits with the site's aesthetic.

I planned to do something similar to it on my blog. The idea being using that page as a public bookmark list. It would contain anything from books to blog posts to youtube videos.

https://mahesh-hegde.github.io/readings/

But I am not a prolific blogger and haven't updated it anyway for a long time.

Not so long ago, every blog has a "blog roll", a list of author’s favorite blogs.

I remember this but I also remember it wasn't called this. (Because in my local English that sounds like bog roll, meaning toilet paper, so I'm sure that would have stuck).

Ads on blog for bonus profit is okay. Blog for ads profit broke the internet.

Ads are never OK.

I tend to think these articles which have become common come from a good place but say more about someone's internet habits than about the internet. I find most social media have a profile section for "personal website". I find many such personal websites by following people's github profiles from interesting repositories or PRs. Sometimes they link to other websites. I do the same in HN, snoop around to see if an interesting comment has a link to a website in the profile. Many articles are posted on HN from personal websites, which again usually link to other websites. I don't know I feel like, if I wanted I could spend all day doing this and would have no problem finding more than hours in the day. So are we complaining about the internet or that we got stuck in the walled gardens of youtube and tik tok and so on and kind of wish we would spend more time on the "old school internet" but don't because the other part is so addictive?

Just like you said - being surrounded by interesting people on interesting platforms who are likely to create small websites, we occasionally stumble across a link in a walled garden profile.

For the average internet user, small websites don't exist. Very few Instagram and TikTok profiles feature links to handcrafted sites. Google increasingly funnels all queries to the same 500 giant SEO'd sites.

I guess the question is whether that matters. The average internet user today looks a lot different from the average internet user 20 or 30 years ago. The internet looked different but the demographics looked different as well. The average internet user primarily uses Instagram, Facebook, YouTube and TikTok because that is the internet they enjoy.

At the same time, the behemoths who operate these platforms are the ones who ruined the open web with their tracking and advertisements.

I have a tiny website. It’s a static site where I occasionally post things. I don’t have any trackers or ads.

Behemoths didn’t kill the open web. They created walled gardens which were more attractive to the 99% of people who aren’t interested in doing their own dev ops. Most people prefer to spend their time with their hobby rather than figure out how to set up a server or create ssh keys for GitHub pages.

In my point of view what's lacking is more places where curators that have found interesting small sites can showcase them.

I used Digg a lot for that, StumbleUpon was also really nice for this type of discovery, then early Reddit had a similar effect.

Nowadays? I don't know where to go, I can do all this effort of clicking around to find them but honestly I don't have the time, I'm in my mid-30s, I won't be jumping around hyperlinks searching for breadcrumbs of potential good content... A lot of people are doing that work already, like you, we are lacking a good place where we can pool this curation work collectively so others can discover it.

StumbleUpon was the best, I haven't had a better experience with finding interesting, relevant things on the internet.

Though I did find https://cloudhiker.net/ recently, which is aiming for the same thing, and I'm optimistic.

For general news: https://upstract.com or https://biztoc.com

I have more Matrix and Discord rooms/"servers" than I can read in a day. If I catch up to the chatter in my Matrix anime rooms and my Discord RPG servers I'm not going to get any work done, any work done on the RPG I'm running, or any chores done at home. This is nothing to say about the personal blogs I read, the substacks, and Reddit, HN, etc. People are starving for personal content? As you say, I think this is more a user problem.

"Small web" advocates may eschew the ethics of larger platforms, but they appear to desire the same positive feedback loops that make the larger platforms addicting.

I think that a problem for some people is that the straightforward solutions to discoverability, such as simply browsing the web in the manner you described or even what is given in the marginalia.nu article, do not solve the desire to be seen as urgently as more technologically coordinated processes.

This resonates with me.

I've posted my language practice website on HN, LinkedIn, blah blah blah several places, and I can't get people to care. I've finally got some traction on slav facebook, but only just barely. Joining a web ring maybe kind of helped?

It's free, actually really free, because it's something I love and want to share. If I post it to several places and nobody clicks on it... What am I supposed to do ?? Buy ads to hopefully get people to use my _free website_? I have tried doing stuff from SEO articles -- open graph tags, descriptions and stuff. I've posted it on social media to lukewarm reception.

Someone else mentioned something like delicious. Maybe stumbleupon. Maybe this, maybe that. Maybe some federated bookmarking thing. I think there's just been a cultural shift to "if it's not on FaceBook it might as well not be on the internet", and I don't know how to get back from that. I think most people use their computers and phones as bootloaders for instagram.

The key to creating unique content is to work backwards from the queries which don't satisfy you currently. Answer those questions and expand upon the entire category of knowledge if possible. If you start with just publishing whatever fits your fancy, you're only guaranteed to have the psychic benefit of putting your thoughts out there.

If you have to buy ads, this means that the content you are producing already has large enough pool of competitors. Nothing wrong there, as long as you have a sustainable biz model.

Understand that at the end of the day, no matter how much HN users disparage Google as a advertisement company, their core product is still search. Search is the process of bringing users to the content which satisfies their queries. We can dispute the quality of results or pine for the search landscape of yesteryear, but the core premise remains. Google still needs to produce a modicum of relevant results.

SEO games will come and go. At the end of the day Google will always have an incentive to deliver the meaningful results users crave. The metrics they use to measure satisfaction will change, but the need for satisfactory content will not. RSS feeds, sitemaps, structured data and other essentials are only tools. At the end of the day the content is what you build. Many high traffic sites have completely bungled these basics and do well.

Simple to say, harder to execute, but entirely within the realm of the possible. Think more about the value you are providing to the user.

There's no pool of competitors -- that's why I'm doing this in the first place. The resources for learning this stuff are scarce. I just don't know how to get the word out there. I'm not looking to make money, I just want to give this away for free, because I think it's worth it. I can't even give it away D:

Then you need only change your page titles and h1 headings to better match relevant queries. The other problem could be that there isn't any search volume for that niche.

I'll try that out, thank you for the advice!

You're confusing two Internets. It's understandable, because they have the same name.

In one, search engines are advertising platforms, and list reams of content, which is also an advertising platform, designed to solicit revenue in one way or another.

In the other, search engines are for finding information, and they list sites that publish helpful/interesting/weird/fun/whatever information for free, in case someone other than the author might like it.

Confusing the two leads to disappointment.

This is a super interesting perspective, I have genuinely never thought of the internet in this way.

Something I used to do way back in the day was answer related questions on forums and have my website in my signature. It worked pretty well.

Wonder if having a "signature" on reddit (or even HN!) like this would get you banned...

Though there's probably a tragedy of the commons where high rep folks start selling signature space for advertising/influencer marketing.

I'm sure it would. They have mod bots monitoring how many of your links are to your own stuff. I run a totally free public education nuclear site (no ads, no cookies, plain old static HTML) and used to answer nuclear questions on reddit. I'd often back up what I was saying with links to detailed writing on my site, but I got banned from a few huge subs for self-promotion. Lol. So for the most part I just stopped answering questions on reddit.

I went snooping in your HN profile to find the link, and that is a really well done site. Clean design, relevant pictures, and interesting material. It's probably going to cost me an hour or two of productivity today.

Link for people lazier than me: https://whatisnuclear.com/

Having self promotion "rules" under the guise of "protecting communities" when it's really to force you to buy Reddit ads. As a user, I've found self promotion via comments way more helpful and relevant than their terrible ads...

I would be fine with paying Reddit for the ability to (tastefully?) promote in my comments

As a former Reddit mod I always found the self-promotion rules problematic. It effectively means you can promote your stuff all you want as long as you pretend you're someone else. It would be better to encourage people to stand behind their stuff. I tried not to remove self-promotion as long as it wasn't spammy (and there's a fine line there).

In the early days it was a bonus if something was OC ("Original content"). Now it's frowned upon.

But I think it's not just a cultural shift, but from being burned by everyone hustling for something. People want to drive you to their dropshipping business, their woodworking course, their OF, buy their self-help book or whatever.

Nice to see this attitude from a mod. I rarely have something to contribute to forums but love to read about people's projects. I've been in the position before of actually, finally, having done something I felt was worth sharing, a super rare occurrence for me, and then posted it and just getting instabanned for "self promotion".. it just feels like such a slap in the face from a community that you were enjoying being part of. Then getting into arguments with mods about it and eventually just having to unsubscribe. It hurts.

Pretending you're someone else won't help you if all you ever do is post links to the same site/youtube channel. In my experience the vast majority of the people who were banned for self-promotion weren't doing anything else on reddit except self-promotion. They'd create accounts then put in the absolute bare minimal amount of effort to get enough karma to create posts, or they'd buy up old accounts that already had some karma, but it was clear from their histories that their entire purpose in using reddit was exclusively promotion.

They could have easily spent a few hours a week exploring and meaningfully participating in other subreddits that interested them, but they had no desire to spend that time or be a useful part of any community. They just wanted to draw viewers to whatever they were promoting.

As a user, I've found self promotion via comments way more helpful and relevant than their terrible ads...

As both a user and an advertiser I agree. The communities I visit, if not the whole site, are faithfully anti-ad. But if I answer some questions occasionally somebody will get curious about my profile and check stuff out.

This is an amazing website. It's horrible that when asking educational questions you will absolutely never see these websites. Just the same horrible quality ones that are trying to take all your data and advertise to you.

I see lots of people with links to a home page in their user profiles (on HN, StackOverflow, GitHub, etc...) I may be in the minority here, but if I find someone particularly insightful or interesting I sometimes click through to see if they have a link.

Hmm, that does sound like a good idea!

Not only are you helping the community by answering questions, it also gives you some trackback links that Google used to weigh higher (not sure if it still does).

@dang

Sigs on HN soon pls?

Click on user name to see their profile.

Please no. Too much noise.

It was said in jest but I think everyone is taking me literally. I liked sigs on older phpBB forums when they were 2-3 lines and just some userbars. Cool back in the day, but they wouldn't really translate to the more minimalist HN.

From my time in the dying days of Usenet, I can remember there were compact codes so you could fit as much about yourself into your signature as possible. Something like the old dating ad codes, e.g. GSoH = Good Sense of Humour, but more geeky.

GeekCode, see, eg. https://geekcode.xyz/geek.html

tbf it wouldn't be a bad signal for search engines that can understand forum markup.

A boon for search is knowing intent and know who wrote something certainly helps in that regard, if a strong enough signal of course. Without knowing who intends what, you basically rely on the topic and words.

I enjoyed yours.

Does it matter that much if only a small number of people know about it?

Yes -- and this is a good question with a good answer -- because I want to help people who might be interested in the language and culture find it. And I _know they're out there_ by the number of people who at least _tried_ some really obscure languages on Duolingo. It's not for my own vanity, I want to help get the language and culture out there. The resources for it are scarce, and I feel like I can help supplement them. I'm doing the building, but I'm still waiting for the "they will come" bit.

Why do you assign such a high importance to 'help get the language and culture out there'?

The small number of people who have read it will further disseminate it themselves if they truly believe it to be valuable. As long as this is more then a few dozen people, then that should be sufficient.

To me, language and culture have intrinsic value. I also feel very attached to my cultural heritage because I'm descended from holocaust survivors. I don't want to simply sit back and watch as the culture and language disappear, and I want to provide an entrypoint for people like me who are interested but perhaps have a little less time on their hands, or who struggle with learning languages.

I'm planning soon to start releasing some videos where I read some of the old stories in English! There's not enough of it out there. It's important to me to preserve it, and the best way to preserve culture and language is to disseminate it.

and edit -- I'm sorry you got downvoted. I think your question was a very good one, and I don't think the answer is obvious at all.

I don't put too much stock in downvotes, there are so many new users joining over the past few years, some fraction inevitably of questionable quality, that votes as a signal have become much less meaningful compared to say 10 years ago.

In fact, it's probably more of a positive signal for the really interested folks.

I'm not quite sure how the language/culture intrinsically having value or not relates though. Surely it would be the relative strength that impacts the successful rate of sharing?

And there are many hundreds or thousands of such languages and cultures competing on the internet.

> And there are many hundreds or thousands of such languages and cultures competing on the internet.

Definitely! I think that on the culture side specifically, there's not very much "English-side" voice for it. Hence why I'm looking to read stories, share songs, etc. in English, so that people who don't speak the language can still find information about the culture.

Then the other prong, I guess, is helping to build out the language-learning side, so that people looking to enjoy the language have more resources to do so.

Which language? Because I've desperately been looking for a good resource to practice/learn Slovene that's not an expensive course from the University of Cleveland.

Bosnian / Croatian / Serbian. I'm adjacent to you, but I don't think they're quite the same, I'm afraid.

https://www.clozemaster.com/languages/learn-slovenian-online

Thanks, I'll check it out!

> What am I supposed to do ??

You're supposed to "growth hack" AKA post on popular subreddits, forums and sites pretending to be a casual user (or use bots) that links to the site while talking about how great it is.

That feels kind of dishonest... But then, is "justifying means by the ends" really harmful in this case if it's free stuff that I'm just desperate to give away ?:D No ads, no cloud, no data sales, just plz look at my site plz plz.

I dunno, what do you think?

Everybody with a subpar product thinks that what is lacking for them is exposure. Most of them start spending a lot on ads.

Most probably your website is not good enough to attract a public.

Edit: I know it sounds rude, but since you haven't linked to the site, there's no way to evaluate it either.

> Everybody with a subpar product thinks that what is lacking for them is exposure. Most of them start spending a lot on ads. Most probably your website is not good enough to attract a public.

You didn't have to be mean, you chose to be. Whether it's true or not, this isn't constructive criticism, it's just mean spirited.

> Edit: I know it sounds rude, but since you haven't linked to the site, there's no way to evaluate it either.

And why would I now? Now that I already know you're expecting it to be bad, I'm guessing that whether it's nice or terrible, you're going to think it's terrible. I don't have any interest in sending my website to someone who isn't interested in giving constructive criticism or giving suggestions.

I've dealt with this extensively in business, taking to business owners who have problems getting customers and think the problem is lack of exposure, while ignoring obvious flaws in their offering.

This might not be true of your site, but the pattern is so common that I assumed so. As you haven't linked the site, it is not a criticism of it. I admit the comment is mean and maybe I shouldn't have posted it.

I think that part of this is that small web sites simply are not being indexed by Google at all any more.

My ~15 year old blog has, according to Google Search Console, 15 indexed pages, and 174 ‘Discovered - currently not indexed’ pages. The number of indexed pages is going _down_ over time, despite occasional new posts.

The Search Console page says “Examine the issues […] to decide whether you need to fix these URLs.” But about the only suggestion I can find is just to wait for them to be indexed - which doesn’t seem to ever happen.

I wouldn’t argue my blog is the best or most exciting content in the world or anything, but I can’t believe anyone would say it’s worse than the often-incorrect SEO-informed duplicative nonsense that fills the first few pages of Google search results for anything technical nowadays.

It's not just you - this is widespread.

I run a SaaS to help site owners get their content indexed. We're seeing an influx of users, I think a lot of the issue is simply because of AI.

New web page additions were pretty linear over time, and then AI copywriting tools came out. Suddenly page additions basically went "hockey stick"/vertical.

Now, you can publish thousands of pages in a few minutes, and it's created a huge backlog in Googles crawl queue, thus increasing overall time to get indexation, disproportionally affecting smaller sites.

> I run a SaaS to help site owners get their content indexed. We're seeing an influx of users, I think a lot of the issue is simply because of AI.

I think that google just isn't interested in putting resources into their search engine anymore. They used to need it to gather data on people and what they were doing online, but chrome gives them people's internet histories now and android lets them collect endless amounts of data on people's lives offline. Google doesn't need search to spy on us anymore. It's only natural that they'd let it stagnate.

Sites are actively being deindexed. My entire website, probably 20 years old now, used to be indexed. Now only a few pages are.

Same here - most of my personal site has been stuck in Google's "Discovered - currently not indexed" limbo for nearly 2 years, despite regularly submitting pages for indexing, and doing all the usual things like optimising Lighthouse score and so on. I went on the Google Search support forums and it was quite sad - just vast numbers of people crying into the void. It is almost like Google have given up on search.

If they actually wanted to improve it, the key would be to move away from the advertising funded model - as Larry Page and Sergey Brin warned in 1998 "advertising funded search engines will be inherently biased towards the advertisers and away from the needs of the consumers"[0]. Personally I think they could at least pay running costs with a paid support model - it might not earn as much money as advertising, but as a gateway to the internet and other Google services it could still be very valuable to both Google and their users.

[0] "The Anatomy of a Large-Scale Hypertextual Web Search Engine", Computer Networks, vol. 30 (1998), pp. 107-117, (noting that the quote is in Appendix A which seems to be missing from some more recent online versions).

40 years ago news organizations (news papers, radio, and TV) had strict policies that the ad department (which was always in house - never outsourced to doubleclick/google) was not allowed to talk to the news department. You bought an ad because you wanted to reach people who wanted news from orginizations would "bite the hand that fed them". Internet ads don't seem to have that. They could, the culture existed before in other forms and wouldn't be hard.

I am somewhat in the same boat - my site isn't as old as yours (eight years old maybe) and I do put new content on it semi-regularly, but occasionally old pages mysteriously become unindexed, and a handful of pages seem to be stuck in the discovered but not indexed bucket too. I periodically try and update them or expand on them a bit but usually it doesn't help get them indexed. And like you, my site isn't the most exciting content but it does get regular traffic.

It's even worse on brand new website, google will index your homepage and ignore everything else unless it has some external links. Even with a good sitemap and good usability score.

This is a well known phenomenon of "Google Jail" where to combat spammers just setting up hordes of new sites, getting dinged and then migrating to a new domain, Google penalizes new sites until they've been "aged" for some number of months.

Dunno if it's about age so much as it is context. Like my search engine does something similar. The actual text on the pages are only a fraction of the signals used by search engines to put a website to put the putative search result into context.

It's really hard to rank a website if there are no links or traffic to it.

you don't serve Google ads like the SEO crap. hopefully the antitrust action breaks up that particular conflict of interest.

nothing is indexed by gogle anymore. it doesnt work at all as its supposed to for a long time. forget about it.

Kagi has an option to search what they term "the small web" [1]. I haven't used it a lot - but the times I did, it seemed to give me good results.

[1] https://blog.kagi.com/small-web

It has a very definite bias in the topics of blogs/websites that it finds which is partly an artifact of the kind of person likely to have such a site but also where they sourced it from (I think they might have started with a HN Blog roll or something). But I agree that it's generally interesting sites, and I find myself going back to it every couple of weeks for a while. It reminds me of a more substantive version of "StumbleUpon" from back in the day.

Thanks for reminding me about this! I just discovered three interesting blogs within a few minutes.

but Kagi uses third party indexes anyway (ie, google). So, if google does not index small websites, how can Kagi show them?

They augment those with their own index (named teclis).

The trouble with the bookmarking idea is link rot. It takes some effort to keep the lists up-to-date. The new webrings are not doing too well because of this either. I know of around 190 of the "new" webrings and can only get around 20% without encountering a 404 message.

The "small web" search engines might well be the way to go. Apart from Kagi, some I know of are:

https://search.marginalia.nu/ https://wiby.me/ https://searchmysite.net/

Indeed, it's hard to discover small websites. While building https://Cloudhiker.net (like Stumbleupon but modern), I collected a few hundred of those websites and people love them!

Also, clicking trough web rings (yeah they still exist), you discover a bunch of cool sites from strangers all around the world. I joined Indie Webring (https://xn--sr8hvo.ws/) and Fediring (https://fediring.net/) a while ago.

Edit: oh almost forgot. I have a few hundred links in my personal bookmark archive at https://bookmarks.kovah.de

StumbleUpon was free, your product is $2.99/mo.

What about that is more modern?

The subscription is entirely optional and not required to use Cloudhiker. It's directed towards power users. It helps the website to stay up and running. I would say this approach is more modern than shovelling tons of ads onto users like it was done by Stumbleupon.

To be fair StumbleUpon seemingly also bled money until it shut down. Unclear if a subscription fee is the answer, but it's at least an answer.

> StumbleUpon was free, your product is $2.99/mo.

What could be more modern than that?

not powered by venture capital that plans to enshittify sounds like a good sign for longevity.

> A proposal, dear reader: Create a list of bookmarks linking to websites you find interesting, and publish it for the world to see. You decide what constitutes “interesting”.

> The model is as recursive as it is simple. There is nothing preventing a list of bookmarks from linking to another list of bookmarks.

This + friend-of-friend commenting on shared bookmarks = Google Reader's "Note in Reader" bookmarklet + friends shared items[1] feature.

For the people who used shared items, the RSS feed reader part of Reader was just another way to generate shared items. The shared items list could be public and positive engagement was broadly open, but commenting on shared items was limited to designated friends and their friends.

This made the sharing aspect sticky. Sharing and commenting on items helped you expand your friends list, which exposed you to more shared items, for which you could be the aggregating conduit that shared unique items (including bookmarklet-captured items with no corresponding feed) for people on the other side of the friendship wall.

Those groups grew organically and were socially insulated from abuse. By definition everyone involved knew or had to vouch for each other, even without real names. And aside from blocking individual users, severing a mutual friend connection effectively cut them off from visibility to others.

It was fantastic.[2]

1: https://www.searchenginejournal.com/google-readers-gets-more...

2: https://www.buzzfeed.com/robf4/googles-lost-social-network

    A proposal, dear reader: Create a list of bookmarks linking to websites you find interesting, and publish it for the world to see. You decide what constitutes “interesting”.

That's exactly what I did with share-links : It's a tool that allow you to easily store and share links of things you like on the web.

Here's the repo where you can find more info (see the file DEPLOY.md if you want to launch an instance on the web): https://gitlab.com/sodimel/share-links

And here's my own instance, whith over... 4000 links: https://links.l3m.in/

Want to be surprised? Open this link on a new tab: https://links.l3m.in/en/random/

Nice, I subscribed to your feed for new links and tags.

Thanks! It means a lot to know that some people care about my projects or the curated links I choose to save and share :)

I'm not a fan of a bland list of websites, because it feeds into my FOMO and either I find the courage to click on all the links to add them to the pile of I-will-read-this, or more likely I realize I will never have the time to do it and just close the website.

A better model is to have a list of bookmarks, to specific pages if needed, with a small description of why it's worth looking at it. That's the model of shared bookmarking behind shaarli (and others before it), and it's much more amenable to serendipity: I can subscribe to your rss feed, you probably don't add an entry every single day so I will have time to look at it, and I'll get to discover other websites.

That's exactly how I used to do it- categorized links with descriptions.

Once that takes off, we can scrape bookmarks list and descriptions. We assign webpages with lots of incoming links as more interesting (bonus if it's updated often). A "page ranking" so to speak. Then everyone can discover pages based on simple searches.

Has anyone else thought of this?

Your sarcastic description of the web shows exactly why TFA exists: by using such tasteless metrics everything can be easily gamed to always the same site, always the same format. And I don't know what I want to discover, so I can't search for it. You're actually making the point against your own view.

I'm just pointing out we had all this before in the 90's, we loved it, and it didn't work. What's different this time? We REALLY want it this time?

The article doesn't point out anything that hasn't been tried. You're telling me that bookmark links is going to solve it?!

Maybe we can pay money for discovery instead of paying with our eyeballs, but do people even want the small web enough for it to work? I do, but I highly doubt enough other people do. And even if a high enough threshhold of people want it to work, what's going to stop the SEO/gaming arms race that happened before?

To me, it looks like the same problems of social media and eventual enshittification. I think despite the best intentions we had, this is the reality of the web (or maybe the reality of humans, or maybe the reality of capitalism).

We ? We the users have 0 say into what the web looks like, where it is heading, how it works and how it is financed. Indeed capitalism, through for-profit companies, decides why list of bookmarks are not sufficient anymore, why we need to fill pages upon pages of artificially generated crap, why strong superficial emotional reactions are more valued than thought-out reflexions, why instant comment is better than slow analysis. It's all about monetizing content, monetizing reactions, monetizing everything. We don't create to pay for the infrastructure, we are milked for personal profit. Not our profit. We haven't discussed at all how we want to share content.

Doing list of bookmarks is our way to organize ourselves. Always has been, always will be. Except now we know why we have to do it, compared to times before where monetization was an attractive light that we didn't know was a trap.

SEO won't work because we don't care about SEO. We care about personal touch, about lightweightness, about sustainability; none of those are provided by ad-based business because none of those are provided by capitalism, by construction. It's not about being the biggest, it's about being ours.

I've have this weird idea in my head:

There's a book I love, Daemon. I love it so much that I'm willing to bet you money that you'll love it. How about I buy you a copy of the book, and if you love it, you have to buy a copy for two other people?

Or... how about, I bet you $1 that you'll enjoy this website. If I'm wrong, you keep the money - or maybe it goes to a charity of your choice? If I'm right, I get my money back... and maybe you have to give me $1...? and you have to pick two other people to gamble?

I dunno - the thought has been tickling my brain for a while...

Someone would just create a bunch of fake accounts to exploit the userbase and redeem all of the free stuff

My idea is that this is just between you and people you actually know. First-hand.

In that case, why not just buy your friends or colleagues copies of the book as a gift?

Because I want it to go viral? Especially when it's a new author, and I really want to support them. It's not just about finding good books for my friends, it's about finding customers for authors / creators I really enjoy.

I have this thing with one of my friends where, if I really enjoy a piece of media and I want to recommend it to the other person, I would watch it again with them because watching it for a second time shows there isn’t anything more interesting right now than the media I’m recommending.

I love this post. Share more "crazy" ideas!

> Blogs limp along through RSS and Atom, but relying on feeds shapes everything you write into a blog entry. It’s stifling, homogenizing. The blogosphere, what remains of it, is incredibly samey.

I don't see this at all: my feeds contain so-called "long form" post and sometimes single-line comments.

AFAICT Gemini is essentially "add non-HTML files to your feed experience". Umm...OK? Certainly this blog post didn't suggest more, and to me wasn't particularly convincing given that I don't even experience (IMHO) the "problem" the author decried.

Discovery is and will ever be a problem, but comments on sites like HN expose interesting things all the time.

What I mean is your website turns into a list. Posts have a definite date, a title, a body, semantics such as a previous and next; before and after.

I agree, why does it matter if my feed is interspersed with everything from long form to a simple short form link post?

I like this author's idea of curating bookmark lists, but I think they are most effective when two criteria are followed: (1) keep the list small, (2) write small notes about each entry.

For example, the bookmarks list the author links to (https://www.marginalia.nu/links/bookmarks/) has 48 URLs annotated only by category. That's too many for my tiny brain to handle and I move on.

A webring like Hundred Rabbits' (https://webring.xxiivv.com/) has 203 entries. For me, this is in the same category as 48. (It also reminds me of those "Awesome X" lists on GitHub that end up flooded with hundreds of links.)

To attempt an example of what I mean, here's the bookmark list I publish on my website:

- Bret Victor (http://worrydream.com/) • interaction and abstraction

- Craig Mod (https://craigmod.com/) • long walks, atmospheric photos

- Hundred Rabbits (https://100r.co/site/home.html) • physical and digital minimalism from first principles, extensively documented

- Mu-An Chiou (http://muan.co/) • rhythm, space, movement, color

- Steven Wittens (http://acko.net/) • 3D sorcery

(edit: formatting)

Something something dead internet theory.

But also yes, I can't believe how many great, small blogs and other useful websites I've found only from HN

This reminds me of the early internet. Every little site had a "links" page or "friends" or "sister sites" or "blog roll"

There were also "web rings" which was a club you could join. You put a widget on your site that would randomly link to other pages that were part of the same web ring (and you'd get back links from other sites).

I haven't really seen any of those things in probably 10 or 15 years though.

> It’s a bit strange, almost nobody seems to be doing this. Looking through a sample of personal websites, very few of them has links to other personal websites.

Bingo. This isn't just the blogosphere, I see it in research papers, on GitHub pages, on social media, and elsewhere.

I could speculate, but I don't really know what causes this...

I'm sure it has something to do with crab mentality. Why promote others? The only people I see doing so are the already super successful.

https://en.wikipedia.org/wiki/Crab_mentality

For me personally, I have been trimming links I have in various public places because a link is a sort of endorsement, and these days there is always someone who gets mad at you for endorsing the wrong thing. Even if a personal website is mostly alright, someone is going to dig up a random quote out of context, and the rest of the internet will judge your links based on that one bit.

Linking to a specific article on a single topic seems relatively safe, but linking to toplevel websites or blogs seems more risky.

Why in the world would you care about that, when there are already millions of people who don't even know you who hate you because of things outside of your control?

I previously asked about about bringing back webrings to discover small sites but didn't gain much traction. there was only one comment linking about webring . I am curious too why there isn't a strong open source initiative like so many other open source projects. https://news.ycombinator.com/item?id=38177128

Is it because there are just fewer personal websites to link to? I run a couple of small websites - one has eight or so links to similar personal-ish websites (non-reciprocal - just links I find useful and I assume my readers will find useful) but the other has no such links, just because I haven't found anything similar worth linking to.

It's simple. Personal websites are the equivalent of "check my Linktree" for the HTML-literate. Most personal site sites I've seen are either boring technical blogposts or an impersonal online resume. People aren't showing their real personalities on their sites anymore.

It's because Google treats all sites with a large number of outbound links as a link farm and punishes them. Because Google couldn't figure out what list of links was organic vs spam it just punished everyone.

Who remembers delicious?

That was a good social bookmarking site. I wonder if anything similar exists now?

There is still something of it remaining: https://del.icio.us/help

> This site is a ghost, haunting the internet. It is a read-only archive of the bookmarking website del.icio.us.... This project is a labor of love (or more accurately, a labor of like). Del.icio.us was founded by my friends in 2003, sold a whole bunch of times, and when it was about to get sold again to spammers in 2017, I took the opportunity to buy it back.

do you have a link to the archive ?

I wish I did. It looks like we will have to be patient.

Can you elaborate more ? Is there a archive that exist but link is missing or the new owner will bring the site back.

Self-hosted version: https://github.com/Kovah/LinkAce/

That was what came to mind for me when I read the linked article.

For those unfamiliar: https://en.wikipedia.org/wiki/Delicious_(website)

Publicly shared bookmarks was a great way to discover new sites, and I feel like delicious died because it was acquired, not because it was a bad or unpopular idea.

Looping this back to some of the linked essay, I've always wondered if there's a way to make a social bookmarking system that's more decentralized or federated, through a browser plugin or something? Maybe something that's hosted on multiple hosting websites?

For $22/year there's https://pinboard.in/

Yup, and comes with an API (originally cloned from del.icio.us IIRC), so you can back up your data or use in other ways. I'm always archiving updates on mine, since he does joke from time to time about getting hit by a bus.

I agree, it was my favorite site on the web before it was acquired.

I've thought this for years that a large part of the internet is essentially ghosted by a lot of the bigger search engines. I miss the old Yahoo/AltaVista type homepage where you would get a feed. Digg, StumbleUpon, del.icio.us were essential tools.

Geocities and MySpace had webrings, so once you landed on something you could generally find similar stuff.

I'd love to see a good "home page" with curated feeds, bookmarking and search across it (and with LLM + Graph you can have your own semantic search)

Whatever happened to stumbleupon? I feel like that was a popular website that served this purpose.

I built https://search.marginalia.nu/explore/random and https://explore.marginalia.nu to try to capture the old stumbleupon vibe.

It's manually curated though so not the most scalable thing I've put together. Wish I had more time to expand on this, seems on the cusp of being pretty cool.

How do I submit a link for this?

I haven't built a clean workflow for that yet, but in the interim, make a pull request here:

https://github.com/MarginaliaSearch/PublicData/blob/master/s...

and I'll poke it into the DB.

If you don't want to dirty your hands with github, you can send me an email at kontakt@marginalia.nu :-)

I miss stumbleupon.

It was a tool for an exciting experimental internet.

Nowadays I can only imagine it’d be flooded with odd numbered list articles.

stumbleupon was reliant on putting websites into a frame with their menu at the top, once websites started blocking the framing of their content, for various security reasons, the idea was dead.

nice thanks for the history!

This is kind of blowing my mind, but I'm looking for people discussing the proposal here and can't quite find the valence that seems most potentially viral or self-perpetuating:

There seems to be a really exciting incentive to share lots of links, to regularly hit a hotkey to add the current page to a link list, because there is a kind of graph traversal thing that can emerge, akin to recommendation engines spotify or youtube use, whereby your (anonymized) "like history" -- might be interesting to include some metadata, including when it was liked, how you arrived there, whatever -- will connect you with other people who have intersecting likes, and then blow open entire other leaves of search trees you didn't know you wanted.

I have this feeling on the net recently where I feel like starved, it just feels so stale and bland, hard to find actually good content, going back to HN or Twitter or Reddit or whatever, these little linear "feeds" with discussions etc.

I want a feeling of opening-up, branching, discovery, excitement.

I feel like if a bunch of people shared their like histories in a pseudonymous fashion, you could see these fascinating interest clusters emerging and if there was a compelling UI for navigating them, it could really be self-perpetuating and awesome.

Sort of like visualizations of LLM embeddings, showing regional clusters of information domains, but with a navigable, social aspect, and where because it's pseudo-anonymous you don't mind running AI recommendation engines on it for you and others.

Does this exist? Should it? I would love something like this!

> But I’m just going to state that automatic link feeds do not seem to work on HTTP any more. You end up with a flood of astroturfing, vapid click-bait and blogspam (i.e. reddit).

This would be true for anything that has a critical mass of users which would make it a target for astroturfers and spammers.

The likely reason Gemini has managed to escape that is because it's a text-only alternative protocol that only a very few people would be interested in joining.

The difficulty with this is linkrot. It is not just that links break, but they change into content farms or domain squatters and unless you are constantly checking them, your neat collection of links quickly becomes useless.

I'd be willing to do something like this. I'm too lazy to implement it (hence why we don't already have something like this) but I would enjoy something like the following workflow:

1. my firefox browser has an extension

2. if I think a website is interesting, I bookmark it to one of N bookmark lists (which can be arbitrarily categorized, whether topical like "tech rants" or Google+ style "IRL friends only")

3. the browser extension does some API calls to flush/fill each bookmark list to one or more of publicly accessible websites like my github bio, my HN profile, my blog listicles for one or more federated bookmarks, publishes an RSS entry, whatever

This approach does not require an account (except that I give the browser extension credentials/tokens to wherever it publishes), and it results in one-click blogroll sharing.

PS: the problem with this is the temptation for feature bloat.

Feature 2: on known websites with user profiles like HN, reddit, github, check the user profile for all users on the page and list out discovered shared blogrolls by username

Feature 3: reports such as 'most shared blogroll links' based on your own personal browsing history, calculated offline in your browser

Feature 4: ability to block blogroll links with a comment as to why you do so

Feature 5: ability to share your blocks with a given blogroll list

Feature 6: ability to follow shared blogroll link blocks from other blogroll lists, then editorialize that shared list yourself

Feature 7: ability to share your editorialized block list with others who trust you more than whoever you are editorializing

...and so on.

Though I'm pretty sure I'm reinventing lots of lost features from the web of trust and semantic web era.

A linked bookmark blog is essentially how many use Pinterest. Discoverability in part comes from shared lists because the related items on a given page are¹ based on what other things are included in lists containing the item currently being viewed.

Pinterest has a very strong visual bias, and often a selling-things bias with the stored links being to things you can buy, so there might be a niche for something like that with features geared around links more generally, or features specifically to help editorialising links to nows and other reading matter, though preferably without deliberately poisoning search results to over-favour the link storing site like Pinterest does.

----

[1] at least in part, there may be other factors like what-you-have-looked-at-before, advertising, etc.

I've noticed this too. My art/advertising project (https://24HourHomepage.com) let's people advertise themselves for free. Lots of people promote their IG/Twitter/Tiktok etc.

I'm not surprised given how much "influencer economy" marketing there is (also, I advertise the site on Tiktok/IG/YT) but I'm over-joyed when I see good ol' fashioned websites.

Even then, they seem to be older sites :/

Other good sources for small website discovery: - https://cloudhiker.net/ - https://theuselessweb.com/ - https://www.boredbutton.com/

With shifting from Google to AI all websites will have a discoverability crisis. If I ask an AI "Where to buy XYZ" - how as a website do I get in there?

So this article highlights a critical issue that urgently needs to be addressed. The advancement of artificial intelligence (AI) is partly contingent on resolving this problem. With the decline in search engine usage (a trend I contribute to, as I keep ChatGPT open all day), discovering websites becomes increasingly challenging. Consequently, this may lead to a decrease in content creation since websites are receiving less traffic. Ultimately, this could hinder AI development, as it relies on training with new and relevant data. Additionally, it's worth noting that this issue may extend to books as well.

Oh boy, I'm the owner of a few small websites that have been blessed to be discovered.

I should do this to pay back the favor. I already have a list of bookmarks!

Problem: it has 7328 bookmarks because I save everything good I come across, not just small websites.

Give me a few hours. :)

You raise a good point that large collections have enough value to motivate people.

The idea of trading "discoverables" is solid. Like in the zine or warez culture it's great to trade a big cache with others. There is a trust issue, that some will poison the well with their malware or spam, but many new solutions do seem to take-off out of mutually motivated sharing. Perhaps something as simple as a library/standard for "exchanging trusted link collections" - and there almost as in-built webserver function would be a game changer.

Like if https://links.example.com/json always returned a most recently verified bundle.

I want to point out Ruben Schade bookmarks page. It's quite wonderful. An OPML file that is also a webpage.

https://rubenerd.com/blogroll.opml

View source on this one!

Wow... I never really knew when I was visiting a website transformed by XSLT

This is really cool

Here is how you find such websites: https://wiby.me

But how are people expected to find that website?

A comment on Hacker News, of course

True, we need a small website search engine search engine.

In all seriousness though I have forgotten the name of this site at least 5 times, and have to look it up every time (and sometimes it is hard to find)... perhaps a rebranding is in order.

I'm a big advocate of The Small Web[0], and care deeply about the internet being a creative space. Something to keep in mind is that the way people use the internet has changed completely since the days that small, independent websites were the norm. When I first started using the internet in the late-90s, if you wanted to find information you would probably end up reading a page curated by a single person, devoted to a single topic. I remember having favourite sites devoted to bands I loved. Now, the main resources for information are collaborative platforms like Wikipedia. These days, if I want to find lyrics to a song I'll probably find them on genius.com, rather than someone's lovingly crafted fanpage. These days I can just right-click 'Search for <term>', and I'll have an answer within seconds. Even though this has taken a way a lot of the creative, human aspects of the internet, I think that the ease at which people can retrieve information on-the-fly from known sources has made the internet a much more powerful tool. I think what's missing is our desire to actually look for, and find creative spaces.

Having said this, I'm not giving up on the small-web. I run https://webri.ng/ which is a platform people can use for hosting webrings.

0: A term that I borrowed from Parimal Satyal: https://neustadt.fr/essays/the-small-web/

I don't know. I feel like creative output on the web is orders of magnitude greater now than it ever was on the 90's web. Unless you limit "creativity" to the design of custom HTML and CSS.

Yes, that output exists primarily on services and yes that presents issues with control over data, data mining, AI training, etc. But the premise that the internet is no longer a creative space simply because people use it as a tool to publish to, rather than an expression of programming and design in and of itself, to me seems to miss the forest for the trees.

You make a great point. You're totally right that the total amount of creative content on the internet is much, much bigger now. That is something worth appreciating. I've found some really amazing artists on social media platforms, and even been able to reach out and collaborate with them. So it's not all bad, of course. What I'm mostly concerned about though is the website as a personal, creative, individual space. This doesn't really need to be a user writing their own visually creative HTML/CSS. I just feel something of creative value is lost if the entire human aspect of the internet is viewed through the window of social media.

Sure there's nothing wrong with honing those skills! I have friends who crochet, an art as old as time, but I just buy clothes. I'm terrible at crocheting (and jealous at my better friends!) I'm quite good at starting fires with flint and the fire triangle but almost everyone I know just fills a lighter with lighter fluid and presses a button. Do I think there's an art to starting a fire? Of course. Do I think humanity has lost something by abstracting starting a fire behind the push of a button? Not really. I'm just glad that I can choose to start a fire now rather than be forced to start one.

“curated link directories were a thing back when the Internet was in its infancy, but the task of maintaining such a directory is a full time job”

Here’s the economically viable answer. It could be a job in the same sense as a YouTuber. I imagine lists being distributed through releases, not an infinite, always-on feed. Releases would not consist of links only, but actual context. A well known curator could use his reach to leverage smaller projects, the political factor of course would be present. It’s not so different than what people already do on social networks, and that’s a good sign. The main difference would be that the linked content would be sourced from around the web, not only from within a particular network. Ad business could also flourish in this context. Ads can be really useful. For people with interest in a given topic, it is valuable to get access to offers regarding the activity. You could have people literally making a living just by thoughtfully wandering on the web and collecting media with a niche public in mind. The mere act of selection from the curator would be an expression of individuality, even more so if they enrich the published volume with their view. It would reach a point where you would simply submit your content to the curator network, bypassing search engines entirely. I would also add that a fully curated internet would feel extremely safe, since you’d always know what to expect from the sources you subscribe to, and also considering media would flow downstream in a parallel fashion, so no intersection of undesirable content. It would be beautiful if it all happened through a torrent or torrent-like protocol.

This reminds me of Dense Discovery (https://www.densediscovery.com/) and Tom Scott's newsletter (https://pad26.aweb.page/subscribe).

Somewhat obvious disclaimer that this is my own stab at this a little while back: https://johnvidler.co.uk/blog/federated-web-rings-and-link-s... with its associated search tool: https://johnvidler.co.uk/webgraph/

I tried to come up with a spec for listing bookmarks in an easily handled format for both humans and machines, and just ended up with using .json files; here's mine for example: https://johnvidler.co.uk/webgraph.json which all follow a very loose specification. Because it just requires a single file to connect to others using the same system its really easy to implement.

I've been slowly pushing for folks I know to add to the sources that the search engine can idly spider, slowly building up a large searchable list of user-selected links.

Every thought we express, happens in the context of something we've read or discussed etc. before. For example, this comment is based on the original text linked, the comments here on HN, and my own thoughts.

I think a writer should include all the references that lead to their thoughts expressed in the texts. This way the reader could follow these links to explore more adjacent topics. To me, it seems like the ecosystem of small website could improve on that. I'd rather explore other writers based on some topic than on a blogroll.

In scientific writing this has a long tradition called related work. In some fields it degraded a bit into a section that merely lists all the works that should be named rather than a deep discussion of their contents.

Obviously, this is also the original idea of Berners-Lee for the web.

A good example that comes to my mind is this: https://www.astralcodexten.com/p/somewhat-contra-marcus-on-a...

Sounds a lot like what used to be one of my favorite site

del.icio.us

Which I believe Yahoo bought off and killed.

I use pinboard.in. Works fine for me.

But saving tags--what was called folksonomies at one point--never really became a mainstream way of sharing links as opposed to just bookmarking them for your own use.

At the end of the day, a lot of this is lamenting that a hand-curated Internet doesn't scale.

I miss delicious. The amount of discoverable content that was staggering. Collaborative bookmarking needs to make a comeback.

pinboard.in is quite Ok, if you don't mind paying yearly for it

Is the claim (that there are interesting small websites that are impossible to find on the major search engines) true? I mean, it feels true, but how would we measure this? How would be detect whether things had got worse, or was getting better?

As I say, I am sympathetic, but I would like to have more confidence in the claim, and better ways to test proposed solutions.

Someone ought to build a search engine to try to demonstrate this fact.

Probably too late for you to see this, marginalia_nu, but I just wanted to say I appreciate the work you've put into your search engine and I should have noted that you would have more insight into this than most.

Haha, no worry.

Duplicate Google, but put a minus sign in front of the ranking formula.

I feel like less and less small personal website owners maintain a list of links to other personal websites they like. That's too bad because I think this is a fantastic way to discover those parts of the web :).

Here is my link list: https://pablo.rauzy.name/links.html

I have like a billion bookmarks and sometimes I tag them well, but sometimes I don't, and once every few months I'll go through the list at random and tag away.

But like, what's the point of all those bookmarks? Outside of implementation details that are captured on stack overflow in convenient question - answer format, I feel like everything that exists on the internet already exists better in books. Even my own blog is basically just me stumbling though photography and philosophy and travel destinations, writing out information that's much more easily understood from published materials.

Well, I suppose one thing small sites are good at is condensing information that should be condensed elsewhere but isn't. My most popular article (gets tons of organic traffic from Google) is titled "How to Rent a Motorcycle in Taiwan." If you searched that on Google I think the top few links should end in .gov.tw but none of them do.

Websites are last century tech - everything else is a natural by-product of that.

If small websites were a good idea, myspace would have won - it was the ultimate 'create your own little website' 'thing'.

The real crisis is app discoverability. Unless you're put on the front page by Google or Apple - good luck getting your app to gain traction without giving half your company away to get venture funding to afford to spam people's lives with advertising.

That's the real crisis - we don't have discoverability services and rely on spam because the companies that control the internet profit from selling ability to spam people. Websites - I don't even know the last time I visited a new website multiple times (aka none of them have provided any lasting value) - it must've been years ago and I'm on the internet all the time.

> Create a list of bookmarks linking to websites you find interesting, and publish it for the world to see.

I do like this idea. But it begs a few questions:

- Where?

- And then what?

The problem statement is that its hard to discover small websites. If I publish my bookmarks on my small website then the problem has not been solved. It seems like there needs to be some aggregator which of course ventures directly into the problem of centralized distribution and algorithms making the little guy undiscoverable. Whats the solution? A site with “friends” whose lists you can see with no overall aggregation?

Personally, I actually think it would be interesting to aggregate these lists. In particular I think it would be interesting to find the most common domains below a certain threshold of popularity. Pretty sure you can do this on google already, although youd have to save the string and copy paste it in all the time.

Well I'm sold, Im re-adding my links section to my personal site. But this time more links less curating.

The small website problem is not too different from the small social media account problem. Most people on social media get zero traction. Barely any views, likes, engagement. Almost all traffic goes to a very small group, and not just that, the big account has a snowball effect. They are ranked higher, thus seen more, thus followed more, and so on.

The algorithm is basically incorrect in that it keeps rewarding something that was already rewarded. You can't say that this channel deserved it most for the simple reason that the rest wasn't even seen at all.

In fact, even if you'd list 3 accounts below each other that we hypothetically consider to be of identical value, the one on the top will get 80% of the followers/engagement.

And that's why blogrolls and link directories also do not help.

It’s a fine proposition but in practice, with a list of bare urls, the author asking me to click on random links with no context[1], and I have to override my instincts, shaped by decades of internet use, to be very wary of doing so.

I don’t think this works without a summary provided with each link. Or even just fetching the open graph tags or something.

[1] it’s not a 100% fair test because I don’t have a relationship with the author, so there is no trust to leverage. But I don’t even like clicking on bare links in WhatsApp messages from close friends without a word or two on why I should do so! There’s always a possibility someone’s account is compromised, and without being able to evaluate that the link is accompanied by their “voice”, I err on the side of caution and usually ignore…

Is this reinventing Google's original "links are votes of confidence" observation? Nothing wrong with that of course.

> on gemini-scale it works pretty well

This might be the problem in a nutshell. Maybe discoverability doesn't scale, and (overlapping) villages are the only solution.

My own repositories:

- bookmarked entries https://github.com/rumca-js/RSS-Link-Database

- mostly domains https://github.com/rumca-js/Internet-Places-Database

- all 'news' from 2023 https://github.com/rumca-js/RSS-Link-Database-2023

I am using my own Django program to capture and manage links https://github.com/rumca-js/Django-link-archive.

Google buying up Feedburner only to shut it down didn't help the small websites federated via RSS/Atom either. Seems like it was replaced with centralized medium now. BTW Is there a good feedburner alternative out there?

It's hard for small, beautiful websites to get traction. One big issue is engaging with a small audience, like having a commenting system, without imposing an account signup on your readers or having them login with Facebook.

What do other HN'ers think about the UX of using email for accepting comments? Sort of like a form submission. Details linking the comment and the parent post could be stored in mailto links. I posted an Ask HN before seeing this post that has more details https://news.ycombinator.com/item?id=38278697

Why do you need comment fields?

I always figure, if someone wants to talk to me, they can send me an email, and popular blog posts, like the one we're discussing now, generally end up having comment fields on sites such as Hacker News.

Well that's exactly what I mean. Comments fields aren't a requirement, but I disagree that HN and other sites are an acceptable replacement.

A much more suitable replacement is like you said, sending an email.

"Web sites"? Isn't a large part of the useful small web about single projects or pages?

And then discoverability is what HN does - although scalability is not there.

That is, it's not all that important that someone has a personal site with travel photos. If in there they have 3 awesome reviews and summaries of economics books then I'd like to find THOSE. There is no or there should be no obligation of consistency, body of work, or overall project in discovering worthy single essays.

Something like "long form cooperative bookmarking"?

A billion small websites cannot be discoverable all at once.

The search engine is an extension of the pop culture treadmill. It determines what is popular.

Some sites inherently deserve to be popular, like "horses' mouths" primary sources of important information.

For others, it's just a popularity crapshoot.

It's the same like in music or anything else. A million equally talented singers cannot all simultaneously get to be popular idols; you cannot introduce that many people to a nation. At some point it's a lottery.

That doesn't amount to a "crisis".

There are definitely more web surfers than there are web sites, so reasonably speaking, most of them, at least the ones built by human beings, should be able to get at least a modicum of traffic.

The reason we have such concentration of popularity today is that we use popularity to direct traffic. That starts feeding into itself fairly rapidly, and the result is inevitably that the same few websites eat virtually all of the traffic.

Remember when content creators of yore were teaching the web newbies how to generate traffic to their sites and get noticed on search engine rankings?

We still have that now. Only it's how to get the YouTube algorithm to share your content or how to optimally use Steam's tools to promote your game. It's all within the walled gardens.

Even Google, a search engine of yore, is a walled garden now much more like a social network than it's prior iterations when it was an indispensable tool for finding new content on the web.

Yes! I've been thinking about this for months now. How would a normal person on the internet find information and enjoy the internet if most stuff they can find revolve around closed platforms, or SEO marketing sites.

The best way to use the internet is to distribute information for everyone to see. Yet, that's not the direction it seems to be heading towards.

I really believe that the fun part of the interner is in small sites, and I think we need more projects that try to find these small, but important corners of the internet.

Instead of putting links to interesting websites on a bookmarks page you could link to related content within the main body of the article itself. You or I don’t know if I’d want to read more about a website about ‘humanities’ or ‘misc’, but since I’m reading through this article you know I’m willing to read about small websites. So, if you have read anything relevant about small website discovery, Gemini, or web rings, you could link to it from where you mention it in the article.

I would love a search engine that only catalogs pages without ads.

I added a bookmark list (or call it a webring or a blogroll) to my website software Hey Homepage (link in bio) and I call it 'shared links'. Indeed, nothing more than a list of interesting links. I also present this list as an OPML file and I dubbed this whole concept 'Other People's Meaningful Links' (not my own abbr).

This kind of thing is so crucial for discoverability, I want to do more with it and I hope this topic gets discussed more on HN.

I've recently thought that a return to site directories might be overdue.

Possibly something based on the dewey system.

If I'm interested in making shoes, I want to see a list of relevant sites (or videos, etc) that have been vouched for (possibly with comments) by actual humans. Obviously rating/review abuse is a hard problem, but imagine if most subreddits had a list of quality sites as determined by users of the subreddit. Then imagine it without the reddit bit.

Think of small websites like those cool, hidden gardens you sometimes find in a neighborhood. They're not famous, but when you find one, it's special. They're made with care, not for fame. It's like that with our little websites. We build them for fun, for ourselves, and for the few who might wander in and smile at what they see. That's what makes them awesome, not how many people visit

I like this approach, more than webrings: it forms a graph, rather than a linked list, which is more reliable, and basically what the "web" metaphor is about. So I both have a "links" page on my homepage, and trying to link relevant resources from public notes.

Embedding RDFa metadata seems potentially useful for forming a graph and helping with exploration as well, but it does not seem to be used much.

We've needed a search engine that excludes the work of corporate giants that dominate our modern internet for a while. It's just never been so clear and dire as today.

Interestingly google used to have a per-user, user-controlled domain blacklist that go excised around ~2008 or so—presumably because this would have enabled automated blocking of high-value clients.

> Simple federated bookmarking

> A proposal, dear reader: Create a list of bookmarks linking to websites you find interesting, and publish it for the world to see. You decide what constitutes “interesting”.

For federated content, we could use ActivityPub if the site content units are posts. At some point, there should be a searchable taxonomy of the federated groupings and sources.

Check out http://metaphor.systems/ - neural embeddings based search that is really good at finding the longtail of high quality content.

Really good for finding personal pages, niche blog posts, etc.. Algorithm doesn't at all weigh website popularity explicitly.

(Disclaimer: I'm one of the cofounders)

On my personal website [0] I've kept a small list of "mildly interesting sites" which just gets duplicated onto each page (all static HTML), but it feels like I've only ever removed dead links from it and rarely added anything.

[0] https://brynet.ca/

There is a big software discoverability crisis, not just websites. The same software gets reinvented a thousand times a day by the industry because that battle-tested FOSS software is lost in the depths of the internet.

Idk what will solve it. My bet is on a specialized web-scraped database specialized for OS software with an LLM frontend.

Does any one know if it is possible to teach a llm to differentiate between a small web website and a commercial one ?

I was actually thinking about this the other day - the approach is much simpler and more scalable than web rings. and I can remember it being everywhere in the 2000s. Found a lot of webcomics that way.

They weren't called "bookmarks" though. Wracking my brain to think what it was usually called.

“Blogroll” was the name for your list of other blogs you like, and was put into an OPML file.

Real question: Could machine learning be used to identify these sites to a high degree of accuracy, with crowd sourced human moderation denying the false positives to help further train it?

Disclaimer: I know very little about machine learning.

Is it true that "traffic is evaporating" for smaller sites? I'd love to see some numbers.

Keep in mind that overall traffic on the internet was orders of magnitude lower back in the heyday of quirky personal sites.

Maybe - I run a few small sites, but the two main ones have seen increased traffic over time (I make some effort to be discoverable and add new content). I think that Google's recent index pruning could be a factor though - a small percentage of pages on one of my sites have become unindexed for no obvious reason - I assume others are seeing something similar, and it's possibly accelerated if content isn't being updated regularly.

Right now, I use Hey Homepage myself to follow around 850 other websites and their updates. I follow some computer and car news. I follow some 'dev blogs' with weekly or monthly updates. I follow some timelines from people that post shorter but more frequent posts (like Twitter). I even follow some Youtube channels without being exposed to their algorithm.

What I'm missing and would like to see more of, are feeds about hobbies/activities other than computer-related stuff. I might be in a bubble or I'm dealing with early adopters, but the only quality feeds I encounter are from programmers who write about... programming. I put my money where my mouth is and added a microblog/timeline to my website about building a bicycle caravan (see theredpanther.org).

What I also miss are more 'photo feeds'. Every update in a feed can have a picture included, why not make more use of that!? It sparks some live into the dull text-only format. Adding a photo now and then also makes the webview of the a timeline more interesting. Just as Twitter-posts can have a picture attached. I make extensive use of photos on a niche site of mine about beautiful cars. Go check Artomotive (https://www.heyhomepage.com/site/artomotive).

The technical side of these things isn't new or innovative. And that's the beauty, it's proven technology. No hype, just natural growth. The technology behind 'feeds' (it's called RSS, I call it 'Really Social Sites') is twenty years old now. It's not tainted by surveillance capitalism, commercialism, algorithms, platforms or AI. But to get the most out of it, you have to do some things. Like collecting interesting feeds/websites to follow, clicking to read the whole article, playing the algorithm yourself by curating the stream of content, etc. But if you want to stay sane on a changing internet, it seems the only way.

Google is also a part of the problem here because everything is automated if Google wanted to help nurture the creative web they could at least feature new sites on web.dev.

All Facebook and Twitter profiles are the personal websites and blogs of this era. The benefit of having a dedicated "website" is between none to negative.

Maybe a big Google query prefix to hide all major sites?

This is a great post. I have about 10000 links saved in Pinboard over the past 10 years and to figure out how I could share.

All links should be tagged. Tags may help to create multidimensional category structure

What is to stop a list of bookmarks not also just being a load of links to spam and astroturfing? You could argue "ah well you have to just use the ones from websites you trust!" But that is basically PageRank when you think about it. So we're back to the same problems.

I guess we can't have nice things.

There used to be directories that indexed those, but the flow of spam became insufferable and the money dwindled. Social media with its free content and auctioned advertising took over. The problem with current algorithms is the fact that they favor currency - if you are a Youtube creator and you don’t churn content at a consistent rate, you get less exposure than the new stuff - creating a punishment for old but gold content.

Netizens should own the means of consumption.

To show how old I feel, my friends and I had a vision for where the web was going in the 90s. We wanted a shareware future where anyone and their grandma could save their MS Word file to a shared drive on their computer and point a domain name at it for a few dollars per year. Then use a real version of PayPal with no transaction fees to accept micropayments. To maybe write casual games priced at $10 and sell a few hundred copies per month to enter pretirement. Or make pottery and sell it themselves rather than on Etsy, and actually get found and paid.

Of course that future never materialized, or more accurately was quickly overshadowed by the attention economy which makes nothing and sells nothing.

Through that lens, it's easy to see where search engines went wrong. There should have been a public domain database of the web and its metadata like archive.org, but fully indexed and queryable through SQL. Search engines could be built above that with clever queries, but the database would always be available to all of us. Then we could have search that does the opposite of what corporations do, and actually make long tail results the most prominent. Instead we got the ensh*ttification of the very best stuff like Google, bringing the worst aspects of capitalism to what should have been a free human economy of ideas on its way to delivering a moneyless society.

I would very much like to work on all of this and help bring us back to the bright future of 1999. I'd also like to write declarative stateless programming languages and AI that actually does people's work for them to get them paid instead of robbing them of their creative opportunities. Basically put real effort and resources into undoing whatever all this is and get back to the real work of solving the prerequisites needed to roll out UBI. Instead like most of you, I'll likely spend the rest of my life throwing all of my energy at the hardest problems with the smallest rewards to make rent. A tragedy of the commons on such a vast scale that we can't even see it.

I still have faith that AI will deliver the semantic web and maybe this could all materialize, since human-curated metadata was always a pipe dream. But I worry that the trillion dollar tech status quo will stop this. Mainly because revisionist history has turned what were once deep insights and winning strategies into easily discreditable flights of fancy and idealism. That's why search engines were corrupted in the first place.

In this article: people rediscovering the concept of a blogroll.

Isn't this the idea of web rings from the 90s?

Yellowpages v2? What’s old is new again.

How do people find links on the darkweb?

Create a list of bookmarks linking to websites you find interesting, and publish it for the world to see...The model is as recursive as it is simple. There is nothing preventing a list of bookmarks from linking to another list of bookmarks...The creation of a bookmark list is a surprisingly fun project.

I agree. I've often thought of people publishing a list of bookmarks in a way that everyone can see. I even created DownloadNet originally based on this idea. I wanted a way to publish one of my bookmark folders as a server for people.

But then, as so often happens, the simple idea evolved, and I got carried away by who knows what (technical challenges? I don't know) and ended up creating a personal archive and search engine with only a scant integration with bookmarks.

This article is a good reminder of what originally seemed to me a good idea. Perhaps I should add it there. Also, perhaps p2p could be an easy way to federate these things? Not everyone can just create their own server, nor do they want to host it on big providers always.

I've been tossing around the idea of p2p as a way to "solve" this, but it's still rather formless: new and vague. Over the last 3 days I created a p2p blog (and again, got carried away -- perhaps with technical challenges -- and added p2p chat). But I think there's something there.

Perhaps I should listen to that idea that keeps recurring for me. To that first version of it anyway.

Something simple, that unifies, publishing a bookmark folder (I have some chrome bookmark reading code^0), over p2p (I have janus^1), and possibly uses either the popularity of DownloadNet, or even some of the search/archiving stuff -- without getting carried away -- to assist in delivery or marketing.

I don't know. A clear synthesis right now escapes me, but that's OK. I think there's something there: bookmarks (maybe a special bookmark folder, something referential, like "/var/www/html"), into which bookmarks go and then become public; a lightweight p2p server (that perhaps in some limit future could be federates effortlessly for p2p discovery, but who knows how?). Ugh...still too complex perhaps.

Bookmark folder + p2p + transitive (my bookmark folder includes a link to another person's bookmark folder ~~ somehow).

So it's like that article recently on the homepage "We need webrings" or sth. I didn't think that was particularly a good idea, but now I see at least a partial appeal.

The "link" to another person's p2p bookmark "folder" will instead be a normal www hyperlink that links to the "signalling access point" where you can do the ritual to make the connection.

People may think the weirdness, unavailability (you have to be running the little service in your terminal or as a daemon), and difficulty makes it a non-starter. But I think these "backward" elements, could be a paradoxical strength.

I don't know. I think there's something there. I definitely want to keep pushing in this direction, anyhow.

0: https://github.com/00000o1/Bookmate 1: https://github.com/00000o1/janus

I think there's one major root cause of this: Google. Search has two problems, relevancy to a query but then the sorting of results returned. In Google's early days they emphasized sorting by relevancy/authority related to a query. If you had the foremost site about stunt kites on the web searches for stunt kites would find your site.

But then Google started weighing results for recency. If you wrote a textbook worth of prose about stunt kites and published it all in 2000, there was little need to update the content unless amazing new stunt kite technology was introduced. But when Google started weighting for recency, the definitive stunt kite site dominating results in 2000 started dropping down the ranks for no other reason than its pages didn't have recent dates on them.

This has led to a bunch of problems. For one it punishes anyone that dared to make a website and then not constantly update it. There's no new seasons of Space Above and Beyond so there's no reason to constantly update a fan site about the show.

Second it punishes sites for maintaining a history of content. It's super difficult in Google to search for a topic more than a few years old unless it's got a Wikipedia page.

Third it helps bullshit SEO where some old blog post just gets a new timestamp and maybe a few words changed and Google thinks "oh boy brand new content!"

This powers combine to form a shit Voltron with an inability to find anything that doesn't abuse the shit out of Google.

There used to be some good ways to aid in discoverability before everyone decided XML was icky. If small web search engines actively support such things they could greatly help discoverability.

The Friend of a Friend[0] (FOAF) project is a decent solution to this problem. Not only could you define data about yourself but links to other people. It was useful in that it made distinct the difference to a person's homepage and blog which might not be the same. The Sitemap XML spec[1] is useful for providing a map on your own site to the benefit of crawlers.

[0] http://xmlns.com/foaf/0.1/

[1] https://www.sitemaps.org/protocol.html

delicio.us or whatever how many periods were

This might not be the appropriate thread to ask this, but I feel that if I create an Ask HN with just this, nobody will see it and I won't get any responses.

I've been running a website for over 15 years, something popular, on a specific topic. It has no ads; I only have Patrons who give me between 100 and 200 dollars per month. I don't get many visits (between 800 and 1200 per day, except on special days with some news where I can get many more).

I grew up in the era of web rings and old websites, and aside from being very nostalgic, I would love to go back to that. And I know that if I don't start, others won't either. But I have this constant debate about whether I should do it and follow my desires or what the audience wants. I feel that if I do that, people will stop taking the website seriously. Unfortunately, visitors come to my website, and if they don't see something modern and well-done, they won't take it seriously.

So I'm between a rock and a hard place: do I follow my desires and do what I like, knowing that it may cost me an audience? Or do I adapt to trends to try to get more visits and new readers?

I don't know if anyone has the answer, but I would like to hear your opinions (any type of opinion is welcome).

What do you want from your website: side hustle or hobby? You're allowed to enjoy stuff without making money...

Sorry - I gave the example of the Patrons to give an idea of how much people consume me. I don't do it for the money - I invest a lot of money in fact, paying contributors, etc. I just want to be an important voice in this topic and have some kind of influence.

You could have it both ways with a prominent "retro mode" button to let users switch css?

Nothing says you can't sorta do both. You can have a simple but clean looking website that both makes people nostalgic for the old days yet is still easy to navigate in the modern era, and that website can be part of whatever webrings you want to be part of.

Besides, what people count as modern or usable varies a lot depending on the niche, and if your content is good enough... well, you can get away with a lot of archaic design there. I mean, look at Serebii.net. Biggest Pokemon site in the world, probably the defacto source for information for many people in the community... and it's barely updated its layout in the last decade or two.

Heck, in more niche subject areas you have literally every website sticking to fairly traditional web design principles. Retro gaming and computing sites (like those about 8-bit computers, video game mods or demoscene stuff) tend to look about as retro as their subject area, and very few people care about that.

So, either mix them together since these things aren't necessarily mutually exclusive, or do whatever knowing that if your content is good enough, the aesthetics won't exactly matter a huge deal.

Can't you do both? Stick with an aesthetic your visitors expect, but also add web rings and other feature that are throwback-y but styled to look new?

It's hard to say without being able to understand your visitors, their expectations, and how much wiggle room that leaves you to be you.

This is actually a very lucid way to frame this. People love to complain that "Google is useless now," but it's pretty clearly not the case if you look at how most people use search.

What they usually mean is "nobody can find my interesting hobby projects and I can't find theirs." And that definitely tracks. As a person who poured a lot of energy into completely free, non-commercial educational content, it grinds my gears that there are 2-3 pages of derivative blogspam peppered with affiliate links - and increasingly, LLM-generated drivel - ahead of me.

What I think we get wrong is demanding that others fix it for us, though. Yeah, it's the cool part of the internet, but it's a commercially insignificant one. What the article is trying to do - pick a specific practical solution and lead by example - is probably better. Even if it's a rehash of what we tried in the pre-Google days.

ChatGPT is really, really good at surfacing small blogs and websites on a huge variety of topics. Ask it for a list of personal blogs about running a hobby garden.

It's not a perfect solution – if you ask for a fairly commercialized topic it can struggle to cut through the noise and will return larger websites with SEO-fodder style blog posts, but even with that caveat, it's miles better than Google.

ChatGPT also hallucinates a lot. Many links generated by it don’t work

Eh, I still ran into SEO spam as the top results, even turning web off.

I find this one of the worst parts of ChatGPT. Recommendations are full of advertisements/astroturfing/marketing. If you know any industry with aggressive marketers(For me, its Video games and seeing Nintendo/Stardew Vally populate those lists. Or iceland as a top vacation destination. Or recommending Apple products.)

Honestly sucks really hard. Nothing you can do because the AI is dumb and can't figure out marketing makes words appear more often.

Reply on news.ycombinator.com