Hacker News Reader

Ya, I'm in a similar boat (https://shepherd.com). Despite sky-high engagement stats, we lost 70% of our traffic from Google. Luckily, we have a strong brand, a growing community, and get decent traffic directly (and from other search engines).

We are surviving, but we've changed our entire focus as the web we know is a dead man walking. AI bots are the new frontend for the web, and I am curious about what the web will look like in 2 to 10 years. I hope the AI companies start paying to access the data as they answer queries or construct dynamic frontends (but I doubt it).

I have many friends who ran great indie websites, and it has been sad to see that 90% of them are now frozen or closed.

I expect to lose a lot of information websites, especially when the outcome of a long research is a few bullet points (such as product/place recommendations). Once tech companies feel the need to monetise LLMs, all bets are off.

Ya, I'm already seeing a bunch disappear.

I am curious what it looks like when you need a good product review in two or three years. Is YouTube the only trustworthy source by real people? For most products, the web is just going to be Astroturfed Reddit posts and Forbes AI-written junk that didn't even have the product to test.

I'm hoping paid MCP data sets might get there, then I could make my data available there, and hopefully make a $1 to $4 per thousand requests to the data set. If that replaced the loss of Google traffic, that would be amazing.

I'd love for 25% of my AI subscription to go toward that each month, i.e., I can pull "a list of all science fiction books the NYT recommended in the 1950s" and it gives NYT a few cents to go through their data...

If you want readers more than impact you might want to consider something like a Substack. The paying audience would be smaller, but, it's not crawlable by AI (at least not today).

Still, it's a bit ironic that your own website advises people to use bots to get apartments (thus bypassing the work put into the adverts and apartment hunting sites), whilst also decrying bots who would bypass your site.

More generally perhaps there's a market for a Steam-like walled garden but for "old web" style sites. Crawlers would be controlled via technical means, micropayments would be built in, and use of AI would be strictly controlled. For instance maybe the garden uses AI to build link directories like DMOZ used to be, but the information itself isn't trained on so you're guaranteed to have organic readers.

>Still, it's a bit ironic that your own website advises people to use bots to get apartments (thus bypassing the work put into the adverts and apartment hunting sites), whilst also decrying bots who would bypass your site.

I believe these are pretty different scenarios. There's an ongoing, massive shortage of apartments in Berlin, and bots here often aren't bypassing "work put into adverts and apartment hunting sites"—they're trying to get an application in at all before the listing is flooded with hundreds of applications and disabled. The content of the application has nothing to do with it.

(Of course, this just leads to an arms race of apartment-application-bots, but this is the situation the city's found itself in. Same goes for registration appointments, residence permit applications, and so on.)

If you have a zero sum game, any advice you give is creating an arms race. The only solution is to have fewer Berliners or more housing.

Or eliminate rent control, allowing prices to float to the level people are actually willing to pay. You only see bot-driven stuff like that in cases where desirable goods are being deliberately under-priced.

This will benefit the wealthier strata of society, but why does the city belong to them more than anyone else?

Why should it belong to people who are good at botting? That's a much less fair way to allocate resources than wealth. In a free society wealth is used to buy things, that's the point of earning it. If you don't generate more wealth and you trade it away then eventually you end up poor again. If you're generating wealth continuously, then the things you can buy with it are the reward for doing so. It's much better for everyone that desirable apartments go to people who earned them rather than those who randomly managed to shave some milliseconds off a TCP connection.

The wealthy can buy bots then.

A walled garden or paywalled website would go completely against the website's ethos. Everything is meant to be free for everyone. "Information wants to be free." Besides, those who stand to benefit the most from that information are the ones who can the least afford it.

The bots are not a solution, they're table stakes. Withholding that information would not do anyone any justice.

If your goal is for information to be as free as possible, then having AI read it is the best possible outcome. But it sounds like you have other goals too involving personal satisfaction, maybe community building etc. All very fair and reasonable goals to have, but they inherently conflict with "information wants to be free".

It's not. It strips me of my income, meaning I have to survive through other means. Of course I have other goals! One of them is to make rent.

For the readers, it also strips the information of a lot of nuance. I obsess over word choice for a reason. Then there are the footnotes, the citations, and the purpose-built tools embedded in the content.

As I said, I also lose access to the community and its feedback and encouragement.

That's now, before tech companies start throwing ads on top of the information it hoovered from my website. I'm glad that for a while, destroying the independent web will create value for the shareholders.

If information is free, then the places it is accessed from should not be limited.

The premise behind libraries being public works rather than, say, private collections is that society as a whole is benefitting from this information being free and creating that information as well.

Here you are limited by the whims of a private actor which can be one way one day, and another way when VC money runs out or when it has a "fiduciary responsibility to shareholders" which is the inherent conflict the original poster is sharing.

Having all of the worlds information being parsed by private actors or non-profits who may become private at some time is a problem and the anthesis of "information wants to be free"

Just a little bit of encouragement: this is a really, really nice site that reminds me of growing up on the 1990's internet. You can tell you, "actually give a shit" about what you're sharing and that alone I think gives it credence and relevance. Don't let a silly thing like economics keep you from this worthwhile human achievement pal.

I came across your website organically and it's been a great resource for me in the last few months, including times when the answers presented by Gemini have been definitively and legally incorrect. Thank you for organizing all of this information so clearly.

This is a great information website, congrats, Google is happy: "thanks for feeding me", they will scrape all and display on their site in the nice table. Sorry about that.

> My main mission is to put new information on the internet. It's harder to do this if AI destroys the economics of it. It's also harder without an audience who provides feedback and encouragement.

I also run an online website that provides valuable information. However, we have a very different philosophy. The mission is very not not to "put new information on the internet." Rather, it is to provide a valuable and much needed service for people in our niche. While we do greatly benefit from SEO, we also work very hard to cultivate our audience, both in terms of finding new people, and nurturing our relationship with the existing ones. "Information on the internet" has zero inherent value without an audience. If you're not working to build that audience (outside of third party platforms), then it's going to be very tough in the long run, and you'll run into situations exactly like this.

FWIW, our SEO traffic has remained shockingly steady, though the search console has reported a dramatic increase in "impressions," likely due to featuring our website in AI generated summaries. We'll see if things change over the long-term, but for now, we're not changing any strategies based on this.

The problem is that there is no audience without a third-party platform nowadays. At least not at a scale.

Google was the lesser evil, but I still fully expected to get killed by an algorithm update some day.

I vowed to ride this website into the ground if I have to. I believe in its value, even if I need a side job to support it.

There is no "nowadays." There never an era of big Internet audiences without third party platforms. The early web only had small audiences, and the reason it worked was that people were okay with that.

You could have a small audience of a few million users, not try to compete with the giants, and still, be serving millions of users (daily)

The internet was a beautiful place

> He's just not getting paid anymore.

Or the credit. Or even just the feedback.

Even LLMs get at least some feedback.

This is a surprisingly taxing change. There's a difference between giving free eggs to your neighbours, and giving free eggs to the Holiday Inn across town.

Exactly.

Thanks for sharing. I've been thinking of creating a non-revenue online guide to my new city, Lyon, France, but I've been concerned that I'd only be feeding the AI machine.

Excuse me for veering off topic but I saw this, was intrigued (I used to live in Lyon), browsed some of your previous comments here, saw the book Carmilla, which I'd never heard of, and am very happy. I will be reading J. S. Le Fanu, and am stunned to learn of him. Cheers!

Major free-rider problem, this is—one, however, that property rights could protect against: You own your data. If the OpenAIs of the world want to ingest your data, they pay you for it.

A good strategy can involve betting on alternative distribution channels beyond search engines. Here are some ideas:

- Newsletters. You own your audience and your list is portable and interoperable.

- PWAs with push notifications. Be careful not to overwhelm users with too many messages. Make opt-out options easy to find.

- Door flyers with QR codes. A return to the basics that can still be effective.

- Social media. Still useful, but its reach and engagement seem to be declining.

I am providing information for specific situations. People don’t casually read about replacing lost keys until they lose their keys. It’s not entertainment.

We need a new information backbone

That was the idea behind Internet2.

Never used it, maybe someone can explain how it has evolved.

its super helpful. supporting. you should also accept crypto!

My niche has been dominated by indian slop for a decade now who cares about a few robots getting in on the action?

What I can gather from your comment is that the problem is capitalism, not AI.