LLMs put the free web at risk | Hacker News Reader

LLMs put the free web at risk

10 pointsanfernan19 comments

My cofounder and I think the rise of AI has put the free web at risk.

Despite their claims, ChatGPT and Perplexity don’t provide live or complete information from the internet. Instead, they index only the parts of webpages their algorithms deem important. These limitations have given rise to a new breed of AI browser agents powered by the same foundational LLM tech.

These LLMs + agents will likely account for over 90% of web traffic in the near future. Bots scrape content without compensating websites for it. As humans access the web through LLMs and agents more, they directly view websites less. The loss of ad revenue from human visitors threatens many websites’ ability to provide their content publicly for free.

We’ve been considering solutions to these problems, and want to know what you think. 1) Are they problems? 2) How big and serious are they? 3) What are possible solutions?

We have some ideas…

View on HN

> My cofounder and I think the rise of AI has put the free web at risk

To paraphrase Upton Sinclair, “When your paycheck depends on you thinking something, it will be very hard to get you to not think it.”

It's a good line and true! We're not working on anything that monetizes this. At least for now...

I think human written, compelling, deep content will survive. Fluff content sites will likely be squashed by search engines favoring brands and authority over linkage stats because there is so much AI generated content that's mediocre, the search engines have to weed thru that to have relevant results.

Often now though before you see results, you see AI summaries. This is perfectly fine if someone is casually searching something like ratio of oil and vinegar for salad dressing. That person likely is making a salad right now and needs to know and does not care about reading through 5 screens of scrolling content, ads, popups, etc. to get a recipe.

Now if someone is researching the history of salad dressings and wants to go deep into types of oils, vinegars, mixture ratios, pairing with different lettuce types, fruits, nuts, etc. having content that provides that level of information is helpful, but to a smaller audience. That audience is more likely to click links to products or ads than someone just looking for a ratio right now.

The human impression counts for content sites is likely to go down, a lot for fluff content, but those sites are likely losing impressions from people that weren't clicking the ads or buying products anyhow.

I'm not in advertising/marketing so I don't know what impacts that has ... maybe for some time period, ad revenues will go down because sites lose the fluff visitors and number of visitors carries weight in advertising processes, right? Then maybe the processes catch up and realize that the good content sites have higher "stickyness" factor so conversion rates and length of users visit are much more important than page impression counts?

Yeah I mean the salad example is great because in that case it feels ridiculous to have to look at ads.

Maybe we don’t need to protect websites that rely on advertised content? Maybe it makes a better internet?

Based on what you’re saying here it sounds like good content will inevitably go behind paywalls in this case, and the number of paid subs people need to have will only continue to explode. Aka no free internet. Maybe that was bound to happen one way or another.

And all LLM providers will pay their data providers directly by API or something?

original internet (DARPA) was meant for communicating research and science. Maybe we return to that being the primary thing that will continue to be free?

And him and his cofounder created a service that scrapes websites and uses AI for something or other…

https://www.goharvest.ai/

We built Harvest to reduce the pain of gathering web data by clicking through websites to copy data into excel sheets, databases, and CRMs. Something millions of people do everyday.

We recognize it as an unrewarding, tedious, and time-consuming thing humans have had to do until the latest abilities of browser agents.

As we built and learnt more about the industry we started to understand the underlying problems. For 99% of web sites web scraping isn’t the problem, the lack of compensation is.

We think there’s actually a better way to do this. If there’s enough demand, we can facilitate a rev share between agent scrapers and websites. Scrapers will pay less than what they pay for proxies and websites get a new revenue stream.

These are our thoughts at least so far. We aren’t ashamed of what we’ve built by any means in the way your comment implies lol. We want to see if we can benefit both parties in a win-win marketplace.

So what you are doing is scraping the data without asking permission and using AI so people won’t have to go to the original site.

How is what you’re doing any better than what you are complaining about?

1) Public websites don’t require any more permission than taking photos of a public storefront. We abide by privacy laws and make sure we don’t overload website servers.

2) We aren’t complaining. We’re curious how others view this topic and space because it’s a contentious topic. We recognize that we might be able to address the larger issue of lack of compensation for websites being scraped by facilitating a win-win marketplace (only loser is proxy providers).

So you really don’t care that “the rise of AI has put the free web at risk”, you care that it is putting your company at risk when you are doing the same thing and making the same argument that the companies training the models are doing?

Are you paying any content providers now?

Why didn’t you just admit that up front or at least disclose you have a business interest in being able to scrape others content for free?

> human written, compelling, deep content will survive

Remember what we all see now is the worst LLMs will ever be. They are only getting better now and that might even mean there will be a thousand different versions of a given article, each tailored for a give user bracket according to how compelling they predict the version will be to that specific cohort.

> Remember what we all see now is the worst LLMs will ever be

This is exactly right. I could imagine a bunch of horse trainers looking at early gas-powered automobiles in the late 1800's and thinking, "This 'automobile' thing isn't going to change much. Sure it's interesting, but they're so much slower and less reliable than a horse".

Depends on the site! If you're trying to sell products, you don't necessarily care so long as your products are discoverable by ChatGPT/Perplexity/an agent -- visibility/discoverability is definitely a priority for some businesses. If you're selling ads it definitely is a risk, though licensing deals are one solution but that seems to be only available currently for bigger companies.

Definitely! If your revenue comes from selling something to site visitors, you want to be scraped. If you're selling ads, (as far as we know) your options are to either offer an API or make individual licensing agreements. There's no standardized, go-to place to do this.

Getting rid of ad supported content and SEO optimizations? Nothing of value will be lost. There is nothing “free” about the tracking and evilness of adtech.

I have been paying $20/month for ChatGPT with web search just so I could avoid sites like yours.

The first thing I see when I go to your site is a half page cookie banner that wants me to accept tracking.

But you still have to visit websites (or use apps) to do tasks online like booking tickets, getting stuff delivered, or paying bills. They are all tracking us. Do you also plan to pay for AI agents that do these chores on your behalf?

I don't have to use the web or apps for any of that stuff. They're a convenience, but are not actually necessary for any of that.

I guess I was a little overwrought. Ads don’t bother me for ideological reasons. They enshittify everything they touch.

Of course I have an ad blocker. All of my bills are auto pay.

Who, me? Haha

What site are you referring to? Ours doesn't have a cookie banner

I had the wrong website. What you are doing is even worse. You are scraping websites for your own use.

Reply on news.ycombinator.com