Hacker News Reader

Comment by skilled | Hacker News Reader

What's next for models like GPT now that a lot of sites will outright block CCBot, GPTBot and others? How big of an impact is this going to have on the LLM itself? Isn't OpenAI in a bit of a pickle in regards to this?

The problem with my question is the following:

Content gets syndicated anyway, so if DigitalOcean blocks GPTBot (which it does), pretty much every single one of those tutorials will be syphoned off to other sites, which are unlikely to block GPTBot themselves. How will DigitalOcean (or any other company) address this?

It looks to me like it's Catch 22 in every direction you look, and unless you're someone like The New York Times who can afford to outright protect the data with licensing...

It's just something thats been on my mind lately but I don't understand the finer details of it.

View on HN

It's going to be a walled garden dystopia, with everyone and everything asking to sign-up, subscribe or whatever to monetize content.

Scraping & siphoning content, ad blockers, both sides have been in arms race for years, AI and LLMs will just be the last straw, before almost anything of value is behind a pay/subscription wall.

Is all the data on the internet from 2010 to 2025 that much more valuable than all the data on the internet from 2010 to 2022? Better data yes, but more data up to the present day? Do we really need that to keep improving AI?

GPT is already capable of incredible generalized language understanding and I'd wager we've long since hit diminishing returns from raw internet data. RLHF, fine tuning, and better (and more data efficient) architectures are what we need now.

People have somehow conflated artificial intelligence with "oracle that knows everything" and so keeping models up to date with recent knowledge has become a must. Of course, that could be done by fine tuning techniques and better architectures that can outsource knowledge retrieval to tools, all very interesting ongoing work on that topic, but even these approaches require to not be "blocked" in their data retrieval tasks for them to work well.

I'm sure you can do lots of interesting research using outdated datasets but for companies creating products this will not be sufficient.

The “somehow” being “it was explicitly marketed that way”

Language shifts, so NLP models need to understand those. For example, compare the use of "gay" before ~1980 and after. Some words change in spelling, like "to-morrow" present in works around 1800 changing to "tomorrow" in current usage. Some words are also coined, like "woke" or "enshitification".

If your data or models don't account for those then it can make mistakes. For example, if a model is only trained on modern sources (and does not know about Early Modern English 2nd person pronouns "thy"/"thine"/etc.) then it can easily get confused when determining parts of speech, which then affects other down-stream processing.

I always imagined a great source of training would be subtitles of live news or other live programming.

Would help to stay up to date on current events, and up to date on its general understanding of the world.

With the risk being adopting the bias of news media.

People involved in AI have incentive to say 'yes' to everything.

Nobody will make anything value in public, that they didn't want to be released for free anyway.

ChatGPT's vacuum has brought back a desire for privacy and will probably contribute to destroying piracy too.

ChatGPT has destroyed the 'study hard and get reward loop' for collaborative effort on the internet. If you use chatGPT, it absorbs all your question data and gives you nothing in return. You can't commit to random people, as they are expected to leak your IP onto gpt.

Isaac Newton using chatGPT would upload the core of calculus to GPT in research questions, and see no personal benefit for doing so.

There is no greater thief in history of academic work, than electronics.

SPAs being difficult to crawl is a feature, after all! :)

I did a random sample of some news sites

across the political spectrum: every single one I checked, other than ft.com blocks GPTBot

Laws. And the lawsuits that arise when such theft is discovered.

Easy to work around. Contract someone outside of the jurisdiction to provide a dataset, then it's up to them to deliver it to you. You "weren't aware" of the data source until people outside the organization starts shouting or the police starts asking questions, but the model "doesn't contain any of the data" so you continue shipping the model/product.

You will need to train the model in the same jurisdiction too, to avoid any kind of intervention into the training process. Ideally that would be a shelter company that "sells" you "training services" in that jurisdiction.

I don't know a lot of judges, but shenanigans like that (maybe the second or third time) are a great way to get a summary judgement, and find your way to county for a few days to think about the intent of a law.

Who is going to enforce those laws?

Big publishers are more than happy to settle with AI companies - they just their slice of the pie after all. But who is going to protect, say, your Hacker News comments? Are you going to sue the AI company? Is YC going to sue? Are you going to sue YC for not banning crawlers in their robots.txt?

Why would I even bother doing that? I just write comments, it's not some ineffable wisdom. I'm writing them in public. I don't expect to profit from them somehow.

Heck, collecting whatever cents I might be owed for being a drop in the ocean is a losing move in my country.

My guess is the big players hope is to steal an enough content and then build a self training LLM based off synthetic content (rehashed original works) before the theft part matters. Not sure how close they are to achieving but this seems to be a common SV gamble.

Steal or do something shady, raise enough money / power so by the time your noticed, you have the money to win in the courts.

You already see the propaganda about “none of this should matter because the cure for cancer is on the way courtesy of AI.” I mean maybe it it but it smells fishy to me.

An LLM can be trained to find relevant knowledge online. It doesn’t have to be trained on the entirety of all existing knowledge.

> An LLM can be trained to find relevant knowledge online.

Why do you think chatGPT lost its Web Search plugin lately? Copyright lawsuits. You can't even use copyrighted content in the prompt because it will make the model makers liable.

But this doesn’t make sense - how is using ChatGPT to find information different from using a search engine? Especially if ChatGPT clearly lists its sources?

Good luck when everything is paywalled

Given the already huge cost of training, and the evident lack of concern the LLM folks seem to have for copyright, why wouldn't the AI groups purchase subs to scrape the paywalled content?

The would possibly need to apply some effort to appear human, but that should only throttle the rate, not stop their scraping all together.

It's more difficult to scrape pay walled content no?

Clearly, places like Reddit have wised up to this and are making API usages non-free for example, so while it's not impossible, you can see the limitations being put into place already. Twitter is another one.

It seems like all this data is now considered gold and people lock up gold?

Reply on news.ycombinator.com