It's because they want to restrict AI companies from stealing content, but they can't do it if internet archive proxies it all for them.
All of the LLMs would be massively less useful if it wasn't for scraping the latest news.
All of the LLMs would be massively less useful if it wasn't for scraping the latest news.
Every LLM company can afford to spin up a new subscriber account every day, proxying to appear different IPs from all sorts of ASNs, do some crawling until the account gets banned, and then do it again, and again, and again.
What's the conclusion from this train if thought? Just because some burglars can pick locks doesn't mean you should leave your front door unlocked.
Locking a door (or robots.txt) is how one can establish mens rea for those who bypass the barrier.
The actual root cause is that we're allowing LLM companies to completely disregard copyright laws for their profit. Whether the LLM companies scrape the Web Archive or the original source doesn't change the copyright infringement implications in any way, and cutting off the web archive doesn't practically change anything (because as I understand, LLM scraping is already prolific all over the web).
That isn't relevant to ordinary media outlets because a) they don't have enough content volume for rate limiting to be effective since it's possible to get everything they publish even at a slow rate limit, and b) getting AI scrapers to subscribe to their bulk download API instead is not the objective in their case.
Which means LLMs have a zillion sources to get the story. Removing any given subset isn't going to prevent it from having the information in the training data, all it does is prevent that subset from being archived for future humans.
There really isn’t even a defensible argument as to how this even should be illegal. The idea that someone can read words about a concept, and then rewording an explanation of that concept somehow violating the rights of the original author, is absurd.
The issue here and elsewhere isn’t LLMs. It’s that IP as a concept has always been a dystopic farce. Despite this we have not only kicked the can down the road on addressing this, we’ve doubled and tripled down and built our society around the concept. The advent of AI has simply blown the scale of the problem up to the point where it cannot be ignored any longer.
How many people do you think use LLMs in some fashion at all in their daily lives? Genuine question, I'm sure my personal experience is a biased sample, but so is everyone else's. Stats from AI companies isn't going to be (seen as) objective either. OpenAI and Anthropic are pushing a feature where I get a situation report at 9am like I'm an important official. With both labs pushing that, I think some people are getting their daily news from LLMs, the question is how many would it take for it to be meaningful, and how would we know if/when that bar gets crossed? What are the implications of that?