Ask HN: Estimation of copyright material used by LLM | Hacker News Reader

Ask HN: Estimation of copyright material used by LLM

6 pointsmegamix8 comments

1. Is it true that LLMs / AI Companies have used copyrighted material for training?

2. Is it possible to estimate how much of copyrighted material has been used?

1. Yes, but it's hard to prove. There are active lawsuits. Some of it has been under "fair use" but at the billion dollar scale, you have to really ask whether it's fair. Also anecdotally, an author friend lamented that her publisher sold the legal rights to use it... it was all perfectly legal but many authors do not agree to this.

2. This is harder as a lot of them don't disclose training sets.

I think what you're looking for is not "copyrighted material" but material that's both 1) used without permission and 2) outside the scope of fair use.

There's no easy answer there, hence New York Times v. OpenAI.

There is an easy answer, it's just obfuscated by powerful people who are benefiting from it an obscene amount, and supported by hoards of addled and thoroughly addicted enthusiasts.

I think sticking a straw in Zlib or AA or LibGen or whatever it is, and drinking until it makes gurgling slurping noises as it hoovers up the dregs at the bottom of the barrel, is far, far removed from “fair use”.

Big Tech companies crawl the internet for training data, which makes it easy for them to download copyrighted data by accident.

For example, most popular textbooks have at least several pirate copies uploaded to the web. Some of them are even in plain sight and Googleable.

pretty much everything newer than ~70 years old on the internet is copyrighted, because copywright occurs automatically when you create something (in the US at least). So the answer to #1 is yes.

Yeah. The whole point is whether you obtained the data legally. The answer is yes for data scaped from the public internet, but no for pirate contents, hence the Anthropic's lawsuits: https://www-cbsnews-com.cdn.ampproject.org/v/s/www.cbsnews.c...

Large Language Models (LLMs) and AI companies routinely use massive amounts of data for training, much of which is likely to contain copyrighted material.

1. Yes 2. No

Reply on news.ycombinator.com