We already see huge amounts of LLM generated garbage on the web, as blogspam, regular websites, etc. Amazon is getting flooded with LLM-written books. Chat bots/search engines regurgitate the actually valuable content
If things continue, soon the motivation for generating new content will trend towards zero.
By the web do you really mean google and FaceBook? Most of the channels/feeds I follow have not been flooded with spam. I am pleased that the large corporation content aggregators are struggling. It isn't like they produced content themselves anyway.
only the ones trying to make money out of it. a lot of people create content because it has to be let out - it's like art. people have stuff to say, and they love saying it.
I see it differently, I see that this generated nonsense will be filtered out, like SEO 'hackers' way back in the day. And those real articles, who actually have some insight will be valued a lot, the issue at hand here, is that the handcrafted unique data that was created, will be pooled without consent to a training set for further changes.
It's the equivalent of the rich eating the poor in my eyes just on automated steroids.
Edit : To avoid all doom and gloom, a method to guarantee your site/data is not added to a training set is required. This sounds good on paper to me, and is really needed, but just like peoples personal privacy, I don't think the sentiment is there to put the effort in.
But from a copyright standpoint, unless it regurgitates training data which is rare and only happens with a specific prefix, LLMs should be safe. They are decomposing, recombining, and regenerating language, not copying it. They execute user commands, not imitating any author on purpose.
Say you want to read Harry Potter without paying, would you rather borrow a book or ask chatGPT to regurgitate the original? It would never work well, and be slower and more expensive to use AI. It's not a tool for infringement, it will naturally hallucinate and degrade the original, it can't possibly store a perfect copy of everything it trains on.
In fact LLMs often use 15T tokens for 15B models, so 1000:1 compression ratio, while diffusion models compress 5B text-image pairs into less than 5GB model, so they hold about 1 byte worth of information from each training example if averaged out. There is no space to put all that copyrighted data in a model, it necessarily compresses the hell out of it. Plain old piracy is still 1000x 'better' than infringement by AI.
From a copyright perspective, I do agree, you're not reproducing someone else's work. However, it's a 'new-ish' area of open source licensing, if your product is a product from others peoples product without at a bare minimum, citations (given the author said 'go wild with this data'), it's maybe legally rude but not a problem. But without permission, IE; no permissive license to USE someone's content, it's SOME kind of new infringement ?
> n fact LLMs often use 15T tokens for 15B models, so 1000:1 compression ratio, while diffusion models compress 5B text-image pairs into less than 5GB model, so they hold about 1 byte worth of information from each training example if averaged out.
This is wild and I didn't know this, but again, the gravy was made from bones.. I'm not sure it matters here ?
Probably not very relevant to the point, but gravy is made from the meat juices and fat. Bones get made into bone broth.