Copyright is just not prepared for AI. Training with copyrighted material could become "officialy legal" under copyleft terms, at least when the amount of training material exceeds a certain threshold.
EU Digital Single Market Directive (2019/790) Art 3 and 4 allow text and data mining. Art 3 for scientific purposes, Art 4 more in general.
Now, some people argue that AI models are somehow compressed databases of the data that was crawled; but that seems patently ridiculous to me - so this should be sufficient.(at least mathematically) (IANAL) (famous last words)
The only question then is if the models have some kind of additional value ("intelligence") beyond being compressed databases.
My take is that either no, or the burden of proof is on those making the claim. Until they prove it, they are just databases and therefore derivative work of their input and their output is also derivative work.
* LLM's aren't databases, you can’t query them for exact stored records, and they can’t reconstruct (most of) their training data.
* But they also don’t reason or understand exactly like humans do either.
They're something else: to wit, Transformer models.
1) I don't think being able to query them and reconstruct input 1:1 are requirements. If i build a shitty db with a buggy query language that retrieves incomplete data and occasionally mixes in data i didn't ask for, then it's still a db, just a shitty one.
If i populate it with copyrighted material and put it online, whether I am gonna get sued is likely based on how shitty it is, if it's good enough that people can get enough value from it that they don't buy the original works, then the original authors are not gonna be pleased.
2) Yes, comparisons to humans are not always useful though I'd say they don't reason or understand at all.
Either way the discussion should be about justice and fairness. The fact is LLMs are trained on data which took human work and effort to create. LLMs would not be possible without this data (or ML companies would train on just the public domain and avoid the risk of a massive lawsuit). The people who created the original training data deserve a fair share of the value produced by using their work.
So the real question to me is how much so they deserve?
My point is that any situation where person A puts in a certain amount of work (normalized by skill, competence, etc.), person B uses person A's work, puts in some work of his own but less than A, then gets more reward than A, is fundamentally unfair.
LLMs are this, just at a massive scale.
But to be honest, this is where the discussion went only after thinking about it for a while. My real starting point was that when I publish my code under AGPL, I do it because I want anyone who builds on top of it to also have to release their code so users have the freedom to modify it.
LLMs are used to launder my code, deprive me of credit and deprive users of their rights.
Can we agree this is harm?
I also believe than unfairness is fundamentally the same as harm, just framed a bit differently.
> deprive me of credit
> and deprive users of their rights.
> Can we agree this is harm?
I might consider it if any of those claims were true.
I think the opposite is true -- especially with Open-Weight models which expand user freedoms rather than restricting them. I wonder if we can get the FSF to come up with GPL compatible Open-Weight licenses.
At this point in time I'm not entirely convinced they even need to. But if future lawsuits turn out that way, it might solve issues with some models.
Please step back and consider what you are replying to.
> LLMs are used to launder my code
If an LLM was trained only on AGPL code, would it have to be licensed under AGPL? Would its output?
> deprive me of credit
They _obviously_ deprive me of credit. Even if an LLM was trained entirely on my code, nobody using its output would know. Compare to using a library where my name is right there in the license.
> and deprive users of their rights.
I appeal to you again, re-read my comment. I am not talking about users of the model but users of the software that is in part based on my AGPL code. If my code got there traditionally by being included in a library, the whole software would have to be AGPL and users would have the right to modify. If my code is laundered through an LLM, users of my code lose that right.
> Can we agree this is harm?
So all of those things are true. And this is clearly harm.
Stealing a little from everyone is morally no different than stealing a lot from one person. Whenever you think about ML, consider extreme cases such as training on data under one license and all the arguments to pretend it's not copyright infringement fall apart. (And if you don't think extreme cases set precedent to real cases, then please point out where exactly you draw the line. Give me a number.)
Spreading the harm around means everyone is harmed similarly but that is not the kind of fairness I had in mind.
They are called large language _models_ for a reason. They are just statistical models of existing work.
If anybody seriously thought they were intelligent, they'd be arguing for giving them personhood and we'd see massive protests akin to pro-life vs pro-choice. People (well, ML companies) only use the word "intelligence" as long as it suits their marketing purposes.
That is actually an intriguing idea and at least aligned with the reasons I use AGPL for my code.
If we could extract correct attribution and even licensing out of work that was produced with AI, I don't think it would help that much. I would even assume that especially in this case the rich would profit the most. They wouldn't care having to pay thousands of artists for the pixels they provided to their AI generated blockbuster movie. It would effectively exclude the poor from using AI for compliance reasons. Or even worse rich corps monopolize the training data and then they can create content practically for free, while indies cannot use AI because they would have to pay traditional prices or give the rich corps money for indirectly using their training data.
I still don't think that's enough to be fair. If their work is used to produce value ad infinitum, them any one-time payment is obviously less than what they deserve.
The payment should be fractional compared to the produced value. And that is very hard to do since you don't know how much money somebody made by using the model.
> It would effectively exclude the poor from using AI for compliance reasons.
Again, this only an issue if you're thinking in terms of one-time fixed payments.
If it was possible to train AI models on just the public domain, them I am sure ML companies would have because it's less effort than lobbying and risking lawsuits (though I am surprised how well creators have accepted that their work is used by others to profit without any compensation, I expected way more outrage).
Virtually all code relevant to training code-completion LLMs is written by people still alive or dead for way less than 70 years. We can try to come up with a better system over the next decades but those people's rights are being violated right now.
At least for Java there are search engines to look for code that call libraries etc. Models could probably be trained on free code and then be fed with the results of these search engines on demand even through the client who calls the LLM:
Client -> LLM -> Client automation API -> Code search on client machine to fill context -> Code generation w/o model that is trained on the code that was found through the code search, but merely used as context
Even if they only feed code into it, that is freely given, I think the difference in quality of output would approach the current quality and better over time, especially when using RAG techniques like the above.
Companies can also buy code for feeding the model after all. So beside the injustice you directly experience right now over your own code probably being fed into AI models, do you fear/despise anything more than that from LLMs?
You could make separate versions of an LLM depending on the license of its output. If it has t produce public domain code, it can only be trained on the public domain. If if has to produce permissive code (without attribution), then it can be trained on the public domain and permissive code. If copyleft, then those two and copyleft (but it does not solve attribution).
> Companies can also buy code for feeding the model after all.
We've come a long way since the time slavery was common (at least in the west) but we still have a class system where rich people can pay a one time fee (buying a company) and extract value in perpetuity from people putting in continuous work while doing nothing of value themselves. This kind of passive income is fundamentally unjust but pervasive enough that people accept it, just like they accepted slavery as a fact of life back then.
Companies have a stronger bargaining position than individuals (one of the reasons beside defense why people form states - to represent a common interest against companies). This is gonna lead to companies (the rich) paying a one time fee, then extracting value forever while the individual has to invest effort into looking for another job. Compensation for buying code can only be fair if it's a percentage of the value generated by that code.
> So beside the injustice you directly experience right now over your own code probably being fed into AI models, do you fear/despise anything more than that from LLMs?
Umm, I think theft on a civilization-level scale is sufficient.
As long as everybody can also benefit from it, I see it as some kind of collective knowledge sharing.
As you stated in the paragraphs before unless the wealth distribution changes, LLMs may lead to an escalating imbalance, unless the models are shared for free as soon as a critical mass of authors is involved, regardless of who owns the assets.
1) The current system is that the rich get richer faster than the poor.
2) You're proposing a system where everybody gets richer at the same rate at the best of times (but probably devolves into case 1 IMO)
3) I am proposing a system where people get richer at the rate of how much work they put in. If a rich person does not produce value, he doesn't get richer at all. If a poor person produces value he gets richer according to how many people benefit from it. If a poor person puts in 1000 units of work and a rich person puts in 10 units of work to distribute the work to more people (for example through marketing), they get richer at comparative rates 1000:10.
My system is obviously harder to implement (despite all the revolutions in history, societies have always devolved to case 1, or sometimes case 2 that devolved into 1 later). It might be impossible to implement perfectly but it does not mean we should stop trying to make things at least less unfair.
---
We're in agreement that forcing companies who take (steal) work from everyone to release their models for free is better than letting them profit without bounds.
However, I am taking it way further. If AI research leads to fundamental changes in society, we should take it as an opportunity to reevaluate the societal systems we have now and make them more fair.
For example, I don't care who owns assets. I care about who puts in work. It's a more fundamental unit of value. Work produces assets after all.
---
And BTW making models free does not in any way help restore users' rights provided by AGPL. And I have yet to come across anybody making a workable proposal how to protect these rights in an age where AGPL code is remixed through statistics into all software without making it also AGPL. In fact, I have yet to find anybody who acknowledges it's a problem.