Like Llama 1 they won't care about personal use, but no corp is going to touch this.
Between these and FSF you've got pretty much all the accepted pontificating about free / open source software. Reproducibility is not mentioned because it's not really a consideration for software.
Model weights aren't software so there's not an automatic correspondence between the freedoms, but the essential one you might think you need the training data for is freedom to modify and inspect the source.
Modification is fine tuning which you're free to do if you have the weights. And the model weights + code fully define a system that can be interrogated to give a practitioner relevant info about how the model works (within our understanding) that the training data isn't needed or relevant for. I don't see that any freedom on use or inspection is violated by not having the data.
It could be nice to have it of course, but it's more about using it to learn how, not exercising any freedom.
Incidentally, the big freedom that's usually violated is freedom of discrimination against field of endeavor. LLAMA et al list uses and industries they restrict from using them and because of that are not "open".
But the process by which that code arose, the ability to modify any line and understand its impact (heh) on a real execution environment, is dependent on a massive process that required billions of dollars and thousands of the smartest people on the planet. For all intents and purposes, without that environment, it is as reliably modifiable as an executable binary in any other context - or a set of weights, in this one!
For instance, I can step through and even modify that code using tooling like AGC emulators like this one http://www.ibiblio.org/apollo/#gsc.tab=0
What makes it open source is access to the same level of source access that the original developers worked in.
That's what's missing here. Mistral's engineers do not simply open this binary in their editor to do their job.
Models aren't just the weights, but also the list of operations to perform using those weights. I haven't heard a good definition that allows for neural network models to not be software, but allows any other table lookup heavy signal processing algorithm to be software.
> Modification is fine tuning which you're free to do if you have the weights. And the model weights + code fully define a system that can be interrogated to give a practitioner relevant info about how the model works (within our understanding) that the training data isn't needed or relevant for. I don't see that any freedom on use or inspection is violated by not having the data.
Mistral wouldn't constrain themselves to fine tuning if they have a big enough change, they would go back to their build pipeline. This argument sounds a lot like 'there's nothing stopping you from patching the binary, so that's basically as good as source'.
The Open Software Foundation ironically screwed the pooch on this one. Open source commonly means source available, more of less. Free software, as in “'free speech,' not as in 'free beer',” is the cumbersome construction for what open source aspired to mean [1].
I think the naming is a challenge because in English "open" gives the impression that the key point is that you can see it, as opposed to anything about freedom. Is that what you mean?
I have heard it said that open source is sort of a "commercial friendly" version of free software that de-emphasizes user freedom. I think some groups push for that (like Meta is trying to redefine what open source means wrt AI weights). But the OSI defined freedoms basically match what FSF pushes.
They didn't screw up, they screwed the pooch on open source != source available. The Open Group's members--from IBM to Huawei [1]--started calling the latter open source, which set a precedent that's stuck.
[1] https://en.wikipedia.org/wiki/The_Open_Group#Member_Forums_a...
For normal programs, it is quite easy to decompile an unoptimized binary. Even decompiling an optimized will lead to source code. To make this harder, an obfuscator has to be used.
A model is different because it relies on its weight, which are quite a bit more difficult to inspect. Way harder than even obfuscated source code. It is magnitudes harder to make statements about which information it might divulge upon careful questioning, or evaluate its biases, if the training data is not available.
Edit to say that I see benefits to having the training data, just that I don't think it's needed to exercise enough freedom to qualify as open source in an analogous way to software.
Also to add, training on GPUs is not generally reproducible anyway because of execution order.
Also, because people are idiotically afraid of E-numbers, some manufacturers figured they can find whatever fruit or bean is naturally rich in the relevant E-compound, and use that in the process, allowing them to replace E-whatever with ${cute plant name} in the ingredient list (at some loss of process efficiency).
Software wasn't for sure copyrightable before 1976 in the US, but there was plenty of closed source code that simply only ever distributed the binaries.
This right isn't protected by copyright, it's protected by the lack of copyright, and how executables are directly modifiable. So the AGPL won't be possible, but it's a lot like permissive licenses, without attribution.
Also, the license to the model assumes copyright (Apache 2) and works by granting you permissions based on that copyright.
A set of weights is a pre-trained local minima in the model space is a both executable and modifiable. It's usually far more useful than the source training data, because the work has been done.
Tons of binary patching works that way. For instance from the gameshark days it was relatively common for a patch that worked for unknown reasons, but who's discovery was tooling assisted and simply displayed a desired effect (and commonly a lot of undesired effects that weren't clear).
> A set of weights is a pre-trained local minima in the model space is a both executable and modifiable.
None of that changes whether it's open source or not. I guarantee you that mistral has some code (and a lot o data) laying around that created this model.
> It's usually far more useful than the source training data, because the work has been done.
For certain operations maybe. I guarantee that Mistral wouldn't constrain themselves to fine-tuning if they had a major change to make to the model.
With these large models, the weights really are quite useful objects in themselves, the very starting points of future modifications, and importantly the desired points of modifications
Practioners in the field call these models "open source" and write open source software to run them and use them to power open source software.
> With these large models, the weights really are quite useful objects in themselves, the very starting points of future modifications, and importantly the desired points of modifications
For fine tuning. You go back to the training pipeline to make major changes.
> Practioners in the field call these models "open source"
Some do. In contrast to the accepted definitions of open source.
> and write open source software to run them and use them to power open source software.
There's plenty of open source code to run closed source binaries.
Obviously zipping something is not enough, even lossy compression is (obviously) not enough. But then how does a language model differ from a lossy compression? Is it just the compression ratio? (are the weights even that much smaller than the data?)
There are ways to train models that guarantee that the amount of information transferred per data point is limited, but to my knowledge those aren't used (and may be prohibitively expensive).
At the core the question is how much artistic input there is in creating LLM models. Are the choice of the model architecture, hyperparameters and training data artistic choices comparable to those by a photographer setting up a shot? Or are they more comparable to technical work that's only protectable by trade secrets, patents and trademarks?
In the same way, data in a database are usually not copyrightable. They are still the company's property (excluding issues with PII and such).
Also, people go to jail for leaking trade secrets, and this certainly qualifies.
Their weights obviously aren't protected by trademark. So, what IP regime protects the weights?
While there is a discussion to be had on sovereignty and international reaches of local laws (such as DMCA for instance), I think it's disingenuous to consider only US legal point of view.
[1] https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=CEL... [2] https://www.legifrance.gouv.fr/loda/id/JORFTEXT000000573438/...
No US court is going to invent new rights under US law because a company happens to be incorporated in a foreign jurisdiction.
> While there is a discussion to be had on sovereignty and international reaches of local laws (such as DMCA for instance)
The DMCA (and GDPR) do not have international reach. The DMCA only appears to apply internationally because it binds a whole series of American companies with an international presence. Meanwhile the GDPR protects European residents. A company dealing with these residents has crossed the border, so to speak.
> I think it's disingenuous to consider only US legal point of view.
Candidly, you're either confused about what disingenuous means or you're being disingenuous yourself.
And by the fact that DMCA Takedown Notice and processes are valid in many countries. You can send a valid takedown to a french company and they'll take the content down. Not because of DMCA law in the US or because of American companies, but because the notice is accepted and has legal value within the scope of the local law.
> Their weights obviously aren't protected by trademark. So, what IP regime protects the weights?
> No US court is going to invent new rights under US law because a company happens to be incorporated in a foreign jurisdiction.
Your original question doesn't specify "in the US". A French company wouldn't be able to copy these without issues.
Does that mean you need to be protected everywhere to be qualified as protected? Or simply in the US? If so, then you can use your own argument to say that a Chinese court (or Marshall Islands I guess?) won't create a law for a company incorporated in the US. Which in turn means that IP is not protecting anything since it doesn't apply to every single country on the world with no exceptions.
And yet, IP does protect things. You can't both not scope your question and except a reponse scoped only to the US.