Meta is accused of "bullying" the open-source community
economist.com
economist.com
The perfect is the enemy of the good, as usual.
The whole crux is to get an objective definition of what "open AI" is, and how it needs to behave. If others agree (or disagree) with the vendor that doesn't mean the vendor is permitted to make its unilateral de facto declaration a de jure consensus decision.
That's not how standards work.
Is it supposed to give the false impression that OpenAI and Meta’s actions have been equivalent with respect to releasing their models? And if it isn’t an attempt to mislead in such a way, then what exactly does it mean?
I guess they’re trying to call out OpenAI more than group them with Meta
Yes, some data is open and the creators of it want it to be shared openly.
Other people want their copyrights upheld and for there to be no intellectual property appropriation (theft) of their creations.
There is also the issue of aggregation. For example, Wikipedia is an open and free aggregation of knowledge. But it also cites, quotes, and links out to sources which may be copyrighted works.
Note that Wikipedia has very formal guidance as to how to avoid and handle instances of plagiarism.
https://en.wikipedia.org/wiki/Wikipedia:Plagiarism
AI needs to face the fact that it is rife with intellectual property theft. It needs standards for infringed copyright removal. It needs standards for the defense of trademarks. It needs privacy standards to prevent the unauthorized use of personal data, images, recordings and other works from people around the globe.
If AI is permitted to simply glut itself on every artistic creation and every human on the planet, sans limitations, it is basically the equivalent of a crime spree.
So, the choice is give up everything that gives you any advantage, and hence any incentive to develop anything in the first place, or you’re the bad guy.
I’m not sure how they think the world works. But I agree that we should be clear about our standards. So I’d suggest that those with these expectations adopt a new brand other than “open source”. Maybe “puritan approved”.
Except it's Mark and Meta, which are household names, and I'm some Internet rando, so my voice is smaller, hence the accusations of bullying. If someone decides for you that your new name is shithead, and everyone goes along with it because that person's bigger than you, thats bullying.
That’s really the equivalent of what’s happening here?
In this particular case, meta's models are not "open source" although Zucks calls them as such. And because of that, people also started to believe they are open source.
Open Source AI will be able to reference and share its underlying truth data. Imagine an Open Source AI trained on all GPL code or a Public Library and it could guide you directly to the middle of the source of any code / book it knows.
"But not everyone is going to spend the money or time to train their own models from scratch!" I hear you say. And there's some truth to that. But if we had truly open source LLMs, then AuroraGPT development would be fast tracked and we would have a fully government funded scientific frontier model instead of only fine-tuning models.
This intention necessarily translates to models. In order to reproduce a program, the source code and a relevant compiler is what is required. In order to reproduce a model, the architecture (source code), training data and initial parameters are required.
With code, you just compile and run it. Models require carefully curated training data (lots of it), training code, final weights, production inference code, etc.
The danger is the world gets drunk on Meta's models, but then Meta pulls the plug. Meta is doing the world a lot of good with Llama, but we'll be left in a precarious position should they stop being so generous. The next generation of LLMs might make Llama completely pointless.
Meta thinks a great deal about the strategy in this space. They license other weights and models as CC-BY-NC and other non-free licenses because they know that they have the SOTA models across several categories and that there's no other competition in the marketplace. They want to retain their advantage when it benefits them.
So what happens when the LLM space shifts and Meta no longer needs to appear to be open source? What if the value switches over to diffusion models and away from LLMs, which is an area where Meta shines?
Llama is a gift we can't replicate or properly repair and extend. And we need to be careful.
The point is cultivating dependency under the color of "open source" is not generally wise.
Users of the "open source" LLM do not have all the software freedoms normally associated with "open source."
They cannot rebuild it and more. Important pieces are missing the "open" part.
Someone says, "here, take this awesome math library, it is open." Then users find out there is a brittle binary blob needed for the whole thing to work.
Nobody will want to build on or with it because that blob works, until it doesn't and when it does work, people are not sure what it does exactly.
Makes the whole thing a lot more like free beer.
In the scheme of things right now, particularly given both the pace of change, and up coming corruption due to feeding models their own output, anyone's latest may just not be relevant that long.
The fact that llama is not open source does not make it less useful or less appreciated. Nor should anything be considered open source because it is useful or appreciated
All of this needs to stop.
The OSI's definition is still new. In fact it's still in draft form so this debate is a bit premature. I suspect that companies will begin releasing the code to train the model once it is finalized (they will never tell you the training data due to legal reasons and forcing them to is a losing battle).
This last sentence gave me a headache. He does not need to publish a bunch of possibly proprietary data owned by Meta, so no he won't bare it all. Data isn't free.
I don’t think this is an issue about open source, or pragmatism or utility or even ideology very much. There’s one big tech company that makes open source models available. Instead of rewarding this behaviour, some people think that attacking them for it instead is the best response.
I think this is a primarily social phenomenon which really just boils down to “meta bad” with various post hoc justifications tacked on.
The model architecture and infrastructure are open source. That is what matters.
The fact that you get really good weights that result from millions of dollars of GPU time on extremely expensive-to-procure proprietary datasets is amazing, but even that shouldn't be a requirement to call this open source. That is literally just an output of the open source model trained on non-open source inputs.
I find it absurd that if I create a model architecture, publish my source code, and slap an open source license on it, I can call that open source…but the moment I publish some weights that are the result of running the program on some proprietary dataset, all of a sudden I can’t call it open source anymore.
Then you don't understand open source. You would be distributing something that could not be reproduced because its source was not provided. The source to the product would not be open. It's that simple. The same principle has always applied to images, audio and video. There's no reason for you to be granted a free pass just becauase it's a new medium.
1. I publish the source code to a program that inputs a list of numbers and outputs the sum into a text file. License is open source. Result according to you: this is an open source program.
2. Now, using that open source program, I also publish an output result text file after feeding it a long list of input numbers generated from a proprietary dataset. I even decide to publish this result and give it an open source license. Result according to you: this is NO LONGER AN OPEN SOURCE PROGRAM!!?!!
How does that make any fucking sense?
You have the model and can use it any way you want. The model can be trained any way you want. You can plug in random weights and random text input if you want to. That is an open source model. The act of additionally publishing a bunch of weights that you can choose to use if you want to should not make that model closed source.
Programs aren't the only things that can be open-source. LLMs are made of more than just programs. Some of those things that are not programs are not open source. If any part of something is not open source, the whole of that thing is not open source, per definition. Therefor any LLM that has any parts that are not open source is not open source, even if some parts of it are open source.
While a model does require weights in order to function, there is nothing about the model that requires you to use any weights provided by anybody, regardless of how they were trained. The model is open source. You can train a Llama 3.1 model from scratch on your proprietary collection of alien tentacle erotica, and you can do so precisely because the model is open source. You can claim that the weights themselves are not open source, but that says absolutely nothing about the model being open source. The weights distributed are simply not required.
But more importantly, under your definition, there will never exist in any form a useful open source set of weights. Because almost all data is proprietary. Anybody can train on large quantities of proprietary data without permission using fair use protections, but no matter what you can't redistribute it without permission. Any weights derived from training a model on data that can be redistributed by a single entity would inherently be so tiny that it would be almost useless. You could create a model with a few billion parameters that could memorize it all verbatim.
Open weights can be useful, and they can be a huge boon to users that don't have the resources to train large models, but they aren't required for any meaningful definition of open source.
That may very well be so. We'll see what the future holds for us.
They’re not required for it to be open sourced.
But I've definitely seen modifications to Llama3.1 floating around. Correctly me if I'm wrong, but open-weight models can be modified fairly easily.
print(“hello world”)
Are you going to tell me I cannot open source this hello world program because it was produced by an algorithm that I have not open sourced?