Hacker News Reader

Comment by dzogchen | Hacker News Reader

For a model to be ‘fully’ open source you need more than the model itself and a way to run it. You also need the data and the program that can be used to train it.

See The Open Source AI Definition from OSI: https://opensource.org/ai

View on HN

Is it reasonable to expect companies to redistribute 100TB of copyrighted content they used for their LLM, just on the off-chance someone has a few million laying around and wants to reproduce the model from scratch?

Redistribute? No. Itemize and link to? Yes.

With LLMs, the list doesn't even have to be kept up to date, nor the links alive (though publishing content hashes would go a long way here). It's not like you can get an identical copy of a model built anyway, there's too much randomness at every stage in the process. But, as long as the details of cleanup and training are also open, a list of training material used would suffice - people would fetch parts of it, substitute other parts with equivalents that are open/unlicensed/available, add new sources of their own, and the resulting model should have similar characteristics to the OG one who we could, now, call "open source".

Perhaps that's not reasonable to expect, but Meta apparently kind of did it anyway, if not in a way that helps reproduce their LLM: https://arstechnica.com/tech-policy/2025/02/meta-torrented-o...

Actually they did, the entire 15T tokens that were supposedly used for training the llama-3 base models are up on HF as a dataset: https://huggingface.co/datasets/HuggingFaceFW/fineweb

It's just not literally labelled so because of obvious reasons.

The RL-only (no SFT) approaches might remove that issue. Problem sets should be smaller (and mechanically creatable) than the entire western corpus.

Would a reference file with filename, size, source and checksum count towards the OSI definition?

For open source model claiming SOTA performance, we could at least check for data leak from its training data.

Fully agreed, someone actually mentioned that to me on reddit and I modified the content + added a disclaimer on top of the article.

https://www.reddit.com/r/LocalLLaMA/comments/1ilsfb1/comment...

That is incorrect -- you do not have to provide the full training data to meet the requirements.

I recommend reading the actual Open Source AI Definition[1] and the FAQ[2]. There's also the whitepaper[3] that goes into much more detail about the state of affairs.

[1]: https://opensource.org/ai/open-source-ai-definition

[2]: https://hackmd.io/@opensourceinitiative/osaid-faq#What-is-th...

[3]: https://opensource.org/wp-content/uploads/2025/02/2025-OSI-D...

FYI their open source ai definition released with a lot of controversy, unsurprisingly because it had heavy contribution from corporations with their own interests. It's best to ignore it for now until the wider community has decided on an appropriate open source definition.

We need a new definition then.

> you need more than the model itself and a way to run it. You also need the data and the program that can be used to train it.

the model reveals the architecture which is all you need to use/run/train it.

Sadly, the OSAID also does not require training data to be available. :(

Yes. I cannot comprehend this to this day. A model weights data + runner is how different from a closed source executable? Why do everyone call these open source?

Because typically adapting or improving traditional code to your needs is very difficult without access to the source code and build files.

For an LLM you can finetune and enhance, distill and embed given just the model weights, the runtime, and a permissive license. Having more is better. Well written detailed model release papers help a lot. Training code and training data are a great bonus.

However, I find the purity contest a bit too dismissive of the great contributions to the AI dev ecosystem that Meta and Deepseek have brought us. Without these, there wouldn't be the open ecosystem we have today.

It's not a purity contest, it's a clarity contest. If Meta and Deepseek want to operate the way they have been, where they release baked models and whitepapers, that's fine - and you're right, it's certainly more than they're obligated to release. They just shouldn't be calling it "open source" when the source is literally not open.

Eh, I can kinda see it. It depends on your definitions of words. People have been muddying the waters with what "open source" means anyway. I have known it to mean code released under an open source license. Other people use it to mean programs where the source is available regardless of license. I would use "source available" to describe that, but some people strongly disagree with my definitions.

If I write a program, then obfuscate it and then release the obfuscated code under an open source license, would you consider it open source(I would)? That's kind of the case here, they are releasing the model weights under an open source license.

Personally, I think it's fine to shorten it to "open source model" instead of "a model with the weights released under an open source license". What I would object to is releasing model weights under a restrictive license and calling that open source.

> If I write a program, then obfuscate it and then release the obfuscated code under an open source license, would you consider it open source(I would)?

I wouldn't. Most definitions of open source say something like "in the form used for editing". You can release a built binary under an unrestrictive license, but that does not mean that you've opened the source. It's literally the plain meaning of the words: the source, as in where the thing comes from, needs to be open for it be meaningful.

Because practically speaking, you can fine tune them I suppose?

But that's also true for binaries, games are a good example of where people pushed this quite far. Based on what little experience I have in ML, I'd say it's about the same thing. Whereas an API is more akin to a piece of software you can't tinker with in any way.

Guess the bar is just lower in the LLM space :P

So a proprietary program with a lot of knobs and configuration files is kind of opensource?

By what appears to be the logic for "open source AI", a locally executable proprietary program would be "open source" (because you can meddle with the executable). To me, that's mostly just "not SaaS". But somehow, a different definition appears to have stuck for LLMs than for other types of software.

Open source becomes really complicated once assets with unclear license are involved in any way. Lots of people for example would say that Jedi Knight 2 is open source because Raven Software released the source code and tools needed to build the game. But that alone doesn't mean you can run it, because you still need to get a hold of all the assets (models, textures, sounds) which may or may not still be property of LucasArts or its successors. Even if you have them, it's actually unclear if it is legal to use them this way. So while there are tons of people working on mods and conversions, noone in their right mind would distribute all the source assets.

Much in the same way, no sane company will touch the legal nightmare of releasing LLM training data scraped from public websites. Even releasing the LLM alone might be infringement, there are literally court cases being fought over this right now.

Games like that, or the open-source clones of commercial games that require original assets to play (e.g. OpenXCOM), actually give a very clear analogy here: open source does not mean open assets. The software code is under a separate license from the data it processes. Emulators like Dolphin are kind of in this situation too - the program is open, the data it processes is not.

And that's fine! It's still valuable to have access to the source code, even if the "batteries" aren't included. Of course, if you really want to call it an open source model you should include the source for the data scraping/cleaning stages too; then the only thing missing would be the compute time and risk of acquiring dubiously-legal inputs.

I personally prefer a taxonomy like:

* Open weights: you can download the artifact and run it locally, not just use it through an application like chatgpt or an API.

* Open source: the code that created the artifact is provided in the same format that the authors used to work on it.

* Open data: the dataset that the source code was used on is available for download.

All three of those could be individually licensed or released, for 8 possible combinations. In the analogy to games, they would correspond to the licenses on the retail binary, the source code of the game, and the original uncompressed art assets or Blender projects, respectively.

If it has already been established that open source doesn't mean open assets, why would we change that now? After all, training data is literally nothing but assets - except that you don't need them to run the application. So in that sense open LLMs are more open than these games.

But the training data isn't open...

I agree that open source doesn't mean open assets, but neither does open assets mean open source. You could make a linguistic argument that the training data is part of the "source" of the model (as in, from whence it came), but in any case the point is moot because neither the training data nor the code is open.

OK, but that leaves the tools used to train the model (aka the build scripts). These could be open sourced.

Because marketing (open source is a buzzword after all), and the media just repeats what they read in press releases verbatim. But most people working with the models themselves call them open-weight, except for some occasional exception like OLMo that publishes the dataset and training scripts and is actually open source.

Because Meta called them in this way to differentiate themselves from "Open"AI and everyone else followed the suit.

Because Meta called llama that shortly after it got leaked and it stuck.

The AI crowd doesn’t care much for licenses anyway.

Reply on news.ycombinator.com