They published the weights. It's an "open weight model", a term that it seems nearly everyone has agreed is appropriate. Why is this company not using it?
That's not totally true - for example Ziphu (Z.ai, developers of GLM) have published their Slime RL-training framework, developed together with Tsinghua University.
https://github.com/THUDM/slime
Also, if you read Moonshot's Kimi 3 report, it does give a lot of architectural detail and details on training.
https://github.com/MoonshotAI/Kimi-K3/blob/main/k3_tech_repo...
Some discussion on this by HuggingFace here:
https://www.youtube.com/watch?v=MW8-kqd2SD8
Yes, we all realize that "open weights" is more accurate than "open source", and anyways the source code would not be very interesting - it's the training data and methods that mostly define these models.
I can understand not wanting to call it open source if they don't give you the algorithm and software used for training, but wanting the training data too? That's completely different
You can‘t reproduce the LLM without the same data
People keep trying to shoehorn OSS concepts onto model weights, but the concepts don’t fit because the weights aren’t software. They aren’t compiled code. They are learned parameters to use with a (very big) function that itself is expressed in the code.
So they’re a very valuable asset that complements the code, but they are not the code. You could use randomly initialized weights and the software will work - it will output tokens. They just won’t have useful patterns.
I don’t think OSS definitions have ever required that assets have their source included. For example, artwork is very important to a game, but nobody thinks a game is not open source if it doesn’t come with sketches and a copy of Adobe Illustrator to recreate the artwork from scratch.
Edit: The distinction I would draw between models like Deepseek and models like OLMo is whether they are open science. With a model like OLMo, they have published everything you need to replicate the training experiment. Whereas Deepseek does share a lot of knowledge, but keeps a lot proprietary too.
Source as a word has a meaning that doesn't include non-source things.
And we have Open Weights to define what those are.
BAR is an open source game. All the game source is open.
> Open source RTS game built on top of the Recoil RTS Engine
BAR is a game, Recoil (a different thing - not the repository I linked) is an engine it is built on top of. Recoil is - incidentally - a fork of spring, an engine many open source RTSes are built on.
I'm not sure how static training data is either (or how you'd distribute it considering its size and nevermind the legality of sharing copyrighted things).
You'd likely get a model with very similar behaviour but the weights would be different.
Please someone correct me if I'm wrong.
Model training now is not a straight forwards process of input data -> run tools -> get model.
There's a whole lot of alchemy going on. We don't quite understand what works and what doesn't. Think of it like painting with water color and having to improvise very often.
The only advantage over water color is that we can revert to a working state.
Really, it’s no more open source than a free calculator. It’s free, and you can use it to no or great effect. But, you sure as hell can’t make one.
So calling it open source is without question, wrong.
It's mostly ad hoc scripts and some pretty horrible hacks being run by a hundred engineers trying to improve a thousand different things at once with a hundred thousand GPUs.
I'm sure once we understand the tech better, the training process will look like running a program.
Open source has benefits even if you can run yourself. You can read the code for understanding/insights. Other labs could replicate/build on it.
I sure wish I had a few 100M of disposable income to train a frontier model.
> or in the future it could be useful when training is cheaper.
I do not think that physics will allow hardware getting that much faster. But maybe we will have different, cheaper architectures by then.
Unfortunately, electricity prices did not fall by the same factor, so I fear that training a frontier model will still cause a an unsustainable dent in my monthly budget.
https://chatgpt.com/share/6a6f3f3b-7624-83ed-a11a-248cb39728...
Cant stand when you guys try and force something as impossible on the rest of us simply because you could never accomplish it.
(Also, I know Tom Jobbins (TheBloke), and have personally contributed to increase the adoption of GGUF, e.g. in the transformers and ktransformers libraries, so I find the personal dig quite amusing.)
The weights are literally a binary blob.
Note also that software is copyrightable and model weights aren’t (in the US anyway).
One can equally argue that firmware is just settings for the CPU.
If weights aren't copyrightable then doesn't this argue against the idea of them being able to have the quality of open-sourceness? Meanwhile the training code could be copyrightable.
The weights are the asset that makes it useful, but I don’t think 40T tokens of pre-training data should be required to call the model itself open source when you can inspect every line of code in the model without that, as well as instantiate and run the model with randomly initialized weights.
These models are a combination of a small amount of code, a ton of training data, and a ton of expertise in how to train them (which I suspect includes how best design/curate training sets for different model improvement goals).
The Chinese models are mostly very well documented in terms of architecture and training processes/flows, with what is missing to recreate them being the training data.
You don't need the source code - just read their architecture docs and implement it yourself.
The Chinese have actually been very open about training, starting perhaps with the DeepSeek-R1 paper which told the world in detail how to train a reasoning model. The Kimi 3 paper also gives a lot of training details.
There really isn't much of a comparison to be had between building a traditional software project where all you need is the (maybe open source) source code and the Makefile that automates the build process, and building a machine learning system where it's primarily about data not source code, and even with a road map of what may be very complex training (cf build) process, you'd probably still have a hard time building it since AFAIK the training process may still involve expert knowledge and intervention - I don't think training has been reduced to a hands-off "Makefile" or build script.
So, basically lack of source code is the least of the issues in being able to build one of these models - it's mostly the training data and training processes/expertise that you would need.
...and because that training data is missing, they can't be replicated. Which means that you cannot assert that the Chinese are being open in their LLM development, because there's no way to verify that the techniques they describe are actually the ones being used.
The reason that the training data is missing is that they're trained on a large amount of American copyrighted data and distilled on American models, which is where a lot of their performance comes from.
Ditto for training algorithms and procedures such as Slime or DeepSeek's details instructions on how to build a reasoning model.
This is the exact value of openly shared details - others CAN copy and try them and modify them themselves.
Yes, the training data specifically has not been released for any model, American or Chinese, but that doesn't detract from what has been shared, and the reason the Chinese are not sharing data are no more nefarious than why the American companies are not sharing - because they are all using data from sources they don't want you to know about, and at the end of the day the data is the closest thing any of them do have to a moat.
That's not related to my comment. My comment was pointing out that you can't verify something that wasn't published. You have no idea what fraction of their techniques they're not publishing, and how much they contribute to their model performance, because you cannot replicate the models, because they don't publish their training data.
> and the reason the Chinese are not sharing data are no more nefarious than why the American companies are not sharing
This is moving the goalposts. Your claim was that "The Chinese have actually been very open about training", which is false, as discussed. Nobody ever claimed that the American labs were open.
Why would you be concerned about THEIR model performance ?!
Surely if you are an ML researcher and read about a new technique, you are interested in how YOU may be able to use it.
If you are Ilya Sutskever sitting at OpenAI in 2017, and happen upon Google's "attention" (Transformer architecture) paper, then what you do is go and implement it for yourself, get yourself some training data, and try it.
What you are NOT going to do is whine about not being given their source code, or their training data, or their training harness, or a dump of Google's corporate secrets. You take the research that has been shared and evaluate it for yourself.
I'm not "whining" about anything. You made the claim "The Chinese have actually been very open about training" and I showed that that was false. That's all that there is to it.
Apparently you are unaware of it and not benefiting from it.
Oh well.
>a term that it seems nearly everyone has agreed is appropriate
Models being considered open source even if the original training code / data is not released also is something almost everyone has agreed to be appropriate.
Open source means open source code. Open weight means a binary file dump, not unlike an exe file. There is nothing open source about it.
Its like having a closed source text editor that censors certain words, and an open source text editor that censors certain words.
The latter can easily be recompiled, the former requires reverse engineering. Both may give you a license to use them freely.
I do prefer open weights as being more precise (like, is it even really software that has source code in the first place?) but I feel like at this point the ship has sailed somewhat (though if this is something you're willing to spend your time arguing then... moral support I guess?)
[1] ...setting aside hardware attestation bullshit, which does need to die in a fire.
I would be very interested to hear someone explain what freedom it is they need the training data to enjoy.
It's the 2nd time I hear this argument and I'm already fed up with it
Is it the preferred way only because training is expensive? It's like saying binaries are the preferred way of modifying program because you can't afford to have a fast enough machine to compile it yourself.
Most people don't have resources to compile their own browser, but what makes some browsers open is access to the source.
Maybe it's the preferred way because the people sharing models are themselves working with weights and no data?
No, it's the preferred way because that's literally how you train it. Contrary to popular misconceptions, you don't "compile" data into weights. You initialize a model (based on architecture, config, etc) and then you modify it via training. But crucially they (i.e. model creators) modify it the same way (technically speaking) as you would. That's what the license grants you. Nothing less, nothing more. The "how" as in knowhow has never been something covered by a license.
Open-weight is something dreamt up by people misunderstanding the basics of model creation and training, and having ideological biases against AI and/or LLMs. It is what it is, but you should know that you are technically wrong. A model released under an open source license (Apache, MIT, etc) is an open source model. Because the weights are the source of the models. Training is not "compilation". Training is the "how" as in knowhow to modify the model. Training deals with values. Source deals with everything, including values.
In the past, if someone would have released a piece of software (say a PID controller algo) w/ hardcoded values, no-one would bat an eye. LLMs are just that, with billions of hardcoded values. Nothing less nothing more.
They’re not really even being precise. The relevant software freedom, from the FSF is []
The freedom to study how the program works, and change it so it does your computing as you wish (freedom 1). Access to the source code is a precondition for this.
Some might argue that without the training data you couldn’t do some classes of experiments to see how it works, say leave-one-out retraining. I’d argue things like that are not really about the model but about ML research or the class of models, which while interesting is not a free software pre-requisite.
[] https://www.gnu.org/philosophy/free-sw.html#four-freedoms
The weights are the output of a program, it's a binary. It's not source.
Now there are reasons you might want to know about the training data when you wouldn't care about the authoring process used by a traditional open-source process. And these get at the reason LLM's are different than traditional software and so maybe our existing definitions of what 'open-source' means aren't a good match for LLM's. Of course there is software associated with LLM's (beyond the weight) -- defining the structure of the particular neural net those weights fit into. In every open-weights model that I'm aware of that software is open source (though trivial).
Yes. The next time you go to your doctor you should hope he's not being overly precise; or the engineer that builds the bridge; or the software engineer that implemented the embedded software in your insulin pump.
There is no precise thinking without precise terms.