Aside from the sliding attention window, I don't see them mention any particular reasons for the incredible performance. I would like to see some tests for benchmark contamination in the training set.
I am suspicious of contamination in every finetune I see, and very suspicious in a new foundational model like this.
(For those reading and not following, "contamination" is training a model/finetune on the very test it will be tested on. Normally these known tests are specifically excluded from training datasets so the models can be properly evaluated, but throwing them in is an easy way to "cheat" and claim a model is better than it is.
In a foundational model with a huge dataset, there's also a high probability that well-known evaluation questions snuck into the dataset by accident).
Its supposed to be searched for and filtered out in the training datasets, but I can even see an active effort to weed them out missing some test Q/A pairs.
Being or not being open source has exactly jack shit to do with that.
The idea behind OSS is that you're able to modify it yourself and then use it again from that point. With software, we enable this by making the source code public, and include instructions for how to build/run the project. Then I can achieve this.
But with these "OSS" models, I cannot do this. I don't have the training data and I don't have the training workflow/setup they used for training the model. All they give me is the model itself.
Similar to how "You can't see the source but here is a binary" wouldn't be called OSS, it feels slightly unfair to call LLM models being distributed this way OSS.