Though it does seem to he very coherent, I am still very suspicous of test data contamination in Mistral.
XWin was the SOTA llama finetune last I checked, so we will see how it fares with Mistral...
XWin was the SOTA llama finetune last I checked, so we will see how it fares with Mistral...
But they are not supposed to train on these test questions. If they do train on those test Q/A pairs, intentionally or not, the answers will be conspicuously good. A purposely seperate test dataset like that is standard in basically all ML research.
But Mistral has given basically zero information about the Mistral 7B training dataset. And they have less reputation to lose from "cheating" than Meta does.
Is there any reason to believe this issue is more inherent to Mistral, other than because Meta and the like have a larger reputation? Has Meta released the relevant information about their training data and evaluation?
Purposely search for the big standardized ones in the training dataset and filter them out. If some scattered examples are reworded enough to be undetectable, thats probably not a big deal.
> Is there any reason to believe this issue is more inherent to Mistral, other than because Meta and the like have a larger reputation?
Its not any one specific smoking gun...
- Even with Mistral's recent code dump, the release is still so light on detail that its kinda sketchy to me.
- The base model beats llamav2 7B in metrics by a lot.
- There is zero information on how to reproduce these metrics. This was also an issue for Llama, as testers had trouble reproducing their scores.
- Mistral 7B doesn't seem to handle random questions completions as well as llamav2 7B base in my quick tests (though this is not true for the instruct model. Mistral 7B instruct "feels" quite intelligent).
I am not a data scientist, I don't want to point fingers. But there are llama finetunes from AI startups with similar red flags, and one I briefly tested autocompletes hellaswag questions suspiciously well, even with some of the "incorrect" answers from that dataset: https://huggingface.co/datasets/Rowan/hellaswag/viewer/defau...
I have this sneaking suspicion that AI may have an "crypto moment" where everyone realizes lots of eval scores are fraudulent, and that significant parts of the training datasets are questionably legal.
Consider the following thought experiment. Let's say we train a glorified benchmark table. We train it extensively on benchmarks from a number of different providers. MMLU, Hellaswag, Winogrande, etc.
Such a model will do well on benchmarks, but its reasoning ability, obviously, will be severely below what we'd expect from those benchmarks.
What happens when we tune it on a dataset like Orca? Will we observe improvements on the benchmarks? Well no - if anything, we might even expect some regression as the model doesn't have sufficient reasoning ability to handle these directly, just the answers it was fed during pretraining.
The fact that we don't observe this with Mistral, and that finetunes that are designed to improve reasoning ability lead to an increase in the corresponding benchmarks as we'd expect, leads me to believe that Mistral's performance is legitimate.
Even apart from that - it's worth checking out StableLM's recent 3B release. While it didn't get as much attention due to literally releasing the day after Mistral and they didn't promote it until afterward, StableLM 3B boasts some significant gains in comparison to its predecessor as well due to training for significantly longer: https://stability.wandb.io/stability-llm/stable-lm/reports/S... and these gains are likely to be more substantial at larger parameter sizes.
There's still a lot to be gained from training models for longer, and I can fully believe that Mistral's level of performance is possible in a 7B model.