If a model is trained only on synthetic data, is it still possible it will output things like this? https://x.com/elder_plinius/status/1952958577867669892
The problem is that the size of the training set required for a good model is so large, that's really hard to make a good model without including almost all known written text available.
If phi5 is trained on synthetic data only then info on how to make drugs must be in the synthetic dataset.
Well, it could also make inferences. Like, it could find a new mathematical proof, even if that's never in the training set.
For an obvious example of generalisation: the models are able to write more code than there is in the dataset. If you ask it to write some specific, though easy, function, it is very unlikely it is present verbatim in the dataset, and yet the model can adapt.
It's not particularly likely that the hidden information encoded in synthetic data would happen to include specific details for making LSD or VX, but it's much more plausible that synthetic data contains some information the model's trainers would prefer to not incorporate in the model.