Meta says later on that they aren't releasing it and give no explanation. I wonder why given how incredible it seems to be.
The github repo associated with that paper is linked below. It links to the paper on arxiv, but also has some data in the repo.
Now of course, the terms are not the law (so don't govern the use of the generated data by any third party), they are an agreement between two parties. If you did click "agree" then that's a binding agreement and there could be legal/contractual repercussions (some of which are outlined in the terms).
Unnatural language used davinci-002 although that was a while ago, they only say "similarly" in this paper and don't specify what they used. I can't see a reason why they wouldn't be releasing it if the unnatural prompts were generated by LLaMA2-family.
In any case, replicating this training seems trivial and very cheap compute-wise for anyone who wanted to do it.
It suggests that synthetic training could be the future in increasing capability of smaller models (and perhaps bigger ones too). AI will train AI.
The differences being it's not just training on unvalidated synthetic data and this specific method (per the unnatural questions paper) results in increased instruction diversity which confers some added advantage and I'm assuming explains the performance gain over the also synthetic self-instruct code?
I may be misunderstanding but this seems more nuanced than just training on synthetically AI-generated code and is more validating of synthetic instructions (i.e. low resource setting) rather than synthetic code (i.e. high resource setting).