Hacker News Reader

Comment by ipsum2 | Hacker News Reader

Yes and no. In terms of LLMs, it's basically figuring out how to exfiltrate information from GPT4 to remove costs of data gathering. The limitations of that are that the model will never be better than gpt4, and when gpt4 produces incorrect information, the model trained on synthetic data will also do so.

In other fields like computer vision, synthetic data is useful for generating ground truth data, like for depth masks.

View on HN

This is mistaken. You use GPT-4 to generate new data using other data sources, for example text books.

Asking GPT-4 to create 50 new conversations from a chapter of a textbook creates higher quality data than most of “the pile” and can extend beyond what GPT-4 has in its existing dataset. This is exactly what MS has done with Phi.

I’ve disputed that fact before. It depends what part of your data is generated from GPT4 if I have high quality code and I’m using gpt4 to synthetically generate variations on how some could ask for the code to be written it’s entirely possible because of the high quality code for the model to be better. It’s not all or nothing if you’re mixing synthetic data with quality sources.

In this case even though parts of the dataset are synthetic the bound is on the code not necessarily the 50 ways I got gpt4 to say “write me a script to do x” or modeled other interactions with that code data source.

If we’re talking about model distillation[0] I don’t think the student can ever be better than the teacher as optimising for speed and smaller model sizes inherently means that there will be precision loss. Even if the student is as big as the teacher, there is still data loss.

[0] https://arxiv.org/pdf/2210.17332.pdf

Reply on news.ycombinator.com