Hacker News Reader

Comment by voxic11 | Hacker News Reader

Distillation a pretty well documented technique that actually pre-dates LLMs https://arxiv.org/pdf/1503.02531

Here is a project that guides you through it if you want to prove to yourself that it works https://github.com/arcee-ai/DistillKit

View on HN

I took GP as asking for evidence that the reduced-cost Sol is actually a distillation of the previous-cost Sol. AFAIK, providers distilling or quantising models and offering them as the same model have not been proven.

I doubt he was claiming that. He's probably saying that the ability of Chinese companies to be able to distill frontier US models has put downwards pressure on the price of all models.

I'm saying that it is unclear that without distillation this wouldn't still be happening. There is a massive narrative that no one but OpenAI, Anthropic, and Google can make a model without distilling. But there's basically no evidence of that.

I think the (non-definitive) evidence is that the chinese models are always just slightly behind the publically released US models, and never ahead.

I think it's more that distilling is enormously cheaper than training from scratch to achieve the same results

Alternatively, modern AI is good enough at optimizing its own kernels that it just keeps pushing costs down. Unlike the semi-decentralized inference provider community, OpenAI has both the talent and the compute to throw at the problem of making their models much more efficient to run.

GPU kernel optimization is just the kind of well-bounded problem with clear success criteria that AI loves.

That distillation exists isn't the question.

It's about evidence this is an active force in competition in LLMs.

I think the biggest actual piece of evidence is how hard the major players are trying to stop it

To be precise, the distillation mentioned in this paper is not the distillation used by other model companies. In the one mentioned in the paper, your teacher and student model typically have similar architectures - and you typically need access to the full logits. What happens here instead is motivated by the fact that these companies don't have access to the training data and compute that anthropic/openai have. The distillation they do basically amounts to using traces from ant/oai models trained on much more data with a lot more compute (in many cases including the hidden intermediate tokens! turns out there were many ways to coax it out) and then either directly training on it or using it in many ways in post training pipelines. It falls under imitation learning, IMO.

Reply on news.ycombinator.com