Distilling Step-by-Step Outperforming Larger Language Models with Less Training

Deploying large language models (LLMs) is challenging because they are memory inefficient and compute-intensive for practical applications. In reaction, researchers train smaller task-specific models by either finetuning with human labels or distilling using LLM-generated labels. However, finetuning and distillation require large amounts of training data to achieve comparable performance to LLMs. We introduce Distilling step-by-step, a new mechanism that (a) trains smaller models that outperform LLMs, and (b) achieves so by leveraging less training data needed by finetuning or distillation. Our method extracts LLM rationales as additional supervision for small models within a multi-task training framework. We present three findings across 4 NLP benchmarks: First, compared to both finetuning and distillation, our mechanism achieves better performance with much fewer labeled/unlabeled training examples. Second, compared to LLMs, we achieve better performance using substantially smaller model sizes. Third, we reduce both the model size and the amount of data required to outperform LLMs; our 770M T5 model outperforms the 540B PaLM model using only 80% of available data on a benchmark task.

(2) Hypothesis: The success of distilling chain of thought learning is that it provides a pseudo large context window. The input pair (label,rationale) allows the student LLM to mimic having a large context window (the context window of a human agent that informs the system by providing the rationale data). So it reduces the embedding distance between words for which entailment is difficult to obtain with short context windows.

(3) The above hypothesis suggests to perform the training process in two phases (a) and (b): (a) A small model is used as a teacher, this allows the learning model to connect d-distant words. (b) A larger LLM is used as a teacher, the training process continues with new data using the rationale provided by a stronger larger LLM.

In phase (a) the learning system learns to infer d-distant words relations, in phase (b) the system is already prepared to relate (2*d)-distant words because it have developed the necessary skills phase (a).

(4) The two-phase approach can be directly generalized to a multi-phase approach using three meta-parameters: the number of teachers, the relative power of each LLM teacher, and the percentage of data in the training set for each teacher.

(5) Need of pruning: From a geometrical point of view, each phase allows the system to strengthen relations between distant words, increasing latent spaces density promotes part of the latent space getting disconnected, that is creating outliers. Pruning should reduce the risk of outliers that promote hallucinations. Outliers are the germ for establishing relation between very distant words but the rationale data perform that function well, so removing outliers don't pause the learning pace.

(6) It is clear that the rationale data enhances the attention mechanism. This suggests modifying the attention mechanism by giving appropriate weights to the attention with respect to the rationale and self attention for the tag.

(7) It would be desirable for the authors to release those small and powerful LLMs. Therefore giving individuals the power to enter into the SOTA realm and allowing us to create new low latency and cost applications with enhanced privacy.

Edited several times for grammar and new ideas.

That sounds too good, can someone more knowledgeable comment?

If we think of intelligence as the ability to reverse entropy (as in being able to compress information into the smallest representation) there's this weird conceptual idea where a strong enough model is able to transfer its knowledge like this without wandering off track and teaching the wrong things.

It'd be interesting to have formal benchmarks of this ability because I think it is closely related to things people care about in the generative space. For example H2O recently introduced a nice new LLM with a good license. But informally in my testing is seems to hallucinate much worse than Llama derived models - for example it my testing it hallucinated that "Crown Prince Harry" (!) married Phillip Mountbatten (!!)[1].

I suspect but can't prove that a model like this would be much worse at distilling than a Llama based model of the same size.

Testing this knowledge transfer would be really instructive I think.

[1] https://twitter.com/nlothian/status/1653267700288987138

What could impact the ability to discern whether it's llama or open llama? Are there artifacts of a given Llm that could be exposed via distilling process?