Can you explain how COT is essential to distillation and why obfuscating it helps defend against distillation?
Or point me in the right direction in terms of what to read.
Or point me in the right direction in terms of what to read.
By distilling, you are training a model via Reinforcement Learning (RL) to mimic the answer of a bigger model. To do that, you need all the steps that contributed to generating an answer.
To give you a better idea, imagine teaching a student how to solve math problem:
1. You give it the problem and the answer only (no thinking trace) 2. You give it the problem, the intermediary steps and the answer (full transcript)
I think you can agree that the second method is more likely to give a well-informed student.