You aren’t the only one. I wonder how distillation of LLMs is done, didn’t research it and hoped this might be a solution. Well, it might be a solution but one that come with more hallucinations.