that would require actually curating the training data and eliminating sources that contain casual conversation
too expensive since those are all licensed sources, much easier to train on Reddit data
too expensive since those are all licensed sources, much easier to train on Reddit data
One way to think of it is that any little bias or undesirable path in your teacher model will be amplified in the resulting data and is likely to become over represented in the student model.