it might not exactly be a waste, when you are inputting your training data you need some context on what the source is and then add a weight to how valid it is. That is how we learn, if i talk to a phd on some topic, I would pay close attention to what they say and give it more significance, maybe even write it down, compared to a youtube video on that topic, compared to a guy on a street talking about that same topic, ill probably nod my head and walk away and try to forget as soon as i can. but there probably still is something valid to learn from that last conversation, how a certain type of person structures sentences, responds, the words used etc...

do these llms get trained with something like a credibility weight on the training data? that was it seems they did in this paper, just manually curated that