I wrote up this blog post in 30 mins, that's why it reads a little rough. I could not find explicit research on the impact of contradicting training data, only on the general need for high-quality training data.

May be it is a pipe dream to drastically improve on hallucinations by curating a self-consistent data set but I am still interested in how much it actually impacts the quality of the final model.

I described one possible way to create such a self-consistent data set in this very blog post.