We may look back on 2022/3 as the last time we had a training set that was clean. The fact that it was already polluted with SEO garbage will be a quaint problem.
I have a completely un-worked thought that the music industry might have some clues. As music becomes more and more entrenched in codified process ie pop culture music. It all starts to become very generic.
I'm starting to notice that music is returning to the old way of discovery. Where you hear music from a 3rd party that is playing it and ask them who that is. There is no longer a good bubble to the top source like radio because the corps have cannibalized themselves with production line pop music that has to play on their vertical stack of companies or be licensed. Seems the stuff I find now is some artist with less than 100k followers like (I'm guessing) back in the pre internet days.
This suggests a sort of cultural event horizon beyond which AI cannot see. Five years from now, the way actual people talk and think about a new topic will be drowned in AI content, creating a gulf between reality as people actually see it and the AI universe. Unfortunately for many people their reality is already slated to become AI-mediated.
I see a lot of uninformed people claiming this is going to doom AI though.
What we're talking about here is a world where 1) the corpus is polluted with an unknown quantity of unlabeled AI-generated content, and 2) reputational indicators (link counts, social media sharing/likes) may amplify AI-generated content, and lead to similar content being intentionally AI-generated.
At that point, can the incorrect info in the training set really be controlled for using noise reduction or other controls?
There is a tribe with a long history of ritual cannibalism that has developed a genetic modification(mutation) that prevents them from getting this disease.