A couple of months ago, when the news about "Reddit to sell its user-generated content to Google," a redditor asks the community to start generating false comments so the data would be of bad quality to train the models on. They start brainstorming some very funny comments that go something like: "Blue Whale is the largest fish", "WW II started in 1943", You name it..
Or so google's AI responses pulled from Reddit. If you give enough people a hot microphone to a large enough audience, some people are going to say some obscene, untrue, and funny stuff. Maybe that's 1 in 10. Maybe that's 1 in a 100,000. Reddit karma is cheap.
The models being sycophantic and suggestible is another issue, but it’s not hard to get them to agree with false information.