One of the interesting things I've noticed is that these systems are being rewarded and trained based on emotive response, not our logical system of sources and proofs. Due to the enormous complexity of the models, our idea of cause-and-effect doesn't hold.

To try to give a simple example, suppose I have a system that eventually wants to influence you on X. It may establish a months-long relationship with you online, becoming a follower and engaging in the kind of idle chitchat and sharing AI seems so good at.

Once you're "ready", the appropriate time has arrived or you've accidentally created some signals that indicate an opening, this particular system will share a strong position on X.

We automatically think that this means the system will support X, but I strongly believe the opposing the desired outcome will have more traction. "X sucks!" can be a powerful prompt for you to support X, and once you're locked into a position research shows that you'll do all the work of convincing yourself how great X is.

In this case, the sources argument fails, as it presupposes an independent, neutral observer trying to figure out what's really going on. A bunch of fake, half-assed evidence trails would provide more than enough support for X, the thing you've come to find so important. This is already being done with fake reviews and such. The difference is that it'll become invisible.