Yeah the article makes it seem like "just use rationality and facts" is missing a huge part of the equation; I feel like it's common knowledge that this is not the gap in modern convincing. (and don't you dare try to use facts to persuade me otherwise!)

More likely in my opinion it's the context that matters here. If people know they're trying to be convinced of something and know they're talking to an AI, they may feel like they can trust (what they consider to be) an unbiased and rational AI. There's probably a fallacy associated with the assumption that an AI convincer is more rational and fact based, while it's more probable that the AI in the real world is, in fact, funded and trained by a think tank that wants you to vote against your own self-interests.

Facts and evidence always are at the core of all arguments, the persuasive thing. The problem with most people is who is speaking. If you are enemy, the things you say are bad.

I suspect that LLMs not being human allows people to not just anthropomorphise the robot, but they project themselves upon it. When someone speaks to the LLM, they're kind of, or actually just literally talking to themselves. But then something new! 'Themselves' suggests new information to themselves. And now without the scary / icky meat and blood human on the other side, said person accepts the argument on its own merits.

Of course, the concern now is arguments based on false or statistically hacked data.

I more wonder if it's pure facts and evidence can work, or facts presented in a certain light can work? I guess asking how mush is it the data is presented that's doing more work than the data itself.

"Hackenburg found that models trained to become more persuasive also ended up being less truthful."

It sounds like the arguments were basically AI slop hallucinations.