Hacker News Reader

Comment by scuppernong | Hacker News Reader

"He looked at me incredulously. Surely, a smart person like me should know that AI, or better said, AGI will be hyperpersuasive soon – already on a bunch of benchmarks it exceeds professional debaters at persuasion." I'm always tickled by this tic of this kind of person, that human traits are something that we can maximize to infinity. That if we trained a robot to tell jokes, it would eventually tell jokes so funny we'd die laughing etc. You can't convince someone who doesn't want to be convinced.

View on HN

Hyperpersuasion is not necessary. AI needn't convince the smartest among us, it need only convince the masses. Which, thanks to the widgets in our pockets, is a highly distributable task. The other risk profile is that of the hyperhacker: a sufficiently advanced AI can worm its way into untold internet-connected systems. Wars are being fought drone-to-drone already without a reduction in human deaths. Every drive-by-wire car and plane is a weapon waiting to be deployed.

> You can't convince someone who doesn't want to be convinced.

I think this is true. But it doesn’t get rid of the possibility of hyperpersuasive bots, just, points to a reason they might not be as consequential as expected. I think bots that are very good at writing rhetoric might happen (why not? They are already good at math and chess), and they would be devastating to the spaces where people go online and argue with anonymous strangers. But I don’t know if that’ll matter. Could be bad for bloggers. Complex for policy people.

IMO if we’re really honest with ourselves, I think we get more of our opinions by stewing in memes and other social processes. Offline or online. The sort of hyperpersuasive ML that we really should have been worrying about was the Twitter/reddit/Tick-Tock engagement algorithms.

yes of course LLMs are already wreaking havoc by scaling up the bot farms of yore but that's not really how I interpreted hyperpersuasion, I interpreted it as a specifically targeted one-on-one conversation with belief in a specific notion as the desired outcome. I believe that LLMs can logos and pathos-max but (depending on the interlocutor) they cannot overcome by sheer force of logorrhea their lack of reputation, credibility, presence, etc.

This has been a core belief of the EA/LessWrong/Rationalist people for many years - that above a certain level of intelligence you effectively get hypnotic powers, and people will obey AIs no matter what. So they said you can't keep a superpowered AI airgapped as merely offering a chat interface will immediately take control of the next person to interact with it and they will deliberately compromise the airgapping.

And if AI wrote a book about these hypnotic powers, then it would be incredibly convoluted, long, and yet worn as a badge of pride to have survived it.

Thinking about it, much of the output I'm seeing nowadays looks like a chapter out of infinite jest.

Maybe that's unfair on infinite jest.

A complete mischaracterization, as usual for HN lately when discussing AI or LessWrong. Obviously, even average levels of persuasion are enough to convince some people. And nobody is air-gapping AI.

I thought the lesswrong folk's belief in mind control was an established fact:

https://rationalwiki.org/wiki/AI-box_experiment

https://www.yudkowsky.net/singularity/aibox

https://www.lesswrong.com/posts/Bnik7YrySRPoCTLFb

It's far from the craziest belief that's come out of that group.

> And nobody is air-gapping AI.

https://genai.mil/

I mean... I would hope that AI used for military needs is not deployed in the public internet.

Maybe less "far out" than hypnosis is just straight up logical debates.

This was a plot point in the Halo lore actually, with Mendicant Bias being reasoned with to defect against its own makers.

Well, we knew since the "Roko's Basilisk" debacle that they're prone to magical thinking.

It's kind of fascinating how toxic the EA/LessWrong/Rationalist stuff has become. I remember reading it back in the early days (~2010 or so) and finding a lot of good ideas mixed in with some dorkiness (Harry Potter fan fiction about rationalism..). It always seemed like a sort of harmless niche with some good ideas and some wacky ones. And yet now it's basically become a cultish religion for the ultrapowerful who use it to justify doing some really awful things.

Reply on news.ycombinator.com