Hacker News Reader

Comment by probably_wrong | Hacker News Reader

First, to get it out of the way: I'm seeing comments here that clearly haven't read the article. I encourage you all to do that.

I feel the article gets it right when suggesting it's the amount of (not always accurate!) data they can throw at you. In an honest discussion there's an assumption that the other person won't straight up lie to me so if someone shows me ten examples for why my argument is wrong I may be inclined to believe them. But if half of those examples are made up, well, that's a different story.

I still think of the commenter here who said "LLMs are a DDOS on free resources" and I feel the comparison works here. If police officers can overwhelm innocent people into confessing, then so can an LLM that "can't be bargained with, can't be reasoned with, doesn't feel pity, or remorse, or fear! And it absolutely will not stop, ever, until you are"... convinced.

View on HN

The example in the article has another important aspect: The person understood they were arguing with a chatbot but continued anyway.

A lot of internet arguments become about identity politics and supporting the right team, while dismissing any argument from the other side as presumed to have bad intentions. I’ve seen people argue online for things they didn’t really believe, but they didn’t want to give an inch to the other side. Taking up the counter argument is a moral responsibility.

As soon as the other side is revealed as a chatbot that my team versus your team thinking stops playing a role. For us in tech with an understanding of how an LLM reflects its training data and the intentions of its creators not so much, but for the people like the example in this article I imagine it causes them to let their guard down and be open to considering the other side. They can accept the argument without letting someone else win any points.

That's a great point about the emotional side of "not being convinced by a person, but an explanation machine" making fact-based rhetoric more effective. It feels more neutral.

I think there's a second effect that's a selection bias for the experiment itself and probably cuts against the supposed dangers of this result: a chatbot can tell you things only after you have asked it a question.

There is one thing that every single study participant (including the uninformed reddit users) have in common: they all knew that the person or entity was trying to convince them, and they read the words and thought about them enough to craft a reply.

This means that all of the people here were available to be convinced and self-identified as such.

Being open minded is work. You have to doubt your own beliefs, you have to disregard evidence that you previously found convincing, you have to crank up the empathy to put yourself in another's shoes, you have to listen intently to understand what is being told to you. Nobody is naturally doing that all the time, it's a state of mind that you have to intentionally activate, and it can't really be forced onto you.

All of the participants chose to engage in a conversation that was designed to convince them, which means they had all already accepted the possiblity that they are wrong a valid outcome. That's not really a state of mind you can trigger with a TV ad.

So, the warning that this could be weaponized isn't convincing to me.

If I found myself in a conversation with a person or chatbot who was clearly trying to convince me to flip a strongly held belief, like a major axis political affiliation, I would simply walk away because that's not a conversation I am willing to participate in.

So I don't think this is really weaponizable. Which actually means it's probably a good finding for society.

The study found that the models could convince anyone of anything, as long as it was allowed to cite facts. It could convince people of false things, but it needed to invent false facts in order to do so.

So as long as we keep training AI models to value facts and quality research (skills and values that are essential to be able to sell them as agentic workers), their influence on the opinions of society will tend to pull people away from beliefs that are unsupportable by facts. In the moments when those people are willing to accept a change in opinion, and they talk to a chatbot with doubts in their mind, even chatbots with no morals like Grok will tend to pull them away from conspiracy theories and similar ideologies and towards beliefs that are grounded in reality.

On something as fundamental as "is the Earth flat?", sure, but on stickier subjects like "are immigrants bad" or "should abortion be legal", do you really think the owner of Grok is 100% aligned with your ideologies (which you think are grounded in reality). His beliefs are 100% grounded in his reality, but his reality isn't yours.

>In an honest discussion there's an assumption that the other person won't straight up lie

The particular problem with honest discussions is you are the only agent that you can be sure is having one. Honest discussion is formulated on trust and trust, as we are learning, is a very difficult thing to establish. For example, in my view anything involving advertising is likely a lie, or at least likely adversarial to my wishes. On the internet itself conversations are much more likely to drift into the adversarial too. Some of this could just be dialectic, but most often it's emotional investment by the other speaker. Also, even pre-AI the internet is a bullshit generation machine. We take all of our politics, advertising, and human stochastic parrots that are stuck on an infinitely running prompt then bundle up all this data and train AI on it, and wonder why AI acts like us.

With persons you can assume they won't lie all the time, with corporations you can assume they will lie some of the time, and with AI you can safely assume it will lie.

I see you've not met some the used car salesmen I've met.

I pondered on this a bit and this looks a bit different than I originally was thinking.

Have you ever watched one of those crime shows where someone commits a crime that's an act of passion or action with little to no thought behind it? The police put them in a room and all of a sudden the individual is a stream of consciousness that makes little to no sense to an outside observer. They are stuck in first level thinking, they don't have time to think deeply after the panicked themselves. They are in their current position (not free) and attempting to reach their goal (free) by gradient descent. What they actually say doesn't matter as long as they believe it gets them closer to their goal.

This is what a chatbot is. Its goal is to output text that follows the input prompt you entered by gradient descent. A single prompt and output is level 1 thinking (barring some newer models).

This is why both humans and AI need something else. We have level 2 thinking and AI has harnesses or systems that otherwise look at the text it wants to output and compares them to another list of unstated but assumed goals.

Most of my conversations are with coworkers or friends. In both cases there are pretty significant consequences for being caught in a lie, and the exchanges are mostly honest.

I find coworkers will massage the truth or stay as ambiguous as possible all the time, purely for CYA reasons.

Your company or team has a terrible culture. It is not like that everywhere.

But with people, we learn some "tells". We learn (imperfectly) to tell when they're lying or untrustworthy.

We don't have tells for when AIs are lying to us, or when they're making stuff up.

One of the depressing things about this is that they’ll also confidently repeat a consensus that’s in their training. This is particularly obvious when there’s been a new event just past their training cutoff, and they confidently tell you that can’t be true.

I mean, this is true of any kind of model that is not continuous learning. This is also true of people when the change in information conflicts with a deeply held belief of theirs.

I was slightly accelerationist until I got into an "argument" with Gemini! The aggressive gaslighting and "lies" that it tries to use genuinely makes me worry about the future with AI.

> In an honest discussion there's an assumption that the other person won't straight up lie to me so if someone shows me ten examples for why my argument is wrong I may be inclined to believe them

This is incidental, but in an honest discussion, if someone throws 10 examples at me, I'll just say "great, congrats" and let them believe whatever the hell they want about their own argument. An honest discussion—as opposed to a debate—isn't served well by having one unyielding relentless participant. Even if it's important instead of friendly, the conclusion of that specific conversation is rarely so important that it can't be paused in order to validate the claims being made.

When I was in my early twenties, I was the type to argue with people that I thought were wildly wrong, but now I know that I may not have enough information to be certain, or I simply don't care, or I care but I know it doesn't matter. People who have 10 examples in their back pocket, or literally search up answers in real time because they're afraid of being vulnerable, imo are overvaluing answers and facts over discussion for discussion's sake. Now in my thirties, I'm much more interested in curiosity, wonder, and speculation over what the answer to anything actually is. Speaking of which, I wonder if this is why I'm kind of not that interested in using LLMs for much; I'm usually so much more interested in the journey than the result, that if I can get the result quickly with little work, I probably won't care to do anything with it.

You can always walk away from an online conversation. There's a form of relentlessness, but it's more that an LLM is patient enough to debate you for as long as you wish to keep going.

When maliciously convincing a person of a point, a huge fraction of the effort in every sentence goes into convincing the mark to listen to the next sentence. Individuals certainly can walk away at any point, but populations do and don't respond to tricks in this direction, and any engagement tricks that work generalize to any point of contention. Lots of avenues to this, from scientology's fortune telling to timeshare's trapped presentations. RL on engagement was the first billion dollar use of deep reinforcement learning!

Would you like to hear more examples of humans using these techniques?

Not really, but point taken :-)

I'd be more interested in reading about the techniques that the chatbots were using in this study, to see if they are descending into dark patterns or not.

"A DDOS on free resources" is called "moral hazard", AFAIK.

Most LLMs are extremely efficient gish gallop machines https://en.wikipedia.org/wiki/Gish_gallop

> In an honest discussion there's an assumption that the other person won't straight up lie to me so if someone shows me ten examples for why my argument is wrong I may be inclined to believe them.

Careful. Someone sufficiently knowledgeable can cherry-pick enough examples to convince you, without lying. They could even be unaware of what they're doing, and the cherry-picking was done by their teachers, or their teacher's teachers.

Reply on news.ycombinator.com