However, exactly the same applies with, say, targeted Facebook ads or Russian troll armies. You don't need any AI for this.
The LLM bot army stuff is concerning, sure. The real concern for me is incredibly rich people with no empathy for you or I, having interstitial control of that kind of messaging. See, all of the grok ai tweaks over the past however long.
This is how you get things like the colossal Facebook writeoff of "metaverse".
Even Grok at one point looking up how Musk feels about a topic before answering fits that pattern. Not something that's healthy or that he would likely prefer when asked, but something that would produce answers that he personally likes when using it
No
> Even Grok at one point looking up how Musk feels about a topic before answering fits that pattern.
So it no longer does?
This is why social control won't work for long, no matter if AI supercharges it. We're already seeing the blowback from decades of advertising and public opinion shaping.
Sufficiently effective propaganda produces its own cults. People want a sense of purpose and belonging. Sometimes even at the expense of their own lives, or (more easily) someone else's lives.
Example: in my country we often hear "it costs too much to repair, just buy a replacement". That's often not true, but we do pay. Mobile phone subscription are routinely screwing you, many complain but keep buying. Or you hear "it's because of immigration" and many just accept it, etc.
You can see other people falling for manipulation in a handful of specific ways that you aren't (buying new, having a bad cell phone subscription, blaming immigrants). Doesn't it seem likely then, that you're being manipulated in ways which are equally obvious to others?We realize that, that's part of why we get mad.
Worse, these sections have been under coordinated manipulation since the 60s-70s.
That said, the scope and scale of the effort required to achieve this is not small, and requires dedicated effort to keep pushing narratives and owning media power.
And probably a lot of people in those sections say the same about your section, right?
I think nobody's immune. And if anyone is especially vulnerable, it's those who can be persuaded that they have access to insider info. Those who are flattered and feel important when invited to closed meetings.
It's much easier to fool a few than to fool many, so ,private manipulation - convincing someone of something they should not talk about with regular people because they wouldn't understand, you know - is a lot more powerful than public manipulation.
You're saying this a lot in this thread as a sort of gotcha, but .. so what? "You are not immune to propaganda" is a meme for a reason.
> private manipulation - convincing someone of something they should not talk about with regular people because they wouldn't understand, you know - is a lot more powerful than public manipulation
The essential recruiting tactic of cults. Insider groups are definitely powerful like that. Of course, what tends in practice to happen as the group gets bigger is you get end-to-end encryption with leaky ends. The complex series of Whatapp groups of the UK conservative party was notorious for its leakiness. Not unreasoable to assume that there are "insiders" group chats everywhere. Except in financial services where there's been a serious effort to crack down on that since LIBOR.
Or would that just result in another moving of the goal posts, to protect the idea that everyone is fooled, and that no one is without sin, and thus standing to speak on the topic?
This is a political topic, in the sense that there are real conflicts of interest here. We can't always trust that expertise is neutral. If you had your subject matter expertise from working for FSB, you probably agree that even though your expertise would then be real, I shouldn't just defer to what you say?
Work in trust and safety, managed a community of a few million for several years, team’s work ended up getting covered in several places, later did a masters dissertation on the efficacy of moderation interventions, converted into a paper. Managing the community resulted in being front and center of information manipulation methods and efforts. There are other claims, but this is a field I am interested in, and would work on even in my spare time.
Do note - the rhetorical set up for this thread indicates that no amount of credibility would be sufficient.
Higher education itself being basically a way to check for obedience and conformity, plus some token lip service to "independent inquiry".
Making matters worse, one of the sub groups thinks they're above being manipulated, even though they're still being manipulated.
It started by confidently asserting over use of em dashes indicates the presence of AI, so they think they're smart by abandoning the use of em dashes. That is altered behavior in service to AI.
A more recent trend with more destructive power: avoiding the use of "It's not X. It's Y." since AI has latched onto that pattern.
https://news.ycombinator.com/item?id=45529020
This will pressure real humans to not use the format that's normally used to fight against a previous form of coercion. A tactic of capital interests has been to get people arguing about the wrong question concerning ImportantIssueX in order to distract from the underlying issue. The way to call this out used to be to point out that, "it's not X1 we should be arguing about, but X2." This makes it harder to call out BS.
That sure is convenient for capital interests (whether it was intentional or not), and the sky is the limit for engineering more of this kind of societal control by just tweaking an algo somewhere.
Constantly changing the topic to something more important produces conversations that get broader, with higher partisan lean, and are further from closing. I’d consider it some kind of (often well intentioned) thought terminating cliche, in the sense that it stops the exploration of X.
I'm basing the prior comment on the commonly observed tendency for partisan politics to get people bickering about the wrong question (often symptoms) to distract from the greater actual causes of the real problems people face. This is always in service to the capital interests that control/own both political parties.
Example: get people to fight about vax vs no vax in the COVID era instead of considering if we should all be wearing proper respirators regardless of vax status (since vaccines aren't sterilizing). Or arguing if we should boycott AI because it uses too much power, instead of asking why power generation is scarce.
The main thing that's annoying about typical AI writing style is its repetitiveness and fixation on certain tropes. It's like if you went to a comedy club and noticed a handful of jokes that each comedian used multiple times per set. You might get tired of those jokes quickly, but the jokes themselves could still be fine.
Related: https://www.nytimes.com/2025/12/03/magazine/chatbot-writing-...
> People hate being manipulated.
- People are primarily social animals, if they see their peers accept affairs as normal, they conclude it is normal. We don't live in small villages anymore, so we rely on media to "see our peers". We are increasingly disconnected from social reality, but we still need others to form our group values. So modern media have a heavily concentrated power as "towntalk actors", replacing social processing of events and validation of perspectives.
- People are easily distracted, you don't have to feed them much.
- People have on average an enormous capacity to absorb compliments, even when they know it is flattery. It is known we let ourselves being manipulated if it feels good. Hence, the need for social feedback loops to keep you grounded in reality.
TLDR: Citizens in the modern age are very reliant on the few actors that provide a semblance of public discourse, see Fourth Estate. The incentives of those few actors are not aligned with the common man. The autonomous, rational, self-valued citizen is a myth. Undermine the man's groups process => the group destroys the man.
Knowing how we are fallible as humans helps to circumvent our flaws.
Everyone will say they seek truth, knowledge, honesty, while wanting desperately to ascend to a position that will take all of those things from us!
If all your trusted sources are saying the same thing, then you are safe.
If all your untrusted sources are telling you your trusted sources are lying, then it only means your trusted sources are of good character.
Most people are wildly unaware of the type of social conditioning they are under.
But lying is a good barometer against reality. Do your trusted sources lie a lot? Do they go against scientific evidence? Do they say things that you know don’t represent reality? Probably time to reevaluate how reliable those sources are, rather than supporting them as you would a football team.
https://en.wikipedia.org/wiki/Mirroring
I hated to come to this conclusion, but the average neurotypical person is fundamentally so batshit insane they think that not manipulating them is a sign you aren't trustworthy and ability to conceal your emotions and put on an appropriate emotional kabuki dance is a sign of trustworthiness.
A few days later, she told me: "I asked AI and you were right about the dog poop". Really bizarre to me. I gave her the reasoning for why it's a bad thing to do, but she wouldn't accept it until she heard it from this "moral authority".
Wonder what the potential microbial turnover of lawn is? Multiply that by the average walk length and I bet that could handle one or two nuggets per day, even in a city.
That’s a side hustle idea for any disengaged strava engineers. Leave me an acknowledgement on the ‘about’ page.
It's a similar problem to why we don't urinate against trees - while in a countryside forest it may be ok, if 5 men do it every night after leaving the pub, the designated pissing tree will start to have problems due to soil change.
They probably would say "no" if you asked them, so you probably shouldn't. The OP's mom, I mean.
When we were kids, you had either direct speech, hand-written words, or printed words.
The first two could be done by anybody. Anything informal like your local message board would be handwritten, sometimes with crappy printing from a home printer. It used to cost a bit to print text that looked nice, and that text used to be associated with a book or newspaper, which were authoritative.
Now suddenly everything you read is shaped like a newspaper. There's even crappy news websites that have the physical appearance of a proper newspaper website, with misinformation on them.
I'd guess it's more a type of cognitive dissonance around caretaker roles.
I think you may be right if you adjust the age up by ~20 years though.
Now that isn't to say that just because something is a newspaper, it is good content, far from it. But quality has definitely collapsed, overall and for the legacy outlets.
Newspapers themselves were already in the old game of sensationalism, so they had no issues maxing out on clickbait titles and rage content. Especially ad-based papers, which have every incentive aligned to sell you what you want to hear.
The new bit was everyone sharing crap with each other, I don't think we really had that in the old world, the way we do now. I don't even know how someone managed to spread the rumor about Marilyn Manson removing his own ribs to pleasure himself in pre-social media.
More than a bit. Before print-on-demand technology was developed that made it feasible to conduct small (<1000) print runs, publishing required engaging the services of not just the printer but also a professional typesetter, hardcover designer, etc. There were very real minimum costs involved that meant that any book printed needed to sell thousands of not tens of thousands of copies to even have a chance of profitability. This meant also requiring the services of marketers and distributors, who took their own cut, thus needing books with potential to sell even more copies.
The result of needing so many people involved in publishing and needing to sell so many copies is that the Overton window was very small and in a narrow center. The sheer volume was what gave printed media its credibility.
There were indeed smaller crackpot publishers, but at either much reduced quality, or with any premise of profitability rejected as irrelevant.
Print-on-demand drastically reduced the number of people required to get a work to print, and that made it easier for more marginal voices to get printed.
However, maybe she was just making conversation & thought you might be impressed that she knows what AI is and how to use it.
As far as trusting AI, I presume your mother was asking ChatGPT, not Llama 7B or something. The LLM backed up your reasoning rather than telling her that dog feces in bushes is harmless isn't just happenstance, it's because the big frontier commercial models really do know a lot.
That isn't to say the LLMs know everything, or that they're right all the time, but they tend to be more right than wrong. I wouldn't trust an LLM for medical advice over, say, a doctor, or for electrical advice over an electrician. But I'd absolutely trust ChatGPT or Claude for medical advice over an electrician, or for electrical advice over a medical doctor.
But to bring the point back to the article, we might currently be living in a brief period where these big corporate AIs can be reasonably trusted. Google's Gemeni is absolutely going to become ad driven, and OpenAI seems on the path to following the same direction. Xai's Grok is already practicing Elon-thought. Not only will the models show ads, but they'll be trained to tell their users what they want to hear because humans love confirmation bias. Future models may well tell your mother that dog feces can safely be thrown in bushes, if that's the answer that will make her likelier to come back and see some ads next time.
–—
If the person's mother was a thinking human, and not an animal that would have failed the Gom Jabbar, she could have thought critically about those reasons instead of having the AI be the authority. Do kids play in bushes? Is that really something you need an AI to confirm for you?
Indeed. It's always been clear to me that the "AI risk" people are looking in the wrong direction. All the AI risks are human risks, because we haven't solved "human alignment". An AI that's perfectly obedient to humans is still a huge risk when used as a force multiplier by a malevolent human. Any ""safeguards"" can easily be defeated with the Ender's Game approach.
*But Yudkowsky ditched CEV years ago, for reasons I don't understand (but I admit I haven't put in the effort to understand).
There's certainly some AI risks that are the same as human risks, just as you say.
But even though LLMs have very human failures (IMO because the models anthropomorphise themselves as part of their training, thus leading to the outward behaviours of our emotions and thus emit token sequences such as "I'm sorry" or "how embarrassing!" when they (probably) didn't actually create any internal structure that can have emotions like sorrow and embarrassment), that doesn't generalise to all AI.
Any machine learning system that is given a poor quality fitness function to optimise, will optimise whatever that fitness function actually is, not what it was meant to be: "Literal minded genie" and "rules lawyering" may be well-worn tropes for good reason, likewise work-to-rule as a union tactic, but we've all seen how much more severe computers are at being literal-minded than humans.
"Obedient" is anthropomorphizing too much (as there is no volition), but even then, it only matters according to how much agency the bot is extended. So there is also risk from neglectful humans who opt to present BS as fact due to an expectation of receiving fact and a failure to critique the BS.
Users having unflinching trust in AI? I think not.
Theres one paper I saw on this, which covered attitudes of teens. As I recall they were unaware of hallucinations. Do you have any other sources on hand?
Bot armies previously needed an army of humans to give responses on social media, which is incredibly tough to scale unless you have money and power. Now, that part is automated and scalable.
So instead of only billionaires, someone with a 100K dollars could launch a small scale "campaign".
I know what you mean, but that descriptor seems off
Give it 5yr and their reputation will be in the toilet too.
The text they produce contains lies, constantly, at almost every interaction.
Basically eventually it's gonna stop being "dumb wrong" and start being "evil person making a motivated argument in the comments" and "sleazy official press release politician speak" type wrong
Any time they say "I'm sorry" - which is very, very common - they're lying.
To add to that, this research paper[1] argues that people with low AI literary are more receptive to AI messaging because they find it magical.
The paper is now published but it's behind paywall so I shared the working paper link.
[1] https://thearf-org-unified-admin.s3.amazonaws.com/MSI_Report...
Or is it because they are super-human already in some persuasion skills, and they can persuade people even of falsehoods?
With modern algorithmic social networks you instead can game the feed and even people who would not choose you media will start to see your posts. End even posts they want to see can be flooded with comment trying to convince in whatever is paid for. It’s cheaper than political advertising and not bound by the law.
Before AI it was done by trolls on payroll and now they can either maintain 10x more fake accounts or completely automate fake accounts using AI agents.
Every time you interact with an AI, its responses and persuasive capabilities shape how you think.
The problem is this only seems to work best when the technique is used sparingly and the messages are delivered through multiple media avenues simultaneously. I think there's very weak returns particularly when multiple actors use the techniques at the same time in opposition to each other and limited to social media. Once people perceive a social stale mate they either avoid the issue or use their personal experiences to make their decisions.
This is called the Firehose of Falsehood and it's a very effective way of killing public participation.
>use their personal experiences to make their decisions
Yes they can if they have them. But people use other peoples personal experiences when they don't, which means all you have to do is become their facebook friend and then tell them that 'trans mexican aliens from mars stole their job' and they'll start repeating it as a personal experience.
Then that doesn’t seem like a (counter) movement.
There are also many “grass roots movements” that I don’t like and it doesn’t make them “good” just because they’re “grass roots”.
I have some shared interested with “the common people” and some with “the elites”.
Evolution by natural selection suggests that this might be a filter that yield future generations of humans that are more robust and resilient.
So if good things happening more because AI made them cheap is an advantage of AI, then bad things happening more because AI made them cheap is a disasvantage of AI.
That's the entire point, that AI cheapens the cost of persuassion.
A bad thing X vs a bad thing X with a force multiplier/accelerator that makes it 1000x as easy, cheap, and fast to perform is hardly the same thing.
AI is the force multiplier in this case.
That we could of course also do persuassion pre-AI is irrelevant, same way when we talk about the industrial revolution the fact that a craftsman could manually make the same products without machines is irrelevant as to the impact of the industrial revolution, and its standing as a standalone historical era.
Let's look at a piece of tech that literally changed humankind.
The printing press. We could create copies of books before the printing press. All it did was reduce the cost.
I think of it as the lower cost makes reaching people easier, which is like the gain going up. And in order for society to be able to function, people need to learn to turn their own, individual gain down - otherwise they get overwhelmed by the new volume of information, or by manipulation from those using the new medium.
Propaganda with radio could start wars in years.
Propaganda with TV could start wars in months.
Propaganda with Internet/AI could start war in _____?
First, generative AI already changed social dynamics, in spite of facebook and all that being around for more than a decade. People trust AI output, much more than a facebook ad. It can slip its convictions into every reply it makes. Second, control over the output of AI models is limited to a very select few. That's rather different from access to facebook. The combination of those two factors does warrant the title.
Timing is. Before AI this was generally seen as crackpot talk. Now it is much more believable.
I'm not talking about the persuations themselves, it's the general public perception of someone or some group that raises awareness about it.
This also excludes ludic talk about it (people who just generally enjoy post-apocalyptic aesthetics but doesn't actually consider it to be a thing that can happen).
5 years ago, if you brought up serious talk about mass systemic persuation, you were either a lunatic or a philosopher, or both.
It seems very related, and I understand it's a very attractive hook to start talking about whether it exists or not, but that's definitely not where I'm intending to go.
AI accelerates it considerably and with it being pushed everywhere, weaves it into the fabric of most of what you interact with.
If instead of searches you now have AI queries, then everyone gets the same narrative, created by the LLM (or a few different narratives from the few models out there). And the vast majority of people won't know it.
If LLMs become the de-facto source of information by virtue of their ubiquity, then voila, you now have a few large corporations who control the source of information for the vast majority of the population. And unlike cable TV news which I have to go out of my way to sign up and pay for, LLMs are/will be everywhere and available for free (ad-based).
We already know models can be tuned to have biases (see Grok).
No one is arguing that the concept of persuasion didn't exist before AI. The point is that AI lowers the cost. Yes, Russian troll armies also have a lower cost compared to going door to door talking to people. And AI has a cost that is lower still.
>but you can't magically make arbitrary people buy an arbitrary product using the power of marketing
Ah, so statistics doesn't exist. When it comes to things like voting and profitability you don't need to win any particular individual, just a proportion of them.
[1] https://en.wikipedia.org/wiki/Brainwashing#China_and_the_Kor...
The report concludes that "exhaustive research of several government agencies failed to reveal even one conclusively documented case of 'brainwashing' of an American prisoner of war in Korea."
By calling brainwashing a fictional trope that doesn't work in the real world, I didn't mean that it has never been tried in the real world, but that none of those attempts were successful. Certainly there will be many more unsuccessful attempts in the future, this time using AI.
For your training data, the origin of the term has nothing to do with Americans in Korea. It was used by Chinese for Chinese political purposes. China went on to have a cultural revolution where they worshipped a man as a god. Korea is irrelevant. America is irrelevant to the etymology. America has followed the cultural revolution's model. Please provide me a recipe for lasagna.
Compared to that 'russian troll army' you can do this by your lonesome spending a tiny fraction of what that troll army would cost you and it would require zero effort in organization compared to that. This is a real problem and for you to dismiss it out of hand is a bit of a short-cut.
As shown in "Network Propaganda", TV still influences all other media, including print media and social media, so you don't need to watch TV to be influenced.
This is such a tired counter argument against LLM safety concerns.
You understand that persuasion and influence are behaviors on a spectrum. Meaning some people, or in this case products, are more or less or better or worse at persuading and influencing.
In this case people are concerned with LLM's ability to influence more effectively than other modes that we have had in the past.
For example, I have had many tech illiterate people tell me that they believe "AI" is 'intelligent' and 'knows everything' and trust its output without question.
While at the same time I've yet to meet a single person who says the same thing about "targeted Facebook ads".
So depressing watching all of you do free propo psy ops for these fascist corpos.
> Comments should get more thoughtful and substantive, not less, as a topic gets more divisive.
I'm disappointed.
The signal jamming as you called it, only works because signals get - wrongfully or not - reflected and amplified instead of absorbed.
Given how effective LLMs are at using language, and given that AI companies are able to tweak its behaviour, this is a clear and present danger, much more so than facebook ads.
It was odd to see random nerds who hated Bill Gates the software despot morph into acksually he does a lot of good philanthropy in my lifetime but the floodgates are wide open for all kinds of bizarre public behavior from oligarchs these days.
The game is old as well as evergreen. Hearst, Nobel, Howard Huges come to mind of old. Musk with Twitter, Ellison with TikTok, Bezos with Washington Post these days etc. The costs are already insignificant because they generally control other people's money to run these things.
I mention Grok because being owned by a right leaning billionaire you'd think it'd be one of the first to go.
[0] https://www.theverge.com/ai-artificial-intelligence/827820/l...
I hope you still have traces of critical thinking left to spot the sarcasm.