Disrupting a covert Iranian influence operation
openai.com
openai.com
What’s troubling is the constant spamming in new subreddits like r/inthenews. Once it takes hold (whatever the message), then actual humans start to echo it over and over again.
It’s scary.
As an extreme example of this, I remember during the corona pandemic, on reddit r/worldnews there was a weekly thread that was cheering on for deaths of Russian citizens to corona. The whole thread was heavily curated only to display pure hate towards Russians. And the moderators always deleted the thread after a week and started a new one. People who posted in the thread and didn't get on with the program, had their accounts simultaneously permabanned on all major subreddits. I know, because it happened to me.
If I wrote this on Reddit, I would be called a dirty Russian, a spy, a bot, a generally evil person, a traitor and I have also been called various anti-Chinese racial slurs, some quite graphic and disgusting. There's many people out there seeking validation and acceptance, and they find it in such hate groups. I believe historians far in the future will be putting Reddit in the same category as the Hitler youth organization.
And HN is very much like Reddit, only with slightly smarter people. The moderators in here also hold very extreme views on various topics and manipulate the discourse to give false appearances of public opinion on those topics.
There was also a marked increase in moderation at the beginning of the Ukraine war in related subreddits
Yeah. Arguably Reddit has embraced inorganic activity since beginning. the founders used it to grow the site and pretend they had more active users.
GMOs are dangerous for reasons beyond immediate health implications. Israel has been determined in the world's highest court to be plausibly committing genocide. The tech industry is complicit in war crimes right now. Media consolidation is a massive and immediate threat, as are climate change and AIPAC.
... Or is it like what the park ranger said about the overlap between people and bears?
https://www.reddit.com/r/interestingasfuck/comments/1dlg8ni/...
Which… actually that’s not true. Iran isn’t using pencil and paper to influence US elections, they’re using LLMs. And if all the big cloud providers cut them off, they’ll move to using home brew models. That is the impact and putting our head in the sand about it isn’t going to be helpful.
This is NOT to say we should not release to models! I think we should release open models, and that in addition to that we should also care about written impact assessments so everyone is on the same page about risks! I think it’s great that Jeremy Howard is releasing open foundation models, but I wish he wasn’t so flippant about the real world implications. I’ve always looked up to him for the work he has done, and found that attitude very disappointing.
It's like claiming a search engine open sourcing its ranking algorithm would help people be informed instead of making spammers able to perfectly hijack all the results.
I requested the Internet Archive grab this copy: https://web.archive.org/web/20240816210620/https://teorator....
> Some of the X accounts posed as progressives, and others as conservatives.
My impression is that these influence campaigns know that they need to produce a LOT of content that appears to reflect different opinions and perspectives if they're going to appear to be "real" - that way they can build trust with an audience before attempting to influence them in one direction or another.
In my opinion the West is already doing a good enough job at polarising itself, doesn't need much help from the enemy.
Well empirically it's actually pretty cheap and easy. What's expensive and hard are the countermeasures against it.
To manipulate politics you mainly need to generate fake social proof. To do it effectively you mainly need to target a relatively small number of counties in battleground states.
Have you ever heard of AIPAC? Rupert Murdoch? The UAE?
Chainsaw dismembering princes getting a free pass with their weapons purchases?
If you're only into corporate news, there's Russia in 2016 scapegoating for terrible decisions from the DNC.
But assuming you're asking in good faith the obvious one is Russian interference in the 2016 election, which the former head of the NSA considers
> the most successful covert influence operation in history
Another former director of the NSA said
> it stretches credulity to think the Russians didn't turn the election
The Wikipedia article is a good entry point if you're not familiar with the details [0].
After the election, the Russian government had scandal-making influence in many positions including the National Security Advisor.
The US seems to believe these tactics work because we use similar ones to influence foreign governments. For example you can read about US involvement in the Arab spring [1]. A lot of the techniques are the same, but things weren't quite as online then.
[0] https://en.wikipedia.org/wiki/Russian_interference_in_the_20...
Domestic enemies came to the same conclusion.
You're missing China & Israel: https://www.theguardian.com/technology/article/2024/may/30/o... / https://archive.is/y0V3A
And, according to OpenAI, they're interfering not just in the US: https://www.thehindu.com/elections/lok-sabha/openai-says-sta... / https://archive.is/RWn8X
Israel certainly derives benefits from a powerful United States. They may want to influence US politics, but they have a strong collaborative interest.
China is more mixed, but our economies are still intertwined. The CCP might wish to shift balance and reduce the power of the United States, but we still have mutual interests.
On the other hand, if the leaders of Russia or Iran or North Korea could push a button that immediately and completely depopulates the United States without attribution, I'm pretty sure they would push it without hesitation.
"Israel accused of planting mysterious spy devices near the White House" (2019), https://news.ycombinator.com/item?id=20954298
> I'm pretty sure they would push it without hesitation
This "let's kill 'em all before they kill us" line of reasoning is... eerily similar to post-WW1 propaganda in Germany about the Bolsheviks: https://www.theguardian.com/world/article/2024/aug/13/israel... / https://archive.is/f0jkN
> This "let's kill 'em all before they kill us" line of reasoning
You made this up, nobody proposed that. On the other hand, failing to acknowledge and address the fact that other people do in fact want to kill you is just plain stupid.
This comes to mind: https://en.wikipedia.org/wiki/Assassination_of_Mahatma_Gandh...
Ghandi's assassin killed him on his third attempt. The prior two attempts failed; the guy was arrested and "released due to Gandhi's own policy of declining to press criminal charges".
There are probably people who think this is some sort of noble act. I am not one of them. The results speak for themselves.
Well, you made broad stroke generalizations and prophesied an imminent apocalypse. The German propaganda was all about this back in the 1940s (as mentioned in the article), drumming up mass hysteria to justify the atrocities against the Soviets.
Even if you're biased and seem to have a binary view of the world (black v white, good v evil etc), may be you mean well, and I'm placing too much weight in your words.
> collaborators spy on each other
I think you read past the part where their spies sold secrets to the Chinese.
(Personally, for all his failings, I like Gandhi: His pacifying politics was net good for the stability of a vast, nascent country as diverse & poor as India, keeping it from plunging into chaos from artificially drawn up borders post 300 years of ruthless colonial repression).
Very strange perspective when talking about the only US president of our times that didn't start a war, made historical diplomatic advances with North Korea, made a deal with talibans to end war in Afghanistan. I'm probably forgetting a few but his track record is impressive.
Last i checked he was killed in Baghdad Airport which is in Baghdad, Iraq. Iraq which is a completely different country than Iran.
I guess your training dataset is of low quality mate.
He's basically an admirer of Putin at this point.
The crux or the deal was a lifting of sanctions on Iran in exchange for an enforced freeze on nuclear programs. The general philosophy was that a non-Nuclear Iran engaged in the international economy was more likely to drive towards normal and peaceful relations than an isolated nuclear power. According to IAEA accounts the nuclear program freezes were effective.
As part of the lifting of sanctions, Iran did get access to frozen Iranian overseas assets on the order of $100 billion, sure. But characterizing a diplomatic agreement to lift sanctions in exchange for cooperation on nuclear proliferation as a 'bribe' seems unnecessarily pejorative.
https://en.wikipedia.org/wiki/Joint_Comprehensive_Plan_of_Ac...
“A man convinced against his will is of the same opinion still” - Dale Carnegie
https://openai.com/index/disrupting-deceptive-uses-of-AI-by-...
OpenAI has also banned Israeli influence operations. Do you think the above isn't going far enough?
It's also strange that you call out all pro-Israel comments as a cause for concern.
How is that far enough when their head of research is a genocidal zionist?[1]
>It's also strange that you call out all pro-Israel comments as a cause for concern.
What's so strange about calling pro-Israel comments as a cause for concern when Israel has been on what jewish holocaust scholars like Amos Goldberg & Raz Segal have described as [2]"undoubtedly Genocide"?
Quite the opposite actually, the OP is spot on: how can someone like Tal Broda still work at OpenAI as head of research when he has openly spewed genocidal[1] incitement?
People in tech always talk about avoiding bias in AI and the proceed to retain such a hateful genocidal individual. There is no way in hell that if the roles were reversed and an arab person had made the statements that Tal Broda has made that he would have ever been allowed to keep his job. They would have fired him immediately.
The double standards are so ugly and glaring that they will be studied by future generations.
[1] https://x.com/StopArabHate/status/1806450091399745608
[2] https://www.jewishvoiceforlabour.org.uk/article/prof-amos-go...
Indeed, and the guy definitely seems to be a piece of work. Some of his more interesting (now deleted) pronouncements, according to that fine site, The Raven Mission:
"There is no Palestine. There never was, and never will be"
"More! No mercy! IDF don't stop!"
"MORE. Don't stop"
"The IDF didn't even start to clean southern Gaza"
"We should have never left [Gaza]. Now we need take it back, by force, and keep it forever. Nothing else works."
"Don't worry about [killing civilians]. Worry about us"
The fact that OpenAI is using kid gloves to deal with the matter -- rather than taking the only decent and sensible course of action available to them at this point, which of course to simply fire the guy -- speaks volumes of its leadership, their obvious biases and complete lack of ethical grounding in regard to these matters.
Raven Mission maintains profiles of other Twitter/X users whose employers also apparently don't being publically associated with deeply obnoxious (and sometimes openly racist) sentiment of this sort. Most are fairly low-level, but the list includes partners at celebrated VC firms such as Bessemer Venture Partners and Union Square Ventures.
https://www.ravenmission.org/people/professionals
The latter individuals, being listed as Partners (one of them a Managing Partner), can be safely assumed to be speaking for their respective firms.
I was thinking that this is probably just a little bit extreme view for supporting Israel but boy this guy place is prison [1]
https://www.npr.org/2024/05/30/g-s1-1670/openai-influence-op...
[1] https://www.reddit.com/r/todayilearned/comments/bdjh8o/til_t...
I really don't think it would take that many people to run propoganda campaigns with modern tools.
https://en.wikipedia.org/wiki/Psychological_operations_(Unit...
Let's say the OpenAI engineers are working on ChatGPT 5 and spent last month scraping Teorater and X/Twitter, where this material ended up. How does OpenAI know that the new model is not poisoned?
This isn't just OpenAI's problem of course. Anyone training on the open Internet now has this problem.
'Low-background Steel' has been a nerd trivia tidbit for a while now, will we have to consider a 'Pre' and 'Post' AI internet era for training uses?
... if the actual purchase hasn't been such a cluster$&@$ so as to thoroughly disabuse the notion there was some master plan.
Side note. This is some pretty terrible propaganda. The post about Kamala, immigrants and climate change barely makes any sense.
X just hosted trump for a live stream, who is being affected by a headline that reads ‘X censors trump tweet”.
It's easy to see which one is more successful. Every time you blame something on Russia, people come out of the woods "explaining" why Russia is actually the victim (Russia is ALWAYS the victim, is what I've learned from this people).
It's dangerous to assume all of the content you see is simply some shady operation or fake, having controversial opinions on any topic is nothing new, millions of Americans don't believe in the moon landings.
In general I don't think it's a good idea to discuss politics online, and I feel like many talk too much geopolitics but can't even tell the policies or programs of their own city mayor candidates which are way more important for their own lives and directly impacting.
If one is from, or has close contacts in or from the regions affected by geopolitics -- then this is unfortunately very much not the case.
As disappointed as I am with local politicians -- they aren't bombing my friends into the ground, or forcing them to move halfway across the planet indefinitely. Or causing these to deal with significant emotional anguish, even if they aren't forced to move, or directly under threat.
While a solid contingent on HN regularly either apologies for / minimizes their actions, or just seems to shoot from its hip based on their hunch as to what is happening, without any indication of having done much research or questioning their sources. Or even just thinking logically about the various narratives they're reading. For most of them it seems to be more about ideological abstractions than anything real and concrete, in any case.
So that, plus the simple fact that this is a global community is why geopolitics floats to the top, as it were.
Whether talking about it online is productive or helpful in any way is another matter altogether. Turns out it's generally not easy to talk about these things in person, even with people one knows rather well (as most of the time they'd prefer to talk about pretty much anything but "the situation").
Being as the "shit" in question is all too real for them.
Is there not anything else that comes to your mind right now that Iran cares about and wants to influence?
To give them the benefit of the doubt, they likely want to keep their detection methods secret to make circumvention more difficult. And it all sounds totally plausible of course. But at the same time, a degree of skepticism is warranted because Microsoft has a huge incentive to fearmonger about AI so they can lock it down and capture the market. And what better way is there than to use the usual bogeymen to do so.
[1] https://cdn-dynmedia-1.microsoft.com/is/content/microsoftcor...
Microsoft is one of the few companies that goes toe to toe with world governments every day of the year.
And I imagine balancing that next to all the engineers who demand admin access everywhere is a royal pain!
Although the best Government VS Business story I heard was during intern orientation at Boeing about French agents breaking into Boeing employee's hotel rooms during a conference in France while the employees were out to dinner, and going through laptops. One of the employees returned earlier than expected, and the men in suits shut the laptop, turned around and walked out of the room w/o saying anything!
It's also the company which was the first and longest member of PRISM, meaning very deep ties to the less savory parts of the US gov and {five,nine,fourteen} eyes. I know it's a boring advice, but I'd take this kind of declaration with a truckload of salt.
This is all part of an ongoing conversation with lobbyists about “safe AI,” and it’s ultimately done to show that OpenAI is making an effort to mitigate the risks that regulators claim it creates.
But there’s also another signal, which is what they’re not broadcasting: “ChatGPT can be used for propaganda, it works in Persian too, and we’re happy to sell to the DoD.”
I know they might lose their teens and college demographic if they do it, but if it's truly this world-changing tool that they claim than not watermarking is scraping the serial number off the guns they're selling.*
* Maybe a shitty analogy
I’d go one step further — the subtext is that only Open AI is willing to do this, and that LLMs are just too dangerous to open source/not another with regulation that only big companies have the resources to adhere to.
I'd be shocked if Twitter weren't stripping the metadata they are checking.
It is apparently C2PA per [1]
[1] https://openai.com/index/understanding-the-source-of-what-we...
worldcoin is a private company owned by sama that verifies identity by scanning the eyeball. He's been rolling out the technology in places like South Africa, where they also recruited people to train openai models by RLHF etc.
It seems it has since been confirmed a scam by some of the participants.
This chap looks to have more details on it https://twitter.com/BenjaminDEKR/status/1824676550379192324
Cmon mate.
I mean. I get that low rent actors will use low rent services to try to generate political garbage. Is there any evidence that this is actually having a measurable or meaningful impact?
Who exactly is fooled by these sites? And is it the sites that are the problem or the relative lack of sophistication in American education when it comes to political corruption?
It's hard to take seriously. On any level.
This is why I only use local models these days.
EDIT: Out of posts for today but I've been pretty happy with Gemma2. The context is short but the performance is very good and it's easy to disable refusal.
But the social impact is significant. I'm reminded of a fake story about kid kidnapping in India, that caused a mob to burn the two people that were targeted alive... They were completely innocent, the mob attacked them based on fake news. Now, that can happen enmasse.
This is coming soon after Trump decided to accuse Iran of being behind his assassination attempt (done by a white 20 year old) and Israel literally assassinated Hamas's chief negotiator while he was visiting Iran.
It seems like the powers that be are desperate for a war with Iran and will continue beating the drum to build consent.
Reminds me of the build up to the 2003 Iraq invasion (you know, because "they have WMDs")
Many nations employ "patriotic citizens" in an informal and semi-formal fashion along with trained propaganda and "infowar" experts. I know of China and Israel doing this but I'd assume it's everywhere.
Israel has one of if not the largest propaganda/bot campaigns on the entire planet, followed by either Russia or China. It's definately working for them so they won't be stopping anytime soon.
I suppose this is a great opportunity for the people whose entire income comes from the fact that the US overpays insiders to supply its own military; if Trump wins, he gets to play tough and have the papers start his term praising him to the skies for starting a war with Iran, he'll also be able to blame everything on the last administration and work closely with the people who replace Netanyahu. If Harris wins, she's made no promises and has no beliefs, and will be aided by the press in blaming Gaza on Iran.
OpenAI itself is surely running larger covert influence operations in order to affect US legislation and elections.
> Similar to the covert influence operations we reported in May, this operation does not appear to have achieved meaningful audience engagement. The majority of social media posts that we identified received few or no likes, shares, or comments. We similarly did not find indications of the web articles being shared across social media.
Sounds like Russian Facebook ads.
For sure the purpose is noble, but it is good to remind everyone that everything you type, submit or generate there is not private but could be randomly snooped by strangers!