Disrupting deceptive uses of AI by covert influence operations
openai.com
openai.com
I can't help but find this extremely funny. you mean to tell me that massive operations trying to use your service didn't actually gain anything from using your service?
OpenAI wasting time and resources shutting them down and the bad actors wasting time and resources using chatGPT no benefit.
It just reads like a massive waste of time for everyone involved.
I’d guess that “covert ops” using ChatGPT are amateurs, by definition (because it’s clearly not covert). Surely, the serious operators would use offline local LLMs.
Anytime they can get good publicity like this is good for their business.
They started their counter operations a while back and regularly draft extensive reports [1].
Reading through those reports it becomes obvious how much effort this requires.
I’m not sure how much effort OpenAI can put into this at all.
[1] https://transparency.meta.com/de-de/metasecurity/threat-repo...
In terms of developer goodwill, they may be the real winners of this whole gold rush.
They really do a good job on many fronts right now.
Additionally the whole TikTok debate will help them a lot too.
Next to TikTok they’re better moralistically. Next to Sam Altman, I mean OpenAI, they’re better moralistically. Next to other companies doing layoffs they’re better moralistically (biggest exit packages?)
Not unexpected but just what you'd expect.
But I haven't used Facebook for about a decade, so ¯\_(ツ)_/¯
Google and Microsoft don't just throw up their hands at the threat of nation-state actors; doing so would be hugely irresponsible.
On the scale of exploitability, I think some of the LLMs we have now are pretty high up there, beyond mobile phones - harder to secure and with bigger consequences when misused. The intelligence agencies of pariah states like Russia, Iran and Israel will absolutely use this stuff to sway public opinion and get large numbers of people killed. I think employees of OpenAI need to be honest with themselves about this, rather than deluding themselves and saying "oh we'll just do security/trust/safety" better than the a nation state.
I think there are a number of ways out of this which are consistent. You could convince yourself that AI's benefit to humanity as a whole over time will outweigh these issues now [citation needed], or you could take a nihilistic approach and say that you just wanna print stacks of money and go relax on the beach. Those are at least arguments you can make. But I think it's intellectually dishonest to say "Bad stuff won't happen with our tech because our team will out compete the mossad".
The clearest example of a foreign influence operation I can remember was during the annexation of Crimea, when there was a flood of pro-Russian comments on Reddit. Thanks to the voting system and the centralized discussion threads, it's a platform that's relatively easy to manipulate.
It's much harder to gain visibility on social media platforms where you choose which accounts to follow.
For all sides, apparently.
hmm. is this legal cya because a name was named? naming and shaming is a curious choice. taking care to specify what was disrupted is… well, it draws the eye.
all others were called operations or groups. unlikely the groups were disrupted even if their operations were. no difference in pattern. just in categorization. company vs cell.
what is the extra care for, exactly?
https://www.reuters.com/technology/meta-identifies-networks-...
> In Ukraine this is a very known problem, to the point when there are jokes about those bots , cause they often post texts translated to Ukrainian from Google translated russian text. For example a year ago or so there was a meme "I can't stand this hellish flour" , which was created cause Google translator translated "hellish torment" from russian to "hellish flour" in Ukrainian , cause 'flour" in Ukrainian written exactly the same as "torment" and 'flour' in russian, which isn't the case in Ukrainian language (well , kinda the case , but Google translator translated word that has two meanings in the word that means only flour)
https://www.reddit.com/r/europe/comments/1d4j1oo/openai_has_...
> “These operations may be using new technology, but they're still struggling with the old problem of how to get people to fall for it,” said Ben Nimmo
Even if true it’s just a matter of time before people will have absolutely no idea what truth really is.
Does it? Troll factories with authentic humans exist for at least a decade. They cost dirt cheap compared to a regime’s budget. There’s no problem with creating a network of bs, the problem is how to make it relevant.
people will have absolutely no idea what truth really is
We already have no idea. The only things that are true are fundamental human motivations. You can’t miss too far starting from there. Once you get that point of view, everything becomes so clear, despite missing details.
"Mr. President, Mr. President, come quick! We've reports that Chechen Separatists have began to delegate portions of their work to ChatGPT."
"Excellent, then we have nothing to worry about."
With these confirmed bad actors, what percentage of their usage attempts were disrupted by automatic safety measures?
Of the blocked attempts, is there any evidence that the influence actor successfully circumvented those protections?
So the protections were bypassed at least some of the time. Many of these are demonstrably trivial to bypass.
Like, yeah ok it is a good thing that they are trying to curtail this abuse.
However, it kinda feels like we are just delaying the inevitable? With open source models like LLAMA, detecting and doing anything about this will become impossible in a year or 2 (in reality its already possible).
It just feels very, "look at us doing this good thing but ignore what we unleashed on the world" so please pat us on the back here.
Maybe I am just pessimistic, but I am not about to give OpenAI any praise here for this when they are the reason we are in the current situation.
It is that.
These AI firms have been very vocal about their "ends justify the means" attitude. (Which is doubly damning considering how little these firms care about any purported AI safety for their AGI dreams. Months of "AI is going to kill us all" while they couldn't even be arsed to stop using CSAM-contaminated training data, nevermind using quality training data to ensure alignment)
And "Foreign state actors generated some propaganda texts" is very small as AI-problems in the here-and-now go.
Does the availability of AI mean that (as long as you own, or at least can retcon, several socials?) spinning up entire innocuous provenance networks for fresh identities is once again a push-button solved problem?
> Does the availability of AI mean that (as long as you own, or at least can retcon, several socials?) spinning up entire innocuous provenance networks for fresh identities is once again a push-button solved problem?
AI doesn't defeat biometrics. I'd bet the problem with social media for intelligence agents is it allows facial recognition to uncover their true identity. If the search uncovers a real profile and a fake AI generated one with fake AI generated friends, it's still fishy to an adversary.
Also if you're an agent and have gone to a country under one name, you can never go back under a different name without being trivially caught.
Hard to say whether there are as many people in younger age groups who are like that.
Suspiciously absent from the list of takedowns is anything on Reddit. Given the amount of obviously-chatGPT puppets on there, I'm somewhat 'surprised'.
Also a good reminder that if you're using ChatGPT for anything, they can and will examine the traffic and contents of your messages / replies.
I wonder if in the future they may offer a "service" to content providers. Say you're a reddit / forum / etc. You get supplied an API that you can run queries through that match any content that may have been generated through their service. You can then tag those accounts as bots. Of course, such an API will have costs associated with it...
I'm not a lawyer, but this sounds like racketeering to me. Basically extorting "protection" money from sites.
The funny part is, if it is all a disinfo campaign, then the very resources behind the campaign will downvote and attack this post and try to silence/discredit such voices. Or, maybe I'm just completely wrong and I'll get downvoted for having dumb beliefs. There's actually very little way to tell the difference between the two scenarios. In addition, does the act of me pointing out that malicious actors would want to downvote this now make it less likely for them to actually do so, as it would show their hand? You can recurse down these rabbit holes forever. Fun stuff.
And learning to stop a Jim Jones, even knowing that you need to, may help you stop you from being named by the UN as complicit in a genocide, the way the UN explicitly named Facebook as having played a "determining role" in the Rohingya genocide: https://web.archive.org/web/20220217165239/https://www.reute...
Of course, if you want to prevent even the most capable of intelligence agencies from spinning up their own local LLMs at all, then at a minimum you can't ever publish the model weights themselves — a solution which itself is the cause of around 50% of the criticisms people around here make of OpenAI.
So, the UN should take a hard long look in the mirror before it accuses anyone else of being complicit in genocide.
I agree that the optics of that event don't look great. But the fact is, the ceremony was officially conducted in the name of "All passengers of the downed helicopter and their families". Because of that phrasing, the UNSC members who went along with it probably felt they had to do so as a matter of protocol.
Again - not something I would go along with, if I were in the room. But it also wasn't the personalized tribute to Raisi and no one else that you're making it out to be.
Meanwhile, Israel's primary airport is named after the architect of the 1948 expulsions, the Qibya massacre and countless other atrocities.
https://en.wikipedia.org/wiki/Ben_Gurion_Airport
The point isn't whataboutism, but rather: "Credibility starts at home".
But more generally, you just haven't rebutted the parent comment. It would be weird for anyone to suggest Israel has human rights moral authority at this point (or at least any more so than any other major world power). But all the parent commenter said is that the UN doesn't have any either, which seems like a pretty easy argument to defend.
(As always, just nerding out).
Do you need to defend the UN to make my (even earlier) comment about the UN criticising Facebook?
I think my point still works even if the UN can only catch really egregious cases, those performed by those not of strategic political importance to one of the permanent members of the SC; and likewise also remains valid in the face of the organisation having a functional requirement to honour dead leaders who can at least pretend to have not been secret masterminds to the degree that Iran isn't officially at war with Israel despite the open acrimony.
If Israel disagrees, then it is welcome to withdraw its membership at any time.
At any rate, scoffing at UN decisions is hardly an Israeli idiosyncrasy. It would be weirder if a country of Israel's scale showed it reverence.
After a five week trial the court found against him in every respect, concluding his evidence was forged, that his actions constituted a fraud upon the court, etc.
Relevant to this thread is that partly through the process he began to use ChatGPT. He appeared to use it in four ways: To create forgeries that he didn't have the technical skills to create (at least not on a short timeframe) with e.g. ChatGPT walking him through the technical steps, to concoct excuses and explanations, and to flood out his opponents with hundreds of pages of ChatGPT authored testimony that expended our resources just reading the material.
He also used it to suggest legal strategy, which didn't cause us any problems and probably worked in our favor as ChatGPT's suggestions on that front appear to have been uniformly idiotic.
Though he concealed his earlier usage, claiming to have lost access to his prior account, all in all his ChatGPT activity from around September 2023 to January 2024 amounted to some 22 million lines of text.
We were extremely fortunate that his usage of ChatGPT began after his litigation began and that he made numerous errors that made his usage unambiguous. Had he been using it all along and less obviously our cost of defending the claim may have been increased dramatically-- even at his current usage had we been funding our defense entirely out of our own pockets his abuse of chatgpt may well have been the straw that bankrupted us before we could secure our victory.
I think the us of LLMs as a tool in crime is interesting. For many negative uses of LLMs like spamming you can counter that people could just hire humans for the task so the LLM is more a question of cost or degree. But when committing a crime anyone you hire is a party that could extort you or whistleblow. We suspect our con made as many errors as he did with his forgeries in part due to his limited ability to enlist third parties to vet them.
The potential for productive, honest, usage of the technology in this case appeared pretty limited relative to the potential for abuse. Some of the abuse could be addressed through improvements in standard practices like filing length limits, but its less clear to me how abuse like using it as a confidential forgery aid can be addressed.
I certainly don't think it can be solved with safety approaches, since the interaction never needs to tell the LLM the true purpose. The same doesn't work for a human forgery aid because their access to outside information and potential for being questioned would make it much harder to avoid the risk that they discover that they're being used as a tool for fraud.
• A previously unreported operation from Russia, which we dubbed "Bad Grammar", operating mainly on Telegram and targeting Ukraine, Moldova, the Baltic States and the United States;
• A persistent Russian threat actor posting content about Ukraine across the internet, known as "Doppelganger";
• A persistent Chinese threat actor posting content across the internet to praise China and criticize its critics, known as "Spamouflage";
• A persistent Iranian threat actor posting web content that supported Iran and criticized Israel and the US, known as the International Union of Virtual Media (IUVM);
• A commercial company in Israel called STOIC, generating content about the Gaza conflict, and to a lesser extent the Histadrut trade unions organization in Israel and the Indian elections. We have nicknamed this operation "Zero Zeno" for the founder of the stoic school of philosophy, and to reflect the low levels of engagement that its various campaigns attracted.
Likely in-house for many of the other actors above; this may just be low-hanging fruit, or operations designed to hit specific demographics in specific countries.
Note also, that in the west corporate propaganda is permissible, whilst state propaganda is orwellian.
It's no mystery then that much state propaganda appears under a corporate guise. If OpenAI were against corporate propaganda, much of its business model would vanish overnight.
Good! Great that they don't give criminals cool names ...
> Activity by a commercial company in Israel called STOIC, because technically we disrupted the activity, not the company. We nicknamed this operation Zero Zeno, for the founder of the stoic school of philosophy
Never mind.
What year is it?
also slashdot, hackernews, and various reddits. it call all be automated, and failing that, plenty of folks willing to do the "mechanical turk" approach and shitpost for pay.