Scammers deepfake CEO’s voice to talk underling into $243k transfer?
nakedsecurity.sophos.com
nakedsecurity.sophos.com
They want to give ambiguous direction and receive exactly what they imagined but still be delightfully surprised. They don't want to hear about the unintended consequences of getting exactly what they asked for; they want it to "just work and don't trouble me with the details".
Pedantry is unacceptable. Everything must be interpreted exactly how it was meant to be. The rules are meant for others. If a subordinate fails while breaking a rule, then they are fired. If a subordinate fails because they didn't break a rule, then they are fired.
Ultimately, as we come closer to perfect impersonation at the press of a button, these 'people in charge' will have a tough time adapting. They want absolute obedience and unquestioning loyalty. Unless their subordinate was talking to a social engineer using a deep fake.
On the other hand. The people in charge who trust their subordinates, who let them fail and encourage them to do better, who let them question their approach and decisions. These people in charge will find themselves and their organizations more resilient to these attacks. "Yeah, Jeff might be a real jerk sometimes and he's always second guessing decisions, but he doesn't transfer $12 million just because he gets a phone call where someone who sounds like me is upset that the money hasn't been transferred."
We need for bosses to understand that this is not the way to request things but to always go through authorized channels which implement better "authentication" processes. This may also dampen their willingness to make "out of band" requests.
That's my biggest pet peeve about Android. (Totally unrelated discussion, sorry) I can't tell it to send all calls that aren't from my contact list directly to voicemail. They're universally spam.
This deepfakes stuff can only help accelerate the move toward bottom-up power and process-oriented thinking.
In reality, a "directive" can be generalized to any "process request", such as a software enhancement request.
A "goal level" statement must be described, and then someone between the implementer and the manager needs to flesh out and escalate what the functionality looks like and any potential conflicts. ...then the person implementing them needs to translate that spec into an implementation, and then escalate any potential unintended consequences found in testing phase. ...and all of this in an iterative cycle.
Unfortunately, 99% of office requests do no merit enough _value_ for this process to be worthwhile. With infinite resources, we can achieve infinite perfection in every management request.
...but in the real world where business demands are transient, the expectation should be that management doesn't know what they want all the time and asks for things with incorrect parameters and conditions. ...and so the implementer (since there's no one in between) needs to bring their industry experience and communication skills to _poignantly_ explain the caveats in the request and explain what the alternatives are.
These are difficult conversations to have. They inherently contain conflict and the emotions of frustration associated with the manager perceiving that they aren't getting what they want, and the implementer being asked to break protocols and safeguards.
There is no perfect solution. This comes down to professionalism and having employees and managers that are able to offer compromise, have dispassionate discourse, and be willing to thoughtfully approach the problem in succinct and timely fashion.
...which is why hiring experienced intelligent emotionally stable people is the most productive choice for companies.
There's no reason this sort of authorization process couldn't be trivially handled with any one of a number of simple technologies. If there isn't already some sort of Personal Verification Service to do this, well, there's a niche to be had. Come up with a better name.
There are other ways for 2-factor/3-factor verification (physical or passcode based tokens, e-mail+voice, or even a video chat).
There are other ways of safety like requiring a 2-person authorisation for large transactions - many organisations and especially charities already do that.
It's in German, but Google Translate to the rescue. Key paragraphs:
'But when asked, how do you know that such a software and no voice imitator has been used, a spokeswoman replies to SPIEGEL by email: "We do not know with 100% certainty. Theoretically, it could have been a human voice imitator But we do not assume that there are some clues (but no evidence). "
'In turn, the clues she mentions have no technical relevance; they are by no means interpreted as evidence of a deepfake. The supposedly "first case" can therefore at best be termed a "possible case". Which is quite symptomatic of the debate about deepfakes and the resulting risks.'
None of them know what they're talking about. Source: my direct experience
You can't go into business selling software on the assumption you'll sell a single copy and your work is finished. Infosec is no different. There is always this paradoxical need to ensure you never become entirely redundant, regardless of the function you specialize in
I think a lot of the great "evolutionary difficulties" in our industry can be phrased this way. It's why things like Microsoft Excel are laughed at even as hundreds of thousands are trained in the art of constructing effectively bespoke spreadsheets apps in Django by the million every year. I hope for a correction some day, just as much as I hope I'm on the right side of it when it comes..
It's all bullshit, and these people are pure scum.
> because 98% of us are completely clueless
Do universities/bootcamps teach OWASP-style classes of programming vulnerabilities these days? Some developers are curious enough to learn them on their own, but many are oblivious.
I suspect there could be a startup idea here somewhere.
> absolutely not thinking about this stuff as we're trying to clear a sprint board
Does anyone have a decent tool/process for remembering all of the detailed tasks for every type of software deliverable? I find myself in a state of cognitive coma after sprint planning when I need to divide tasks into subtasks.
I had exactly one course that touched on security.
A course in web programming.
The instruction we received consisted of: "If your project is not secure, it will lose points."
I'm not sure a single person on that class had one point taken off for getting security wrong.
Why would a university care about educating people in something ephemeral, and domain-specific, like security, when it could instead be teaching them about complexity theory and Djikstra, and third normal form?
(Crypto and security in general were a hobby of mine until I realized how difficult the field really is. "Programming Satan's computer" was one paper that contributed to that.)
The real problem is that there is a vast need for professionals and the lack of them calls a lot of smoke-sellers. And for a lot of people is hard to tell wether they are legit.
Just look at the Machine Learning / Artificial Intelligence industry. Also "fraudulent" if you apply the same logic.
And yeah, most of the ML industry is in precisely the same category. A bunch of frauds.
You'd be surprised just how many places run public unpatched stuff with admin/root holes here and there. Ive seen passwords like "123456" and "password" and plenty other badness that nobody really bats an eye to.
And even simple things like "Use WPA2 and a password manager", for low barrier infosec is routinely ignored. Companies can barely even manage that.... and they have the funds.
And it's not like the bad stuff is scare quotes. Ransomware is a thing. Ive seen a hospital network up north get hit by it. City of Madison IN (1h away from me) ended up paying a large sum, cause they thought backups were pointless. Even know of a story where a state government's machine ended up being a warez server. The lead though to clean up a trojaned linux box, was to rsync from a clean one. Left most of the trojan kmods intact. I caught it down the line.
Sure, if the likes you're talking about is complaints about IBM with QScan or similar, with grandiose claims that their software will save everything - thats obvious bullshit. Security is definitely a process and procedures, ALONG WITH technical means to facilitate that. Even automated scanning of "front doors", or doing routine searches in Shodan is a magnitude better than nothing.
For the executive who messed up, by using the 'deep fake' story, they become blameless. If they admit to a voice actor, they can be questioned for incompetence.
Everyone wins, except those who care about the truth.
The whole thing reminds me of: "The majority of men prefer delusion to truth. It soothes. It is easy to grasp. Above all, it fits more snugly than the truth into a universe of false appearances—of complex and irrational phenomena,” ― Friedrich Nietzsche, The Antichrist
I don't really see why the two should be considered any different.
> Any sufficiently advanced technology is indistinguishable from magic.
The mostly tech illiterate boses and the unthinking masses can't really parse the deep fake story. For most, there's simply no basis of comparison even they wanted to expend the energy on system 2 thinking.
But talk about voice actors and everyone knows someone who does imitations. So they will base their anecdotal experiences in judging the possibility of what happened. This also mostly comes from system 1 thinking, which is pleasant to do, since it removes uncertainty from the world.
Edit: when thinking of deep fakes, it applies to me too. I don't know how advanced they CAN be. I'm sure some military has some amazing ones that aren't open to the public. It's a point that goes beyond my imagination and into a world of uncertainty.
Trust your instincts, look for second sources, and wait to follow up before you decide one way or the other.
Regardless of the advances in AI, I think text-based social engineering will still be prevalent and efficient. IIRC, the Anonymous hackers who targeted HBGary got SSH access by fooling the company's chief security officer. Sure, they did by emailing from the company owner's account (thanks to a weak admin password surfaced through other vulnerabilities). But the hackers didn't even know the account name when asking for a password reset. The CSO could've stopped things by asking to do a call, e.g. "Hey let me call you, I need to walk you through this part" or even just texting the phone for confirmation.
https://arstechnica.com/tech-policy/2011/02/anonymous-speaks...
...or will it force humanity to finally acknowledge the very basic security concept of public/private keys for signing important shit?
But in short,
Several observations...
* A major threat is end-to-end VoIP/telephony encryption. Currently, the most common way to verify one's public key and ensure that no MITM wiretapping is going on, is reading a hash (encoded to words) aloud in a phone conversation. It's used to many protocols, such as Signal, or ZRTP by Phil Zimmermann et al. There is no real security, but it's considered a shortcut with reasonable security for most people, the assumption is that voice synthesis cannot be done in real-time yet. But with DeepFake it's disastrous for cryptography. Now, this shortcut is going to be blocked soon. Full verification, like signing your VoIP key with your long-term public key, or asking security questions (e.g. OTR's SMP algorithm) is needed (perhaps not to everyone, but it's now required for a lot of people to a greater extent).
* KirinDave suggested that, in additional to signing, timestamping will also be important. If automatic synthesis of video and audio becomes widespread, one way to prove the authenticity of the material is to timestamp it as soon as it's recorded, or even timestamping it in real-time if real-time forgery is a serious threat (I hope not). This is one of few use cases that a blockchain actually makes sense if you want minimum trust over 3rd-parties.
The final thought is that it's time to rewatch Ghost in The Shell (the TV animation series, not the movie). Released in 2000s, it portrayed and predicted our world remarkably well, and it'll give you a lot of inspirations of what would the future society look like. In one episode, the protagonists realized a government conspiracy aimed to intensify a military conflict that involves nuclear weapon, and they had the following dialogue.
> "How do we stop it? Can we post the video footage online?"
> "No. Video footage is seen as completely untrustworthy today."
The more verifiable pieces of data that you can associate with a recording, the more you can trust that it came from when/who it claims to. If I send a recording that can be tied to:
- a timestamp service with my request for one stored in a blockchain,
- a tamper-evident device that signs the data with its own private key,
- my own private key
then you can have a high degree of certainty that I am the one who recorded and sent the content and that it has not been altered. I could still be tied to a chair while a voice actor impersonates me and forces me to send it. This is after all basically the modern equivalent of a proof-of-life photo with the kidnapped victim holding today’s newspaper. But it’s a lot more effort for someone to go through compared to having none of those other guarantees.
It’s a fascinating topic that will only become more relevant as deepfaking gets easier. Whoever makes the first device/system to do this, if it’s not a flawed premise, will make a pretty penny.
There is at least some hope for EU citizens, that they wouldn't have to worry about these authorization/identity issues, because the entire European Union just recently created the legislative framework required for solving this problem in the entire EU. With national trust/PKI services there isn't any need to resort to insecure ancient methods (phone calls, fax) that can be spoofed or intercepted, with increasing simplicity. It is somewhat sad how long it has taken, nearly 20 years later the EU is following Estonia's practice/example! I might be biased about how good such systems are as an Estonian, but it isn't bias speaking that the system seems to work - issues like identity theft and account takeovers generally don't exist here. The fact that we have to specifically teach people how to use mostly foreign services safely, because none of them can provide really secure authentication and identification together, says quite a lot about the differences.
For example: I create photorealistic product pictures in 3D (Blender). This saves my customers a lot of money. But the thing is: we present fake pictures to the world looking very real. The products look perfect as render but in the real world this can be a little different.
Another example is IKEA. Almost everything you see in their catalogs is 3D rendered. In those renders everything fits together perfectly. But when you build it at home you might see millimeter offsets.
Maybe those are exteme examples but truth is not black and white.
But these deepfake scams or fake news or what have you goes a step further because it intrudes into what is supposed to be authentic and genuine.
These days, DKIM is necessary for even the simplest of email servers that wants to interact with any of the major email platforms.
"Oh why is the quality bad? I was recording it secretly on my phone in my pocket that's why it is muffled and sounds off" Will be the explanation for all of these, it will come to such a head that politicians will be able to say anything they want to people, racist, sexist, genocidal, whatever, and continuously claim that it was merely deepfake, and thus not their opinion.
People will invent world events out of thin air with deepfake footage and audio. If the CIA cut off the internet access of a small island nation, and create a bunch of deepfakes of a politician stepping down, whilst covertly doing operations on them how would anyone in the west ever know it wasn't real?
Remember when people used to let their kids fuckin play outside? =\ Sometimes for 12 hours!
We're less violent as a planet/society than ever.
But back then, the "gore" of all of humanity's everyday life wasn't smashed into our faces as a routine - only occasionally, as life dished it out to us personally.
Perhaps this opens up a market for secure identity verification that is accessable to the layman...
It wasn't the voice that tipped my grandma off. Just that I do not drink. So that was close.
Anyway- that's my guess why this is difficult to reverse.. if it was easy to reverse, both sides would be very suspicious of each other during a large transfer.
I mean most people can usually pick up on the difference between a voice recording and a generated voice sample based on the recording. Have there been studies where generated voices are then subjected to telephone compression and layered with background noise to appear to be a phone call from a car?
If DeepFake voices only need 5 second clips to produce good enough versions of anyone's voice and any discrepancy in quality could be attributed as a bad telephone connection or masked with background noise, is anyone really safe?
"When we said 'quality and training purposes', we were referring to training a neural network."
So it would have went like this:
1. scammer deepfakes CEO voice
2. demands X dollars transferred elsewhere
3. Employee hangs up and calls back using pre-shared phone#
4. Gets confirmation
But in this article's case, we're talking about $.243 million , which flies counter to "don't authorize multi-million dollar unreversable transfers".
Not sure what your point was here.
I mean, people do incredibly dumb things, so it is certainly possible. I doubt it, though.
"The target of the scam was convinced that he was speaking with his boss due to a “subtle German accent” and specific “melody” to the man’s voice and wired the money as requested.
According to a representative of Euler Hermes Group SA, the firm’s insurance company, the CEO was targeted by a new kind of scam that used AI-enhanced technology to create an audio deepfake of his employer’s voice. " - https://cyberscout.com/en/blog/voice-deepfake-scams-ceo-out-...
I'm missing something here. How could they replicate the sound of his voice using only text inputs[emphasis theirs]?
[1] https://medium.com/dessa-news/real-talk-speech-synthesis-5dd...
[2] https://medium.com/dessa-news/realtalk-how-it-works-94c1afda...
"Honey, I sent $20k to that camgirl because I thought she was you! Deepfakes!"
So I'm curious how, with regard to deepfake technology, video seems to be well ahead of audio. Is audio deep fake technology simply less interesting to people? Are listeners far more sensitive to voice not being perfect? Is the human input just a lot less helpful in modeling the output with voice vs video (where the initial human input is only slightly more helpful than just synthesizing from text-to-speech)?
It's ok to get spam and scam for emails if you're not paying anything but there's no reason to get the same for phones if you're paying top $ for it.
Until the scammers are able to take control of your phone number which I hope never happens I think the above is a good solution to fight this junk.
Only requires 5 seconds of voice audio to synthesize believable speech.
If there was any sort of proof that this happened, like a recording, then sure. But there's no facts in the article other than "analysts suspect..." and an attention-grabbing headline.
Maybe I'm just jaded but after everything that's happened over the last few years, but I take everything the media publishes with a grain of salt these days.
Regardless of what happened in this particular case, this type of scam is now possible, feasible, and will get easier to perform with time. We should be concerned.
I ask you, how will we be able to trust moving forward? Will all transacting be in person? We can fake id's too.
How will we be able to verify reality in this age of deception?
Genius.
But yeah a voice actor seems like an even easier (if less press-worthy) way to run this grift
I just went to cereproc, one of these companies advertising a very realistic synthetic voice, and none of their voices convinced me at all, though I have to admit it was pretty good.
When they get so good that they are indistinguishable except through Turing tests, and maybe not even then, we'll all be in trouble. I somehow expect that we haven't yet reached that point, though it can't be long in coming.
I can't distinguish most of them from human, even knowing ahead of time which one is which:
English: https://r9y9.github.io/wavenet_vocoder/
English: https://google.github.io/tacotron/publications/speaker_adapt...
> "If this turns out to indeed be a deepfake audio scam…"
It didn't literally happen, they _think_ that's what happened. I'm sceptical personally, I didn't realise deep fake audio could be done in realtime now? And who is this CEO that must have hundreds of hours of publicly available audio that the voice could be trained on?