It sounds like science fiction but it’s not: AI can destroy your business
theguardian.com
theguardian.com
Unlike scam detection and filtering methods based on pattern recognition or reputation, LLMs can analyze the content of a scam rather than its artifacts. By that I mean, you can ask the AI "does this make sense, or is there reason to believe it's a scam? If so, how can we verify that it's not a scam, knowing that deepfakes exist" rather than "does this fit x y z pattern that we have seen before?"
I'm hoping, and I may be wrong, that most scams rely on emotional hooks, unsuspecting victims, or desperate victims. LLMs aren't vulnerable in the same way.
Maybe.
Surely you have witnessed (Generative) Adversarial Networks. It's called "an escalation" (attack vs defense, etc.).
More to the substantive proposal in the post: remember that incidents during surgery are dramatically reduced just by using checklists - you can already reduce incidents through deterministic regulation of action. And you do not require "fuzzy" systems like ANNs for that.
The scam has to be both human-comprehensible and plausible in order to be delivered to the victim in the first place, which places a limit on its complexity which the adversarial anti-scam LLM isn't bound to.
The only ones that benefit really are the AI makers.
Most scams are fairly transparent and rely on wetware failures, right? The anti-scam AI doesn't have to overcome the scam AI, it has to compensate for the emotional responses or unpreparedness of the target. I think that's less of an ask than generating bulletproof scams. I could be completely wrong.
Information follows the same S curve most adoption follows.
Anything you collect, is a liability once its stolen and eventually it becomes freely available regardless of promises to the contrary.
If you meet with somebody in person, you know they are not a bot and you know what they look and sound like directly.
You already need to meet someone in person to get a true feel for them. There are so many things that don't come throug video. Smell, for instance, generally plays an important role in our perception.
I am curious if these scammers are pre-recording conversations and playing them back in real-time or if they are actually able to speak using the cloned voice technology.
Yes, it has been so for years. The attack to the Dubai Bank in Hong Kong was effected in January 2020.
"Though users at the Audio Fakes Discord and the DeepFaceLab Discord are intensely interested in combining the two technologies into a single video+voice live deepfake architecture, no such product has publicly emerged as yet."
I am not aware of what was used in that occasion; there is also a good chance that the scammers could hack a system internally - the stake is stellar.
Surely, to convince the other part during a phone call a high degree of capability for interaction is required; from the same article, right above your quote:
> we can reasonably assume that the speaker is using a live, real-time deepfake framework
The quoted article is from late 2021. In these current days, we are seeing denounces according to which
> Voice scams are on the rise in a dramatic way. They were the second most common con trick in America last year, and they’re headed for the top spot this year (ex Voice scams: a great reason to be fearful of AI at https://news.ycombinator.com/item?id=35459261 )
In operation in the video's it seems to work pretty well. :(
nor the other possible point that Statistical Large Language Models are not problem solvers, as in fact are special in Machine Learning for optimizing for goals transversal to actual "solutions" (not to mention that it is already a Sam Altman, while proud of the results («They are not laughing now, are they»), the first to jump into alarm when people drool "So it's AGI!?" on him),
but it must be noted that those scams happen because people live lightheartedly their responsibilities (among which, realizing that they are not in the XVIII century anymore) - and the post has a tint of this dangerous laid back approach.
(I would suggest that you mark sarcasm, e.g. with '/S' or equivalent. Once upon a time we all thought that rhetoric is immediately recognizable, then we met people who would believe ideas beyond the boundary of "anything".)
We have been there for the past few years. And yes, it has been actually used for scamming.
Submitted only a couple of days ago: Voice scams: a great reason to be fearful of AI - https://news.ycombinator.com/item?id=35459261
That said I think trying to verify someone's identity through online means only became viable a few years ago when everyone had a somewhat working camera and microphone available, and with any luck the risk of deepfakes will cause an early end to the the scourge of people trying to film themselves holding a form of ID.
Actually applications like this will probably be implemented on the cloud-based models, since 98% of the public does not care about privacy as much as people on this forum.
Fake ads for your opponent in their voice spouting bonkers positions and nonsense. Everywhere. Online, voicemails, far as the eye can see.
It’s gonna be rough.
But the education of the public has already started, with the deepfakes that have circulated in the past few weeks, from odd or dreamed depiction high "authorities" to discussions about the status of documents related to e.g. the protests in France.
personally I'm more worried about the next one where the AI infestation will be far more subtle
What about The Spectator - https://news.ycombinator.com/item?id=35459261
The very agents that can be scammed into damaging you are not aware of the state of facts. (They may read those "«tabloids»". It's educational.)
Video for anyone interested:
As for bank manager transferring hefty sum without some proper verification - the problem is not scammers but the manager themselves.
The way the scam is said to work is that someone pretending to be your relative is telling you they are in an unusual situation away from home and it is very urgent so you must send them money in a different way.
e.g.
"The caller said she was locked up for a misunderstanding between herself and the Mexican state police of Quintana Roo. They needed money for a Mexican lawyer and court costs"
(That particular incident, apparently they were impersonating someone who had been to Mexico, but had - unfortunately for the scammer - returned prior to the call)
https://consumer.ftc.gov/consumer-alerts/2023/03/scammers-us...
Sadly, you are only one person on a planet of more than 7 billion.
I worked with someone whose mother had been scammed multiple times by folks claiming to be his (deadbeat) brother. Social engineering works, especially as the attacks get more sophisticated.
How would you implement that? You cannot certify a timbre you can simulate.