The golden age of scammers: AI-powered phishing
mailgun.com
mailgun.com
The technology for customised text based attacks at scale has been available at least since Llama was open sourced. The tech for custom voice and image based attacks is basically there too with whisper / tortoise and stable diffusion - though clearly more expensive to render. I'm honestly not sure why social networks aren't being leveraged more to target and spoof individuals - especially elderly people.
Tailored attacks impersonating text or voice messages from close contacts and family members should be fairly common, and yet they're not. Robo-calls that carry out a two way conversation convincingly impersonating bank or police officials should be everywhere. Yet the only spam-calls I ever receive are from Indian call centres or static messages using decades old synthesised voice tech.
They recently took my mom for $25k for what was a few hours of “work” over the span of two days. When I reviewed their communications and got the full story from my mom they’re in some ways laughably bad and in other ways very cunning.
Turns out it all started with a comically bad initial e-mail, pop-up, and then remote access. Then follow-up calls and text messages (Bitcoin QR code with her bank logo) and multiple people impersonating banks, various government agencies, etc. The end to end pipeline to replicate this via AI would be very complex and difficult (today).
I imagine the increased scale, additional opportunity, and reduced “payroll” that could be realized utilizing AI (given the initial level of development effort) just isn’t there. Yet.
My Mom got one of those calls where the guy said my son was jail and they needed to pay his cash bail to get him out. They never called me or him, just panicked and went to the bank to get the cash out and the bank refused to give them the money because they knew it was a scam. They started asking my parents questions and pointing out how the scam worked when they got agitated they weren't giving them the money.
They called me from the bank and I texted my son and he was completely confused since he was at work and was just fine. I thanked the people at the bank profusely for their intervention.
It was a great learning experience for my parents as well. They are way more leery about strangers calling and have already hung up on several scammers. They also filter out email messages and won't click on any links in emails or pop-ups.
The whole experience really put their radar up for this stuff now.
A friend of mine runs a retail bank branch for a major bank. The other staff refer customers to her if they suspect a scam. She sees a few of these each month.
But those are just the people who personally come into a bank branch.
As part of that we went to the branch where she made these cash withdrawals. I had the opportunity to speak with the teller that was there both days. The teller's story (and I believe her) was that she was extremely suspicious of the overall situation and had an extended conversation with my mom about how unusual this was, often used for scams, etc. However, back to my original point the scammers were well ahead of this...
My mom banks with Wells Fargo. Once the scammers discovered that they were able to capitalize on the news story from years back with the Wells Fargo fake accounts/fraud scandal. They even sent her links to news stories about it.
They were able to convince my mom that her money wasn't safe at Wells Fargo because the government was investigating another Wells Fargo scam and that all of their employees are in on it. They had another scammer in the org impersonate someone from the Federal Trade Commission (ridiculous) who was supposedly investigating this.
They prepared my mom with a robust script to dance around the teller's questions and skepticism that the cash she was withdrawing was for legitimate purposes. In this case the script was something about my mom doing construction on her home and paying laborers, contractors, etc in cash.
The entirety of the scam is wild. Once they got remote access from the phishing e-mail they got opened up Windows Command Prompt and pasted in a bunch of echo statements with stuff like "Child pornography detected" and a bunch of other ridiculous stuff. However - let's remember that thanks to Hollywood any terminal interface looks scary and sophisticated to the general population (every movie ever "hacking" with CLI gibberish).
They had her convinced she was being watched/followed, investigated for CP, her phones were tapped, etc, etc, etc. She was so freaked out she didn't know if she was going to be killed or go to jail.
My sister happened to try to reach her and my mom called her back from a friends phone. Was sister somehow stumbled on this and of course knew it was a scam. My mom didn't believe her. Needless to say I have a lot of credibility in my family on this stuff (given what I do) so thankfully my sister was able to get my mom to call me.
At the beginning of the call I was able to say to my mom "This is a scam. Let me guess: they did X, then did Y, told you Z, something with cryptocurrency, etc, etc". Only when I more-or-less nailed/predicted most of the details before my mom could even tell me her story did she realize that this was, in fact, a scam.
It made me realize something: when we were growing up a saying was "talk to your kids about drugs". There needs to be an equivalent campaign for these scams. Like many here I follow this kind of stuff pretty closely but only thought of it as a curiosity because they're so ridiculous (to me/us). It never occurred to me that I should be regularly updating less sophisticated friends and family members on the scams du jour.
HN Community: Talk to your friends and family about scams.
THIS.
I've been saying this for years. I have no idea why an a world that has been inhospitable to older folks for decades, why there are no programs to help elderly and at risk people to keep an eye out for these.
I'm sorry to hear about your mother as well. That's heartbreaking.
(1) cheaper methods work too well for now to even bother with sophisticated approaches
(2) it’s too expensive to be worth the effort
It would be interesting to plot out the cost of an integrated AI stack for this over time.
https://www.npr.org/2024/05/31/1197959218/fbi-phone-company-...
Pig butchering scams uses all forms of kidnapping, human trafficking, and slave labor
For perfect audio spoofing, lots of audio would be needed. Bear in mind there are literally millions of podcasts available [2], and billions of youtube videos. Should be trivial to grab biographic data and voice samples from a subset of them.
[1] https://github.com/neonbjb/tortoise-tts
Afterwards I realized it might have been some sort of scam attempt. He seemed to be in such a rush to get it done tomorrow and book a time now now now. And he sent no mail.
I guess the scam would have been to prepay some sort of fee or the whole cleaning. But I've got this feeling he was fishing for something more. Dunno if they have the know-how to actually use my voice to scam e.g. my mother.
That sounds like a great recipe for spear phishing public figures.
I'm being glib here but also if you're the type of person who gets texts from the IRS from a number you've never seen and take it at face value that you can pay off your overdue tax bill with gift cards... like, you are already the perfect victim for this sort of scam. They don't need to be good, they just need you to self identify and leap right into the trap.
At the same time, most people are more clever than one tends to expect.
-Justice Robert H Jackson
Always has been.
Tbh the browser/email client makers are complicit in these phishing attempts for hiding the URLs and the actual email addresses.
Put them back!
It's worse. Research "Scamicry".
Big business now is so fake, such a grift, drenched in PR deception, and lacking integrity and trustworthiness, there isn't much space left between what is "legitimate" and what is a scam.
If businesses like Google or Facebook hide URLs and email addresses that's not a casual "mistake". It's because that's to their profitable advantage to do so. And they know it puts you in harms way. So yes, they're complicit in scams.
To make themselves a little more competitive businesses are always learning from scammers, meanwhile good scammers keenly learn from businesses to look more legit. Some ransomware "services" even have better customer support than billion dollar companies. And big business is certainly using the same AI tools as cybercriminals.
So a problem isn't how clever and scurrilous scammers have gotten, it's how far legitimate services have fallen so that ordinary folk struggle to know the difference.
How can we trust our own insticts for selecting what is good and wholesome from what is rotten, when there are few moral differences? The only difference resides in a digital identifier.
The last two companies I worked for insisted that customer account security was the highest priority, but as soon as I said we needed to stop hiding links to our own website behind Hubspot tracking URLs so we don't train our customers to click links that look like gobbledygook garbage, the marketing team melted down and it became clear where user account security actually fell on the priority list.
I don't think it's always malicious, though. I think most people in most companies just don't realize the risk. Like, I had to explain to my doctor's office why I'm never going to "confirm my identity" by rattling off my DOB and address at the beginning of a call when they called me. I even think of those specific data points as public information anyway and I'm not going to participate in that nonsense. It had never occurred to them that this was risky behavior.
It did make me appreciate my parish priest's method, though. Every quarter or so, he reminds people from the pulpit that he will never email them asking for gift cards or anything of the sort. If the parish needs money for something, he promises he'll ask for it right from the pulpit!
[0] https://www.fastcompany.com/90457589/science-proves-it-men-w...
The natural set of incentives does not work well with human psychology. It does not prevent mishandling of motor vehicles, or at least fails to prevent it in enough cases that additional disincentives are needed, to ensure the safety of the public.
Stated another way: Enough people are bad enough drivers that we need laws and civil liability to create additional incentives against bad driving. The threat of collision, property damage, injury, or death to oneself, passengers, and people outside the vehicle is clearly not sufficient.
Driving is actually a great analogy, but maybe not for the reason you intended. If we relied only on individuals to act responsibly, the roads would be much more dangerous than they currently are.
Not sure we should stretch the car analogy too far, but you're right about rules and regulations. Most people can be relied on to be considerate, but that one percent of assholes ruin it for everyone.
From where I'm standing that one percent is basically American big tech. Problem is, we have traffic lights, stop signs, speed limits, and highway cops patrolling, but the big US corporations just drive like assholes and get away with it anyway. In fact they're more like terrorists that drive a truck into a crowd of people, and we all stand around helplessly and wail... "what can be done?!"
Now, if this was a proper American tale, there would be a Blues Brothers style 500 car chase, and at the last moment, just as the bigtechmobile is about to jump the unfinished bridge Dukes of Hazard style, Bruce Willis swoops down in an Apache attack helicopter and blows them off the map!
oops I think I've stretched the car analogy too far.
It really does help especially the elderly parishioners to hear it directly from the mouth of someone they trust, though. And he tells them if they have any concerns, any doubts about a situation they've gotten into, they can always call the parish office and they'll do the best they can to help them with whatever issue they're facing.
Even just training people to always hang up and call someone who definitely would know if the person they think is reaching out to them actually needs help is such a good first step. Even if the scam email "from" the priest says, "I can't talk right now," call the office. If he were in a bind, he wouldn't be the only one who knew about it.
They get paid every time a scammer makes a phone call.
https://news.ycombinator.com/item?id=40942307
Imagine old people getting phone calls from frantic children. They won't know real from fake. Add tech like this to SIM forgery ..and we will devolve from a high trust society to a no trust society.
* Bayesian poisoning https://en.wikipedia.org/wiki/Bayesian_poisoning
* Weeding out poor mark candidates https://josephsteinberg.com/why-scammers-make-spelling-and-g...
Easy. As you say all you need to do is to compare and as I occasionally have a glimpse of the spam for some non-English language with a non-Latin script - it's the same.
I noticed how well it understands general parlance after it created marketing style copy for me and I told it to sound "less wanky" and it made it much more to the point.
Probably the funniest thing here is that this call reached me despite the fact that I am French, living in France. And so I really wonder how they ended up calling me. I mean, chance I would understand some English speaker with an Indian accent (I like how it sounds, but it’s definitely an additional difficulty as a non-native).
I read here and there how extortion of old USA citizens by some organized Indian citizens is really a thing. To my mind the main issue at stake is that we have global level communication facilities, extremely high wealth disparities at world scale, and no compelling global social endeavor to reach an harmonization of human quality of life for everyone. I don’t mean the latter is on the official agenda of most countries out there either, but at global scale it’s obviously even worst.
With all that in mind, blaming a whole nation for the illegitimate actions of some minority in the country, all the more when the international geopolitical context itself is all but fair, is probably not going to solve any issue.
CBC the Canadian nations news service has been trying to track scammers in India
https://www.cbc.ca/news/world/tech-support-scam-india-market...
Even Jim Browning and Kit Boga on YouTube two guys who scam the scammers it seems to be 100% people based in India.
Note that India has had a larger population than China for at least a year or two.
(That is, on top of all the other "free drops/skins" impersonation that twitch can't seem to shut down)
Seems like a fairly simple thing to solve though, limit ages of accounts needed to be featured etc
On the other hand, twitch keeps firing employees, so they probably just ban the stream account every 20 minutes because they don't have the manpower.
I would be very surprised if they are not using a new, clean residential IP on a US ISP for each and every account.
Note that the admiration/detestation opinion might not be as socially mandatory. But probably it’s as optional as an agnostic position is tenable in a society full of theists looking for some heretics to burn on the one hand, and fanatical atheists eager to decapitate any devout on the other hand.
A friendly fast talking man in an extremely busy sounding (fake) call center asked me if I was $name_I_put_in_the_form. The voices in the background were people further down the sign up process.
I said yes, then asked how he got my number. He said I just filled out the form on the website. Then the form was replaced by a new page.
They did a good job confusing me, it was very impressive. I don't confuse easily.
That's necessary for realism.
If the deepfake were doing or saying something decent, that would be a dead giveaway.
a common heuristic to look out for is "badly"-written/spoken communication. the "AI vs Actual Indian" comment and nigerian prince emails stand out for most people, but they still ended up working well enough to become this wide-spread.
you just need to employ some critical thinking now for most external communication now. it is no different from some highly-motivated scammers doing it the old-fashioned way. at the end of the day, we are trying to replicate the success of some native-speaking teens (https://news.ycombinator.com/item?id=32959001).
Can someone show me a modern OS that would install software by clicking a link?
Because you know how to do that, and it's so much easier than helping them when they get hacked.
Friend receives an email from ISP, asking her to contact them.
She searches, comes across a "customer service number" on a legit looking page, calls them up.
(Whoever she called) plays out a 30 minute charade about how she's been flagged by IRS for illegal activity and is about to have her business accounts frozen, including multiple phone transfers to "another party" (played by different people) to boost authenticity.
And during this whole time, they not once asked her for any "red flag" information (e.g. account #, SSN).
Instead, it seemed to be a shell game of extracting limited information (last 3 of your account #?), then having "unrelated" parties parrot that back as proof of their "working for the government."
I expect it would have eventually escalated into an actionable ask, but they were definitely playing the intermediate-term game.
If not for the utter moral black hole of the endeavor, I'd be kind of impressed.
Last time I did this, it took three days of texting my new friend before it was finally clear that what she really wanted more than anything was to teach me to trade cryptocurrency.
Once, I thought I had her, because she spelled D&D like: D&D, but she played it off real cool and just explained that her English isn't that great so she used translation software.
In retrospect I think that all of her probing questions about my Svirfneblin cleric were because she later intended call him up and teach him to trade cryptocurrency. I like to think he's in some scammer's database now, causing confusion. He'd like that too.
Once I understood what she was after, I explained that my problem with cryptocurrency was that it resembled money too closely and really what I'd like to do with blockchains is to do away with money in favor of something entirely different.
Her training dataset had not prepared her for this conversation, so it was quite clear when her human handler took over. They were very rude, unlike their AI pet, and tried to bully me into sharing other people's contact info, which is when I lost interest.
Even then, that won't help scams and fraud that just trick you into sending money, or direct you to install malware.
Talk to them about investment / romance scams as well. Unfortunately, most folks do these things "willingly" and get in deep.
I bought Mom a Yubikey and helped her set it up on her Google account. She has it on her keychain. She doesn't need to remember how to use it, though, since it's only needed when she buys a new computer.
For good measure, I also helped her print out backup codes (and I know where they are) and I registered my Yubikey, just in case.
Nowadays, an old backup phone might also work, but I think paper backups are better because an old, unused phone might not start.
If you want to create an average looking fingerprint, use an AI
But true randomness is not useful for determining if you are who you say you are.
Yet my bank just turned this on for me as well in 2024. Now I have to figure out how to disable it...and will it really be disabled?
Just give it to everyone. Today, it can likely be embedded in a cell phone instead of separate physical card.
https://krebsonsecurity.com/2018/07/google-security-keys-neu...
I adore my Yubikey.