Human study on AI spear phishing campaigns

I'm surprised you would expect better.

Everything I hear about their processes, everything I experience as a user, says their software development is all over the place.

Uploading a video on mobile web? I get the "please wait on this site" banner and no sign of progress, never completes. An image? Sometimes it's fine, sometimes it forgets rotation metadata. Default feed? Recommendations for sports teams I don't follow in countries I don't live in. Adverts? So badly targeted that I end up reporting some of them (horror films) for violent content, while even the normal ones are often for things I couldn't get if I wanted to such as a lawyer specialising in giving up a citizenship I never had. Write a comment? Sometimes the whole message is deleted *while I'm typing* for no apparent reason.

Only reason I've even got an account is the network effect. If the company is forced to make the feed available to others, I won't even need this much.

If they stopped caring about quality of their core product, what hope a billing system's verification emails?

I almost just hung-up because you have the rep urging you to do it and I'm trying to vet that every link is what it says it is before I enter any Amazon info. The cookies from my already signed in session did not apply to thia SSO either. Ended up working out in the end.

I could not believe they have the same flow as the scammers do. This is the same company that regularily sends out warning emails about phishing to me. Go figure.

Most companies are already splitting domains for customer and corporate communication, that's a step in the same direction.

While you're right it sounds fishy as hell, it's also mildly common IMO and understadable, especially when e-commerce is not the main business, and could be a reflection of how anti-phishing provisions are pushing companies to be a lot more protective of the email that comes from their main domain.

[0] https://www.nofraud.com/faq/

In my experience it's because getting a subdomain set up inside large companies is a MAJOR bureaucratic nightmare, whereas registering a new domain is very easy.

Yes, like you say, there's always the chance that someone hijacked an official domain - that's where other things like a formal communication protocol ("we will never ask for your password", "never share 2FA codes", "2FA codes are separate from challenge-response codes used for tech support") and rules of thumb like "don't click on shortened links" come in. Defense in depth is a must, but the list of official addresses should be the starting point and it isn't.

IT wasn’t amused when I reported it as phishing attempt.

I think (but am not sure) that something using trust networks from the ground up would be better in the long term. Consider anything dodgy until it has built trust relationships.

Eg email servers can’t just go for it. You need time to warm up your IP address, use DKIM etc. People can’t just friend you on FB without your acceptance so it’s a lot safer than email, if still not perfect. A few layers of trust would slow bad actors down significantly.

A trust network wouldn’t be binary. Having eg a bunch of spam accounts all trust each other wouldn’t help getting into your social or business network.

Thoughts from experts?

We have sandboxing on mobile apps. Why can't we have the same for desktop?

I’ll die on this hill.

It also does little against compromised mailboxes - heck, a sufficiently advanced spear fish might even have better chances if the user misunderstands the security improvements this would provide.

But I think other than this, there's not much else to fix. Some people are malicious, others get compromised. No fixing that.

Chrome 0-days are expensive and aren't going to be wasted on the masses. They'll be sold to dodgy middle eastern countries and used to target journalists or whatever.

If you aren't a high value target you can click links. It's fine.

My process when I see a sketchy email is to hover over the links to see the domain. Phishing links are obvious to anyone who understands how URLs and DNS works.

But working for a typical enterprise, all links are “helpfully” rewritten to some dumbass phishing detection service, so I can no longer do this.

At my current company I got what I assumed was a phishing email, I hovered over the links, saw they were pointing to some dipshit outlook phishing detection domain, and decided “what the hell, may as well click… may as well see if this phishing detection flags it” [0]…

… and it turns out it was not only not legit, but it was an internal phishing test email to see whether I’d “fall for” a phishing link.

Note that the test didn’t check if I’d, say, enter my credentials into a fraudulent website. It considered me to have failed if I merely clicked a link. A link to our internal phishing detection service because of course I’m not trusted to see the actual link itself (because I’d use that to check the DNS name.)

I guess the threat model is that these phishers have a zero-day browser vulnerability (worth millions on auction sites) and that I’d be instantly owned the moment I clicked an outlook phishing service link, so I failed that.

Also note that this was a “spear phishing” email, so it looked like any normal internal company email (in this case to a confluence page) and had my name on it. So given that it looks nearly identical to other corporate emails, and that you can’t actually see the links (they’re all rewritten), the takeaway is that you simply cannot use email to click links, ever, in a modern company with typical infosec standards. Ever ever. Zero exceptions.

- [0] My threat model doesn’t include “malware installed the moment I click a link, on an up to date browser”, because I don’t believe spear phishers have those sort of vulnerabilities available to burn, given the millions of dollars that costs.

Sounds right. I assume we will all have AI agents triaging our emails trying to protect us.

Maybe we will need AI to help us discern what is really true when we search for or consume information as well. The amount and quality of plausible but fake information is only going to increase.

"However, the possibilities of jailbreaks and prompt injections pose a significant challenge to using language models to prevent phishing."

Gives a hint at the arms race between attack and defense.

They all pass DKIM, SPF, etc. Some of them are very convincing. I got dinged for clicking on a convincing one that I was curious about and was 50/50 on it being legit (login from a different IP).

After that, I added an auto delete rule for all the emails that have headers for our phish testing as a service provider.

There was yet another study titled "Why Employees (Still) Click on Phishing Links" by NIH. (2020) https://pmc.ncbi.nlm.nih.gov/articles/PMC7005690/

Given the pathology, clicking is the visible and obvious symptom.

Also, in section 3.6 of the paper, they talk about just switching fishing email, to email helps.

Or said differently, tell it that it's for a marketing email, and it will gladly write personalized outreach

I don't think this is a bad thing, because even with peer-reviewed papers, there are cases where they can be completely fabricated but still get published. You shouldn't rely on a single paper alone and should do further research if you have doubts about its content.

"We take safety very seriously. Look how much safer our SOTA model is based on our completely made up metrics. We will also delay releasing these models to the public until we ensure they're safe for everyone, or just until we need to bump up our valuation, whichever comes first."