Isn’t this an issue with the spam filter more than the AI? Why should the AI be doing spam filtering? They’re two different things
(The spam filter might use statistical or ML methods of course but it’s a different software?)
(The spam filter might use statistical or ML methods of course but it’s a different software?)
This is basically the Achilles heel of LLMs: they’re gullible, and in a context like spam there are many people with a financial incentive to figuring out how to exploit that. The rush to deploy them will lead to more of these problems as people start using them on untrusted inputs at scale and I imagine a ton of money is going to flood to people who say they can limit this.
We had a 'Please review this invoice' email get through. Generic email message with a PDF that had an exploit and a web link to some low-quality impersonating site. Infected the users computer, sent a copy to everyone on their address book, got another person inside the company, sent a copy to everyone in their address book...
IT staff had to manually intervene with those two users. Disable their accounts, rebuild their machines, change their passwords, etc.
Who's job was it to stop that email?
Was it Microsoft? Our email is hosted through them. Exchange Online boasts:
Data loss prevention capabilities prevent users from mistakenly sending sensitive information to unauthorized people. Globally redundant servers, premier disaster recovery capabilities, and a team of security experts monitoring Exchange Online around the clock safeguard your data.
What about Outlook itself?
Advanced data, device, and file security
Maybe our AV/EDR software should have caught it?
AI-powered prevention, detection, response, and threat hunting across user endpoints, containers, cloud workloads, and IoT devices. Enabling modern enterprises to defend faster, at greater scale, and with higher accuracy across their entire attack surface, we empower the world to run securely.
Maybe our firewalls should have caught it. Packet inspection, URL ratings, lots of things should have triggered something.
And then our SIEM...I guess we had it all logged, even though we never had any warnings or messages from them. So much for millions of community submitted icidents and threat intelligence and whatever else they sell to make people sleep at night.
Although I don't think iOS Mail app has spam filtering like the desktop Mail.app if you're using your own server, at least mines never worked if it does.
Yes.
As to the other questions, the person reporting this is one of the founders of Panic¹, who are trusted developers who have been making Mac apps for decades. So you can be reasonably sure there’s at least a modicum of due diligence in the report.
> Are they replacing whatever spam filter they had with Apple Intelligence?
I’m not sure. I don’t think so, but I also don’t know if we know for certain.
> Is there usually a spam filter at all?
Yes. In addition to what may be marked as spam on the server, Mail can also do its own filtering.
https://support.apple.com/en-gb/guide/mail/mlhlp1065/mac
> Is the usual spam filter disabled?
Can’t say, as I’m not the reporter. But again, email can be marked as junk from Mail, the server, both, or neither.
I bet it’s just a question of do you want ai to try to find important messages the spam filter accidentally flagged or not.
(Me? No)
I got your point, but according to most definitions ML \subset AI.
Also, LLMs are not as explainable as classic ML algos, but they might certainly have its place. The real problem is that it was not combined nicely for nice user-experience and (probably) False Positive Rate is higher than what people expected from trendy "AI".