How is it not easy to check a reputation database for domains when evaluating a mail server?
Using DKIM records, Google could cross-reference the reputation for the sending domain and, where applicable, recognize that the domain in question has never been malicious or negligent. And in that case, extend a probationary reputation sufficient to allow the IP to establish its own new reputation.
> How is it not easy to check a reputation database for domains when evaluating a mail server?
The point I'm making in this thread is that they have the ability to maintain a database of domains that have been proven to be trustworthy. I am not talking about a database of IP addresses, and I am not sure why that is being raised here.
Huh? With dataset of ~100k classified hams/spams I get like >99% precisison in identifying spam/ham with just bogofilter and 0 heuristics whatsoever (my mail server accepts everything, and I just use bogofilter client side).
I guratanee you MS has a dataset with > 100k emails, lol.
It's not like this is unsolved problem.
My god, are you joking or what? Spam, a solved problem? Far from it.
Google reads and indexes every email that they deliver to a gmail inbox. They probably have the worlds largest and highest paid staff of ML experts, and they resort to auto-flagging based on IP address?
Fix it.
It's an unsolved problem for anyone.
> Why can I identify spam at a glance, with greater than 90% accuracy
I don't think you understand how terrible even 99.998% accuracy is at their scale.
> They probably have the worlds largest and highest paid staff of ML experts, and they resort to auto-flagging based on IP address?
You think they aren't using them for this? IP addresses very likely go into their model as well, when determining something is spam.
> Fix it.
They are and adversaries are breaking them again.
What does scale have to do with it, am I missing something? Wouldn’t a spam detection rate of 99.999% mean that only every 100000th spam mail would get through on an account basis, how’s that terrible?
The person I replied to said they can identify spam with 90% accuracy with a glance, my point was that it's abysmal as accuracy. Even orders of magnitude better it'd still be bad.
> Wouldn’t a spam detection rate of 99.999% mean that only every 100000th spam mail would get through on an account basis, how’s that terrible?
I didn't specify if it's per-account or a false positive rate, it's also actually not relevant to the point I'm trying to make.
When you receive billions of letters a month small errors start to matter and they're doing a better job than the person I replied to thinks they do.
Honestly, the best filter in my experience is Spamhaus Zen; it spots most spam with very few FPs. It's an IP blocklist.
In the end it's still a hard problem, so takes a bunch of work.