How Spam Filtering Works: From SPF to DKIM to Blacklists
deliciousbrains.com
deliciousbrains.com
The new hotness is a DMARC record, and it finally allows mail senders to basically say "Here are my suggested handling policies for DKIM and SPF, and please actually enforce them because they are properly maintained and tested." Best of all it has a feedback loop, so you are able get reports of how receiving hosts are treating your messages and forensic reports of forgeries which can help with detection of several types of phishing threats.
It's still a long way from perfect, but it's better. It doesn't help deliverability though. For that you will want to stop emailing shit that people don't want.
For DMARC, it's not so awesome :\ https://news.ycombinator.com/item?id=17900765
DMARC is overall a very good thing, and has significantly cut back on the effectiveness of forged email for spam/phishing, at the expense of mailing list/forwarding headaches.
On a related note: do you know if DMARC feedback leaks information to the sender on whether or not you opened an email, or reported it as spam, or similar? If so, do you know why this isn't considered a privacy or security issue?
"Again" as in get other actors in the field to allow/ease that: spamlist managers should stop blocking residential address by default and make it easier to appeal. ISPs should make it possible and easy to get a reverse-ptr with every fixed ipv4 allocation.
In this monitoring age I want to be able to handle my private conversations (and my data) on my own iron (running somewhere in my house).
All this impediments to running your own mailserver and take back control of your own data really look like a push towards state-/corporate-backed espionage of individuals.
It's not 1995 anymore, most of us have an internet connection well capable of sustain enough mailboxes for a whole family.
The server-hosted software would never see the actual emails; it would just manage the provision of email service. Perhaps the server could store locally-encrypted inbox backups too.
If the service was responsibly run, it could even "vouch" somehow for the email inboxes it provisions. So even though the emails come from, like, Comcast ISP address space, other ISPs would recognize a trustworthy source of email. (Dynamic IP provision would complicate this part.)
But the big question is... what's the addressable market?? It would likely be even more expensive than something like ProtonMail. You're not going to build much of a business off of 347 neckbeard paranoics. (I joke with love.)
...
> If the service was responsibly run, it could even "vouch" somehow for the email inboxes it provisions. So even though the emails come from, like, Comcast ISP address space, other ISPs would recognize a trustworthy source of email. (Dynamic IP provision would complicate this part.)
If you are willing to to let the server-hosted software have some visibility into your mail traffic (but not the content), there's a way to handle this that would not have problems with dynamic IP addresses [1].
Briefly, whenever your email client needs to send a mail through your Comcast outgoing SMTP service, it would first contact the vouching service and provide a hash of the message ID of the message it is about to send (and maybe the from address?).
The vouching service would verify that it is talking to one of its legitimate customers, and then return a line for the client to put in the header of the outgoing mail. This line would contain a copy of the message ID hash (and from address?), a timestamp, the client IP address, and would be signed using a key of the vouching service.
The client puts that line in the header and sends the mail through Comcast.
The SMTP server that Comcast relays it to can look at that line, check the signature, and if it verifies it can compare the information in that line to the corresponding information in or derived from the rest of the header. If the information all matches and the timestamp is in the recent past, accept that the mail really came from the person it claims to be from.
This would give the vouching service get a good idea of how much email you send and when you send it, and also give them hashes of your message IDs, so it is not ideal.
A problem with this is this it depends on every SMTP service that Comcast might relay your mail to (such as the SMTP server for every domain you send mail to) being set up to verify the signature of the vouching service you use. This would be a pain in the ass.
This could be addressed by having the verification of the signature and header data done at your ISP's (e.g, Comcast's) SMTP service. That way you just have to pick a vouching service that Comcast recognizes.
Comcast would then do the check on your outgoing mail to see that you have used a vouching service they recognize and the data matches. They can then add a header that says they believe the mail is verified. For any mail that they cannot verify thusly, they can add a header that says it is not verified.
Now instead of having to know about and trust your vouching service, the SMTP servers of the domains you mail to just have to trust Comcast. That seems much more likely to come about.
If Comcast (and other residential ISPs) were willing to do this, why bother with third party vouching services? Why not handle the whole thing themselves?
One reason would be that providing vouching service requires getting more involved with the details of the customer's setup than an ISP might want to deal with. They might not want to have to deal with support questions dealing with how to get the vouching plugin or whatever working with the various email clients and operating systems customers use.
[1] Maybe...I've not sat down and tried to actually design the protocols for this. This is just off the cuff.
There are two major, somewhat interlocking problems, though: home users with compromised machines currently represent the population of home "mail servers" (spam malware) at the moment. Selling this involves convincing mail administrators that not simply blackholing all of what is currently a cesspool is a good idea. That's not easy.
The second problem is that fixed-IPs, open well-known ports and DNS have become a product differentiator for ISPs. Those are "business" features; dumb ole' consumers don't "need" them.
I don't see any compelling argument that would convince either prong of that trap - hell, I agree with you, but as a mail admin both professionally and personally, the first one gives me hives, thinking about the years of work it'll take to re-stabilize a functional antispam infrastructure to handle it.
And after that, you need to convince Comcast to change their revenue-extraction plans and generally be less dickish.
If just you want a mail server now, your best bet is a cheap VM somewhere. It can be really low-end and cheap.
An ISP's job is to forward IP packets. Not read them. TCP/UDP ports number are the content of those IP packets—not the meta data. Filtering based on such content is already a form of discrimination. What's next, deep packet inspection?
Some ISP filter by default, but they do this at the router level, and you can either configure or change the router. That's okay, possibly even beneficial. Blocking at the backbone level however is just evil.
It's a real headache. The average HN user is in like the 99.9th percentile of residential internet users. HN users, give them a proper DIA feed, sure. Everyone else, I'm wary.
And we have the right of doing this.
Anyhow: the dynamic ip problem might be solved with ipv6. The spam problem... Imho graylisting + spf + dkim + reverse-ptr should be enough to kill spamlists once and for all.
I'm all in favor of ipv6 everywhere but the current reality is that a huge number of organizations presently have v4 only smtpd in their MX records, if you try to do a pure v6 approach you will break delivery to/from something like 65% of the internet.
For postfix, we now have the project at <https://github.com/stevejenkins/postwhite> to help automate the maintenance of whitelists of sites that break gr[ae]ylising.
IMO, we passed the point years ago when gr[ae]ylisting was still a useful technique. For anything within the last decade or so, I would say it probably does more harm than good.
I've seen customers max out gigabit links with outbound spam from a compromised desktop. Contacting the customer leads to ignored emails and unanswered voicemails at best, angry Facebook rants about "getting in my face" at worst.
Blocking ports by default is okay, and will address the clueless user problem (clueless users won't change the defaults to begin with). It's blocking them no matter what that's evil.
Now I live in a country where most ISPs lend you a "box", a router, which is capable of doing all the blocking they want, and is configurable by the user. One of them uses that box to block the SMTP port by default, and let the users who want to change that setting.
Problem solved.
I understand the fixed ip thing, but closing well-known ports by default should be considered a form of censorship.
It is like selling a mobile phone, except the user find out that they can only listen but cannot talk.
How do you make that work? In my experience, the mail will just be blocked because the IP has no reputation and the range is probably grey/blacklisted.
This will be really hard, because unless you're willing to give home users actual permanently assigned static IP addresses and treat them like medium sized businesses (IP space SWIP to customer, etc), any given /27 to /23 sized v4 DHCP pool of residential customers is going to have a ton of abuse coming from clueless people running infected routers, virus/worm/trojan infected home PCs, etc on it.
Getting the average residential ISP to do proper rDNS for your IP address is impossible in many places.
Therefore when recipient SMTPd are scoring the IP space reputation of your residential netblock, your home mail server is going to be mixed in with all the shit.
Proved difficult when I once went on holiday as I switched it off and then it got flooded with emails (mostly spam) when I got back (from backup mxs run by friends). Had my ISP call me as worried my PC was on a botnet due to sudden traffic. Explained I was running Linux, and they immediately said: "thats fine then have a nice day". Thought perhaps they would need more detail but no. ¯\_(ツ)_/¯
As others have said. Today's cheap nano-esque cloud instances (with swap disk enabled) on AWS and similar is easier to have running 24/7, and less risk it will burn your house down...
Have a static IP and properly setup your SPF, DKIM and DMARC...you should be good to go.
[1] - http://www.gabacho-net.jp/en/anti-spam/anti-spam-system.html
Also, why not keep it on a hosted server somewhere? You could encrypt it at rest to prevent anyone coming in and getting the data, which would make it as private as storing it at home. And then you don't have to worry about deliverability or backup power as a nice side bonus.
Today, the single most effective thing you can do if you run your own mail server is to completely block all gTLDs. Screw 'em, they are 99.9999% spam.
Plenty of spam has valid SPF and DKIM records. They are sent through legit services, either through cracked credentials of real users or rotating through new accounts.
It also doesn't seem like anybody cares about abuse@/spam@ reports anymore...
[citation urgently needed]
Anyway, in my own experience with many years of self-hosting mail - until giving up and going Fastmail a couple of years ago - the real problems were in sending. No matter what rigorous level of DKIM'ing and ip-hygiene and whatnot, Google and Microsoft - Microsoft to a grotesque degree - would randomly ditch incoming mails from my server. Would sometimes happen in the middle of a conversation thread, and for the most part without warning. The kind of person using Hotmail is typically not someone you can convince that the error lies in his end.
That's for _my_ e-mail. Your experience may differ. Perhaps you communicate often with people on .loan domains.
E-mailing Microsoft accounts is just not an option. They have no process for fixing incorrectly blocked IPs. I take the same approach with Microsoft as with gTLDs...
Never had a problem with any other mail provider. Google has never blocked or spam-holed me.
I also looked into the result of SPF [2] and again, found it not worth implementing. I rechecked the results for SPF earlier this year [3] and the results were the same.
[1] http://boston.conman.org/2015/05/11.1
We have reviewed your IP(...) and determined that messages are being filtered based on the recommendations of the SmartScreen® Filter.
Email filtering is based on many factors, but primarily it's due to mail content and recipient interaction with that mail. Because of the proprietary nature of SmartScreen® and because SmartScreen® Filter technology is always adapting and learning more about what is and isn't unwanted mail, it is not possible for us to offer specific advice about improving your mail content. However, in general SmartScreen® Filter evaluates specific words or characteristics from each e-mail message and weights them, based on their likelihood to indicate that a message is unwanted or legitimate mail.
Unfortunately, after reviewing the information you provided and in compliance with our mail policies, we are unable to offer immediate mitigation for your deliverability issue. However, we have some specific recommendations for you to consider that can help you to improve deliverability over time.
It's a struggle running your own mailserver.
The problem is the burden of not upsetting a mail filter got somehow shifted to the sender in case of big services filtering mail, which is ridiculous. Customers should talk to their providers if they're not receiving mail, because providers don't deliver it. But the correcting feedback loop is not there or/alternatively is much longer with more steps than in case of sender getting a bounce.
From: VlAGRA_&_CIALIS <sale_support0oag@xxxxxxxxxxxx>You need SPF and DKIM, but before that you need FCrDNS.
Without FCrDNS, your server will look like a dynamic IP address.
It's also not enough to monitor your own dedicated IP address. You need to monitor your entire /24 IP address neighborhood at your hosting company. You might be sharing the same /24 IP address space as a hacked Wordpress installation and find that Outlook.com will block you just because of that.
After that, you might run into issues with large SpamAssassin setups hosting hundreds of thousands of mailboxes blocking your email because of arbitrary rules such as too little text combined with a large image etc. Plain text email will improve your deliverability for these setups.
SPF nor DKIM helped, I don't own the mail server nor want to pay Microsoft's service provider to verify me.
In the end I had to bounce the emails from a third-party address. Which you'd think shouldn't work -- certainly not when whitelisting doesn't -- but solved the problem.
So to avoid being flagged as a spammer, do the spammiest thing possible: set up a new, free mailbox and bounce your mail from there. SMH.
The neighboring IP space and the general IP space reputation of the ARIN/RIPE/APNIC/AFRINIC (or whatever) block assigned to your ISP is extremely important. Your neighbours can ruin everything for you. The only way to ensure that this doesn't become an issue is to host your smtpd at an ISP that has a highly attentive, clueful NOC and responsive abuse policy that actually deals with network abuse (real abuse, not DMCA notices).
So in your example, you'd train it on your patterns of behavior AND patterns of burglars. So unless you had a clear burglarly behavior, the system would probably not flag you out for just behaving differently.
It turns out that no matter what I do, I'm pretty much out of luck because I'm sending from a non-standard (.link) domain which gets treated as spam by Gmail as well as others. I get a perfect score on every email test site I've tried, but I still go to spam (or worse) regularly which is a big problem if I ever want to use my personal email for anything important.
For anyone looking to host their own email using a personal domain, I'd suggest sticking to something like .com to avoid the hell that I've been through with this.
I'll be blunt, and maybe others will disagree with me, and say that I don't think there are any impactful levers you can pull.
In my experience, Gmail frankly does not give a shit about receiving e-mail from any senders except ones that fall in one of two categories: a) very large so users will complain in large groups if receiving from them is impacted (put your Outlooks and your Yahoos and your huge residential ISPs in this category) or b) technically-savvy senders that aren't quite as large but Gmail employees routinely interact with so they're otherwise "trustworthy" (Fastmail is an example here).
I've ranted about here but I've had terrible success being smaller than a multinational ISP and sending e-mail to Gmail users. Messages will silently disappear, even though Gmail's SMTP server claims to accept the message, and the postmaster tools are less-than-useful (that might be my one suggestion, sign up for them on behalf of your domain and see if you get any reports; I didn't, 'too small'), and no amount of tinkering or asking would help.
I admit I'm just a SMB org whose business isn't sending e-mail--it's "just" a communications tool for us--so I had this option: I threw in the towel and switched to Fastmail and have had no problems since.
I had the same issue, what I found though is that DMARC reports give at least some insight into the process, but that's about all the information I've managed to get from Google.
This is a big red flag, though. Are you saying it just happened one single time? In that case, change your IP and start over.
Or are you saying that you regularly trigger a lot of hard bounces? Because if that is the case, that might explain it all right there. High rates of hard bounces are taken as evidence of spam or other nefarious activities, since the typical reasons for high hard bounces are a) a really old list (implies it has been transacted somehow, therefore not opt-in), or b) enumeration attempts.
b) DKIM correct and signing properly?
c) rDNS is right?
c) the IP block your smtpd is in isn't in any RBLs?
d) You're not sending commercial bulk email?
e) Your mail server identifies itself properly to the other smtpd?
f) Your mail server properly implements TLS1.2 opportunistically?
I run my own mail server and have zero delivery issues to gmail. But I do not run marketing mailing lists, nor bulk communications of the same mail content to many customers, it sounds like you're mass-sending mail.
Tighten your engagement segmentation for Gmail. Repeat every 2 weeks if you don't see improvement. If you get down to 30 day actives with no improvement, the problem is almost certainly elsewhere. Occasionally it can be related to external factors - 3rd party email traffic which might use your domain, say if you buy advertising in other newsletters, bad SEO practices (think 'blackhat' rather than 'mediocre'), or poor customer experience with your business/website.
Hard bounce rates on your initial email are always a little elevated vs known recipients, but if you're above 2-3% that's too high and you should work to fix problems with the signup process.