I work on Microsoft's anti-spam team, AMA!
Several times per year—I can practically guarantee it’ll happen sometime in December, and indeed had to deal with this just five days ago—I end up with a bunch of users whose email notifications stop working because Microsoft have started blocking the entire netrange where my server lives. I don’t have control over other Linode customers, guys! I even wrote extra code to stop sending mail to addresses that start bouncing specifically to avoid blacklisting, so after MS finally processes a blacklist mitigation request, someone also has to go in and re-enable those accounts.
SPF, DKIM, DMARC are all configured; I’ve sent from the same IP address for about a decade; I’ve not once received an email abuse report; mail volume is low (most days, volume does not reach the minimum threshold for SNDS to report data[0]). I’ve never had any other mail provider blacklist my server. SNDS always says everything is OK as I am S3150s. What is even the purpose of SNDS at this point when it lies about what is going on?
[0] P.S. The janky SNDS calendar widget resets the month to the current month every time you click on a date, even if the date being viewed is in a previous month. I don’t have any hope that anyone will ever touch SNDS code again since it was clearly designed in the early 2000s and the copyright on the site is now ten years old, but this is a pretty silly bug.
About the calendar widget thing… man am I glad I our team doesn’t own that. No one ever touches legacy stuff cause they’re afraid it’ll break or no one will update but the trick is to file it as an accessibility bug since that gets someone to actually prioritize it since it shows up in reports that the execs read. But dude good luck getting that off the backlog, the one engineer we have who is good at UX stuff (i.e, can code with both quality and velocity instead of just one) has her hands full as is.
Thanks and good luck!
[0] https://www.linode.com/community/questions/22287/550-57511-a...
Got a link or cite for that paper? It sounds interesting.
You do have control over being a Linode customer though. If Linode isn't doing enough to prevent abuse, they deserve to be blocked.
Realistically, what can they do here? Make servers unaffordable to discourage abuse? Give most servers "Internet*" access where some ports are missing?
Disallow SMTP traffic unless an account has a certain reputation or verified identity related to it?
I mean, they don't have to do that, and I would agree the government shouldn't force it to happen. But if someone is constantly causing you problems you shouldn't be required to deal with their shit. If you don't want to behave, expect consequences from everyone else in society.
If every time my friends invited me over I brought over another random person that smears feces all over the walls and pees in the corner I probably won't get invited over very often. Linode (and other cheap VPS hosts) are that person constantly enabling abusive people and subjecting them to others.
Dumb question, but wtf is the solution even? I’m confused about what you expect us to do. I haven’t thought about the problem much so I might be missing some obvious Pareto improvement.
My main suggestion here is just that an established reputation for a specific IP/domain should override the reputation of the IP block it's inside of.
Thanks for clarifying, ok, hmm… that seems hard to do if you can check the IP block by using a subnet mask but the specific IP isn’t resolved until later in mailflow. It might not actually work like that in… ProtocolFilterHub? I always get this mixed up, wait… I think this might be something that we are already working on. And have been working on for a while, wow. Looks hairy. It’s stuck since the guy working on it transferred to another team, and no one picked it up, but some PM noticed before I did and put it up for vote in semester planning. Always creepy to see engineers get referred to as “resources”.
90% of the spam that I receive from a DMARC-validated sender comes from Google; should every Gmail user be punished because Google aren’t “doing enough”?
[0] Linode twice threatened to shut off services within 24 hours due to some vigilante scanning the internet with a broken virus scanner and automatically sending reports: https://virtuallyfun.com/2014/04/23/dumbass-of-the-year-awar... (n.b. this is not my site)
One instance of them supposedly responding quickly to an email abuse report isn't showing they're consistently responsive to abuse reports. I don't know if they are or are not. I don't even know that this blog post even refers to Linode, they're not mentioned once.
And its not true they always block outgoing SMTP by default. Loads of old accounts do not have SMTP blocked. New accounts since 2019 sometimes have it blocked, but given the last few times I've made an account and didn't have any blocks it doesn't seem that often. Maybe I just got lucky though.
And don't get me wrong here, I'm not intentionally singling out Linode here. There's loads of cheap VPS providers that enable this kind of abuse. They're not necessarily better or worse in this regard to many others.
> 90% of the spam that I receive from a DMARC-validated sender comes from Google; should every Gmail user be punished because Google aren’t “doing enough”?
Yes. Just like those telephone companies originating most of the spam phone calls should get disconnected. If they're going to enable abusers, they should get cut off.
a) if a mail sever looks like it’s gonna send spam, then you gotta block it. I personally have philosophical hang ups about this, like it’d be wrong to sentence someone to prison for crimes they didn’t commit just because a system added up some points and made a prediction with high confidence, but in real life, you absolutely need to be proactive. b) there is literally no way to do this that wont immediately get abused. Trust me we’ve tried. We make it nearly impossible to get unlocked on purpose because if it was easy, then it’d be like 1 innocent person using it and 99 attackers due to the adversarial incentive structures.
Now ofc there’s more nuance here, we really do want to get it wrong less often, and you do pay us so it’s not fair to blame it all on the bad guys, so I’m grateful for the feedback but I think you should give me even more detailed feedback since there’s not much I can do except give a vague high level explanation unless you help me by being specific.
> made a prediction with high confidence
Do you somehow track the amount of false positives these predictions generate? How do you tune the prediction to not generate too many false positives?
> but in real life, you absolutely need to be proactive
Why is Microsoft the only provider who needs to do such proactive blocking? Why don't you need to do that for email addresses associated with Office 365?
> I think you should give me even more detailed feedback since there’s not much I can do except give a vague high level explanation unless you help me by being specific.
My story is very much the same as for everybody else having the same trouble, including the person whose blog post sparked this discussion: A root server for personal use located in the data center of a mid-sized hoster, running a mail server as part of its duties. In my case the whole mail setup runs on IP-addresses separate from everything else. Mail volume to Microsoft would probably be on average 1-2 emails per month. No issues whatsoever getting emails delivered to other mail providers, only to Microsoft. This whole setup is in place since several years.
They're not.
I get it, you're afraid that some VPS from a cheap cloud provider suddenly floods the inboxes of thousands of Outlook.com customers. I realize that a fresh IP that sends dozens of emails out of the blue has to be blacklisted.
But why don't you allow my VPS to send, say, 16 emails a day to Outlook.com inboxes? And if ⅛ of the recipients report junk, I get blacklisted. But if all 16 recipients are happy, my IP can now send 16+16=32 emails/day for the next few months (as long as the non-ISP hostname matches; otherwise, it might be a new VPS customer), and so on.
This way, your customers are happy (I don't think spammers rent/hack a fresh VPS in order to send 16 emails, and I don't think they are very good at building up IP reputation), and I'm happy (my personal VPS can send a few emails to my Outlook.com contacts every few weeks/months, and my project VPS can gradually build up and maintain the reputation it needs).
I'm obviously being naive about that approach, but I don't remember having trouble reaching Gmail inboxes or those of local providers, and at least for Gmail, I know that they have pretty effective spam filtering too, so I reckon that they use some approach like the one I described.
For a side project, I have just given up contacting olcsupport and instruct Postfix to send through our @outlook.com address instead, but that is a wobbly workaround at best. For personal email, I now relay through SMTP2GO because GDPR doesn't matter that much, but it makes me sad to have that gaping hole (called Outlook.com) in my decentralized email fantasy, after having spent so much time researching, configuring, diagnosing.
There are plenty of those who do have such issues with Gmail.
The simple reason behind all this is that spammers also have near endless patience. If it takes sending 15 emails per day per IP, they'll do it. If it's a criteria you can figure out as a legitimate user, the spammer can as well. They'll "subtract one" and bypass it.
So the end result is that there's intentional fog over the methods. Just things you can try and get right and maybe that's sufficient. Eventually the good side tends to prevail, with some effort. Other than that it's one of the hardest problems out there with insane weight on both sides.
Why? Why can Microsoft not learn that an IP has been healthy and spam-free for 10+ years and only bother me when there is actual spam is being sent?
…I think this is just a systemic issue beyond my ability to comprehend, let alone solve, and— I hope I’m wrong about this but honestly when I look ahead it seems the future is only going to get worse for people like you. Which I wish I could phrase in a way that was more kind and respectful, it’s not what anyone wants, these unthinking scars inflicted on email as a medium.
But what I can do is make sure that it’s not worse for you, specifically. If I was perfect I’d attack this rot at its core, but I’m not, so I’ll just solve the problem in front of me even though I know it doesn’t scale and hope God forgives me. Get in touch with me directly and I’ll figure out how to make sure you don’t have to jump through those hurdles again.
Outgoing email volume is a handful a week, zero automation ever, and I must have spent dozens if not in the low hundreds of hours over the years on e-mail deliverability to Microsoft alone until finally giving up. Not comparable to anywhere/anyone else.
Just to say, behind every single false-positive is a story like mine and TonyTrapp. Missing out on a group tour with the local club. An old lost friend or family member not being able to get back in touch. Missed recruitment opportunities. A lawyer not receiving a time-sensitive follow-up.
Regardless, there are always better options than silently discarding the whole email: delete attachments, erase everything that looks like a URL, even erase the whole message body, but please tell the recipient that you accepted an email and from whom.
What's the rationale there?
Eg. Known bad domains, known bad IP addresses, incorrectly setup DKIM / SPF, no reverse DNS, non-matching reverse DNS, and that's before even looking at content to determine whether spam.
My hot take is that this prolly won’t last because every org descends to doing a creepy level of data collection eventually so I have a textbook on privacy preserving ML downloaded for when we join the “surveillance but we found a way to make it technically legal” squad. We haven’t done that yet though.
What do you mean by tiers, exactly?
By tiers (which may be the wrong word, maybe just 'layers'), only relating to my setup, I mean things like:
- Tier 1: Spamhaus DROP and eDROP lists are outright blocked
- Tier 2: IP addresses that have illegitimately connected to my mail server ports are outright blocked (port scans, invalid login attempts, etc. - I manually check some of these against abuseipdb.com to determine their validity)
- Tier 3: IP addresses that have scanned non-open ports on my systems are outright blocked from connecting to my mail server ports
Just running these rules for a couple of months has dropped unwanted connections to my mail server ports a heavy percentage. One theory being that if you can block known-bad and highly-likely-bad connections, then actual spam detection (through email content review) is minimised to a certain degree.
I actually want to implement additional anti-spam IP address block lists and just haven't gotten around to it yet, but the above does a good enough job for my essentially unknown domains (as I said, a universe of difference to what Microsoft has to deal with)
- Tier 4: Black-box spam detection built-in to the all-in-one mail server solution I use (I don't know how it works, I don't know how to edit the 'rules' or even if I can).
'Tiers' I would expect Microsoft to have would be:
- Their own lists of known-bad IP addresses / ranges / ASNs
- Reverse DNS lookup validation
- DKIM checks
- SPF checks
- More protocol level 'things' beyond the understanding of a simple network admin such as myself.
- Weighting the results of all of the above to determine some kind of 'spam likelihood' score.
All of this is before reviewing the content of the actual message.
Does MS ignore IP reputation in cases where the domain has a good reputation?
How would you go about getting a new domain and an IP address from a public cloud provider working consistently?
I've had issues with outlook when it comes to new domains and IPs, but after some time it works. I do however usually have more email than a personal server so what's the best way - if such a thing exists - for a personal server that has much lower volume of mail to be trusted?
There isn’t a quick way, by design. You need to wait a minimum period and meet some predicates, and the organized scammers already know what the period is via empirical testing but I’m not comfortable disclosing details of those predicates for disorganized scammers to use. More so because I’d definitely get into trouble for it than due to any belief in security via obscurity. Cushy job makes you risk averse.
Since I can’t share any of the tricks, some general advice— the main thing that matters is a long track record of good behavior. You can end up in a vicious cycle where you fight the system when it punishes you and then it doubles down on the beatings— this is bizarre and kafkaesque and happens all the time. What you want is for there to be two-way communication, if it’s unbalanced with traffic being broadcast but no one engaging with it, that’s going to be cracked down on sooner than if recipients reply.