Hacker News Reader

Comment by bhauer | Hacker News Reader

The annoying thing about the major email services (Gmail, Outlook, et al) using reputation as a key input in their spam prevention algorithms is that they don't handle reputation in a way sympathetic to small and hobbyist mail server operators at all.

Google and Microsoft have the computation and data storage resources to forever record how much spam they've received from the mail servers operated by any given small company or hobbyist. If a domain's authoritative mail servers have a history of never sending spam, when that company or hobbyist (for any reason) has to switch hosting providers and gets a new IP address, Google/Microsoft should recognize that the domain, and its referenced mail servers, has never been malicious.

But instead, the majors treat a new IP address for a long-established domain as entirely new to the internet and assign it zero, or even negative, reputation. And they don't care to address this because, well, why care about hobbyists and the rare small company insolent enough to try to self-host email?

View on HN

This seems like a major legal liability for Google. It could be shown that Google and other major email hosting providers act like a cartel by unfairly discriminating against companies who don't use a major email provider.

I hope so! I am tired of explaining people that it's actually Google's fault they didn't receive my mail. I'll be happy when they pay dearly for the disservice they do to e-mail.

Are you using S/MIME certificates?

Why are you asking? Is there a relation between spam handling and usage of S/MIME?

Last time I looked into it (I run a mailserver and mailman list for one of my hobby groups), S/MIME wouldn't change your "spamminess" reputation score.

DKIM, DMARC, and SPF do, though, and basically are table stakes if you want your mail (especially mailinglist messages) to go through to people at major providers.

No, would that help? I am however using SPF and one of DKIM / DMARC (I forget which). But anyway, I can live with a mail missing here and there. It is just annoying and isn't right.

It's not really Google's fault to be honest. The need to warm up new IP's has existed for a while and a lot of providers do it. Any postmaster with experience knows how and why it's done.

The root comment literally explains that that's BS.

Every post master knows that it is done. But it doesn't have to be that way, although it certainly can feel like it when a company like google decides to not budge on the matter.

> But it doesn't have to be that way

Sure, if you want spam. Don't like it, get people to deploy DKIM, then the domains will be used for reputation purposes.

What the root comment says is BS, the industry uses these methods for a very real and practical reasons.

No, it can be a different way and still not have spam, by trusting/tracking domains instead of IP addresses.

I did write how using domains for reputation purposes instead could happen. The second sentence.

DKIM in no way helps to get past the cartels' "reputation" filters. I send maybe one email every few months to microsoft accounts & it's always received as spam. My server setup & ip have been solid for a decade. It's only ever the globalist providers that block me. Google is 50/50 I get through. Everyone else (eg Protonmail) is no problem.

It absolutely does. Also that was an "if, then potentially" sentence about reputation tracking in the future.

In your case, it's likely that your volume and sending patterns aren't consistent and trustworthy enough to keep track of your domain and IP reputation.

You have to understand that they get millions of letters from new domains each day, sent from compromised Wordpress blogs and the alike. If you want to be deliverable, you have to be consistent and not suspicious.

Or, more likely, there's some other mistake in your configuration somewhere.

You also wrote:

    The need to warm up new IP's has existed for a while and a lot of providers do it. Any postmaster with experience knows how and why it's done.

I think the point people are trying to make, and I'm sympathetic to, is that if an ultra-low volume email poster, with a full-set of SPF DKIM and DMARC credentials configured and zero history of sending spam - that the majors (Yahoo/Google/Microsoft) could start off by not sending email from that domain immediately to spam, just because it isn't a well established and trusted IP address.

Alternatively - come up with something akin to D&B registration system so people can attest that they won't engage in spammy behavior.

> I think the point people are trying to make [...] not sending email from that domain immediately to spam, just because it isn't a well established and trusted IP address.

Yes, and I'm saying what's the prequisite for that to happen. As long as it's okay (which it currently is) to send unsigned mail, IP addresses have larger weight. DKIM needs more deployment for that to change.

There's absolutely no way that IP-based reputation schemes will be deprecated before alternatives are viable. Sure it would be nice for a few people here, but no, won't happen before the ecosystem improves.

> Alternatively - come up with something akin to D&B registration system so people can attest that they won't engage in spammy behavior.

Already exists. That too gets abused.

Oh - that is interesting - would you mind sharing what the registration system is?

It's cheaper and easier to munch through lots of throwaway domains than to keep moving IP neighbourhoods, isn't it? I don't know - is free domain tasting still a thing?

If you filter by IP block (or address!), it might be a block that has changed hands and is no longer spammy. Or it might be a block from the Zen Policy Blocklist, which blocks ranges that the responsible ISP has submitted as domestic or retail blocks that are supposed to send outbound mail through the provider's smarthost.

If you filter by domain, that could be the envelope sender, the From:, the Reply-to:, or the domain of the SMTP client. Only the last is reliable; and you also have the IP address for the client. In my experience, the IP address is more useful, for longer, than the domain name. But any good blocklist should age quickly (i.e. old stuff should drop off the list).

> It's cheaper and easier to munch through lots of throwaway domains than to keep moving IP neighbourhoods, isn't it?

Depends on your approach. If you hack IoT devices then you have a lot of IP's. If you hack Joomla sites, you have a bunch of domains.

> I don't know - is free domain tasting still a thing?

Yes. There are also discounts and stuff like that.

Even if you have DKIM, SPF and DMARC all set up, at least Microsoft still seems to give a decent weight to IP reputation and assign a negative reputation to unknown/low use IPs.

Absolutely, my second sentence says how that could change.

I would have thought that it's fairly self-explanatory that anti-spam measures utilize the strongest signals. If sender domain becomes that, it will get more weight.

So if in the future email providers could reject both SPF-less domains and DKIM-unsigned letters, IP's would definitely become less relevant. So, get people to deploy those things.

Sorry, but this is bullshit. If I employ relevant techniques to protect my domain and the protection works, and I am not sending spam (which I am not), then Google (no idea what MS does) should not care about the IP I am sending my e-mail through. I have proved that the e-mail is tied to my domain and they know that my domain is not spamming - what more do they want?

Even better - Google could help small mail server admins by actually providing the information that landed their e-mail in the spam folder. If the protection is tied to the domain, no spammer will be helped by this knowledge.

And I understand that maybe a new domain might be suspicious at first, but after a few years of unchanged ownership (backed by whois data) there is simply no reason to put any mail messages from these domain to spam. Whatever the IP is.

Stop making up excuses for them. They are negligent at best, malicious at worst. Can't wait till they get hit by a lawsuit over this.

> If I employ relevant techniques to protect my domain and the protection works, and I am not sending spam (which I am not), then Google (no idea what MS does) should not care about the IP I am sending my e-mail through.

They definitely take the IP less into account if other things are more trustworthy. Totally ignoring it would be short-sighted from them. There are many cases where the domain is fine but looking at the IP and its usage patterns helps prevent abuse. Be it misconfigured (and then abused) SPF, stolen DKIM keys, public website that's email-capable getting compromised, these things happen a lot.

> Even better - Google could help small mail server admins by actually providing the information that landed their e-mail in the spam folder. If the protection is tied to the domain, no spammer will be helped by this knowledge.

They have a significant amount of content-based filtering, knowing that helps spammer reword their crap and bypass those.

> I have proved that the e-mail is tied to my domain and they know that my domain is not spamming - what more do they want?

That's also part of the thing, you can't prove and enforce this for both envelope and header from at the same time. Not to mention how minuscule the amount of perfect and strict SPF+DKIM+DMARC is out in the wild. At this point in time IP's are a very strong signal.

> Stop making up excuses for them. They are negligent at best, malicious at worst.

I haven't made a single excuse, I'm explaining why things work the way they do. You calling it bullshit won't make it so.

Sure, then they relax things, spam explodes, and people are going to complain that they're not doing enough to fight spam or actively enabling it.

Google has more than adequate capabilities to handle spam, as several comments point out above. Also, Hashcash[1], which is a virtual silver bullet for email spam, especially when combined with existing trust mechanisms (i.e. the less reputation you have, the harder the Hashcash challenge).

The problem is not that the level of strictness that Google is applying is necessary to reduce spam; the problem is that Google doesn't care and is in fact incentivized to not accept email from non-Google domains.

[1] https://en.wikipedia.org/wiki/Hashcash

> Google has more than adequate capabilities to handle spam, as several comments point out above.

Yes, and sender reputation is part of it :D that's why they can handle it.

> Also, Hashcash[1], which is a virtual silver bullet for email spam, especially when combined with existing trust mechanisms

Who do you think has to and can spend more compute or money, well-trusted Google or a small player? Far from a silver bullet.

> i.e. the less reputation you have, the harder the Hashcash challenge

Who do you think will have better reputation? I won't even start at how different people define spam. Though, I can promise you that on average SpamChimp would get to send a lot of spam for much less "HashCash" than a small player.

> is in fact incentivized to not accept email from non-Google domains.

Everyone is.

The thing with hashcash is that a small player doesn't need much compute power, cause they aren't sending a million emails a minute.

Oh, so it's ineffectual against spam, got it.

Either botnet devices can send tens if not hundreds of letters, or it's going to be slow and/or expensive for all the legitimate small senders. You really can't have both.

Let me try to explain. If I'm a legitimate small sender of email, then Gmail requiring that my email server solves a cryptographic challenge (Hashcash or a more modern version of it) that takes a minute to compute isn't a big deal if I only send 5 emails per day. All of my outgoing emails are going to be delayed by a minute which won't affect me at all, even though it's computationally slow.

On the other hand, spammers are likely sending hundreds of emails per minute per server, so limiting them to 1 email per minute would have significant consequences for them.

I hope this demonstrates the disproportionate effect Hashcash (or modern equivalent) would have on spammers when compared to legitimate senders?

We use exactly the same idea for secure password storage. Password verification is slow, really slow, on purpose. This makes it "slow for legitimate password verifier" (the website you're logging into), but it also makes password cracking on a large scale completely infeasible.

> I hope this demonstrates the disproportionate effect Hashcash (or modern equivalent) would have on spammers when compared to legitimate senders?

Yes, in a scenario where you're a legitimate sender that sends very few letters compared to a bulk spammer. Apples to oranges.

You unfortunately ignored both the low-rate spammers and the high-rate legitimate senders. The former is barely hindered and the latter is PoW-walled into inconvinience.

If you now bring in reputation schemes, then those can be played by adversaries, you'll end up at step zero but you have made a large climate impact.

> but it also makes password cracking on a large scale completely infeasible.

But this is not the current scenario. Intentionally complex password hashing is not really PoW we're talking about.

> You unfortunately ignored both the low-rate spammers and the high-rate legitimate senders.

Both of these seem like imaginary edge cases that don't really exist. I'm going to gloss over the oxymoron of low-rate spammers.

> If you now bring in reputation schemes, then those can be played by adversaries, you'll end up at step zero but you have made a large climate impact.

How do they do that? If we focus on the current state of things, high rate legitimate senders already have an established reputation so they don't need to change how they handle outgoing emails.

The only thing that would change is how large email providers such as Gmail handle incoming emails from small/unknown senders, by accepting a sufficiently complex PoW as an indication that the email is not spam. Over time this should increase the sender's reputation to a point where PoW is no longer required.

TLDR: Nothing would change for the big players, small senders are given an opportunity to prove they are legitimate, and spammers remain locked out.

> Both of these seem like imaginary edge cases that don't really exist. I'm going to gloss over the oxymoron of low-rate spammers.

I'm sorry? Both of those cases are very real and happen often, you don't get to just ignore them because of your lack of experience.

> How do they do that?

Many ways. Sending legitimate mail for example. You're acting like it's hard to show good behaviour for a while. It's one of the telltale ways I catch those fucks, they're doing it already.

But they also use their own inboxes to build up volume, hijack accounts, abuse SPF policies and DKIM misconfigurations. There probably are more ways to get a "trusted" domain I can't recall this instant.

Things regular senders can do, spammers can do, but they'll spend more effort to do it at scale!

You have demonstrated repeatedly how you have little knowledge of spammers' and others' behaviour, you handwave away all "edge cases" and say it will work. No, it won't.

All you'd add is wasted compute resource and time. Compute and time that especially spammers get for free by abusing others.

> Both of those cases are very real and happen often

"Often" but as a tiny fraction of all sender cases, something that anyone who's actually worked with email spam would know, so you either dishonestly withheld that fact or you don't know what you're talking about.

Also, low-volume spammers are quite clearly an oxymoron - the definition of "spam" includes high volume. It's literally high-volume by definition: "unsolicited usually commercial messages (such as emails, text messages, or Internet postings) sent to a large number of recipients or posted in a large number of places"[1]

And, the vast majority of individuals do not send high volumes of email. This is a fact. It is an edge case.

> because of your lack of experience

You seem to be making a lot of arguments from authority, yet you don't even have the credentials to back them up.

> You're acting like it's hard to show good behaviour for a while.

They're definitely not. It's crystal clear that the usefulness of Hashcash is not dependent on it being hard to show good behavior for a while, and in fact its most effective in the long term.

> But they also use their own inboxes to build up volume

Something that Hashcash solves, because they'll have to spend a lot of cycles solving challenges to send non-spam email to build up reputation repeatedly, because for each "clean" IP/server they'll have to repeat the "reputation ramp-up" that every legitimate sender only has to do once.

> hijack accounts

Literally no anti-spam situation protects against this, so it's irrelevant, and I don't know why you're bringing it up.

> abuse SPF policies and DKIM misconfigurations

Then these need to be fixed. Google indiscriminately blocking smaller senders isn't a fix.

> Things regular senders can do, spammers can do, but they'll spend more effort to do it at scale!

Exactly - that's what Hashcash does, is force spammers to spend more effort on the things that regular people do, because spammers do it at scale far more often than regular people do, and so it introduces a disproportionate cost on them.

> All you'd add is wasted compute resource and time. Compute and time that especially spammers get for free by abusing others.

This conclusively illustrates that you have no idea what you're talking about. Spammers do not get resources for free - they have to either spend time to acquire botnets and hijacked domains themselves, or pay money for someone to do it for them. This is pretty clear to even people who aren't in the industry.

Spamming only happens if its profitable, and the reason why its still profitable is because the cost of sending a spam email is currently low enough. Hashcash directly increases the cost of spending spam email by making it compute-bound on top of needing to acquire domains, IP addresses, and machines, and spending time establishing a good sender record, while (if combined with a reputation system) barely penalizing normal users in the long run.

Let me state it again: every new sender has to go through a reputation ramp-up process, but spammers are uniquely penalized by Hashcash because they have to repeatedly because after they start sending spam mail, they lose their reputation and have to start that process again. Normal users do not have to do this - they start a new server, solve a bunch of challenges, and then they don't have to do that again because they don't regularly burn their acquired reputation.

The fact that you can't actually counter these arguments about Hashcash sounds almost like a spammer who's worried about the deployment of an effective system based on Hashcash.

[1] https://www.merriam-webster.com/dictionary/spam

> "Often" but as a tiny fraction of all sender cases

Not really, no. I mentioned and you're dismissing high-rate legitimate senders and low-rate spammers. The former happens for example when companies send bills each month, they hit very high rates compared to their usual baseline. The latter is not the majority volume-wise, but is the most annoying - fairly logical that some spammers choose to fly under the radar and drop a letter or two in a minute. Do that during the night and you've landed a lot of letters in people's inboxes before you get reported.

> And, the vast majority of individuals do not send high volumes of email. This is a fact. It is an edge case.

It may be an edge case for you, but it must be accounted for. You can't wave it away because then people won't receive their mail. Just one minor example, some less tech-savy users emulate mailing lists with hundreds of people in CC. You can't say there that "oh this is my anti-spam solution but it doesn't account for you because you're an edge case"

> Also, low-volume spammers are quite clearly an oxymoron - the definition of "spam" includes high volume.

There's a difference between total volume and rate. You confusing the two is your problem. Not to mention that a dictionary definition is not the ground truth neither does it define "high volume". I'd also say unwanted unsolicited advertisement sent to 30 people is still spam, but you do you.

> You seem to be making a lot of arguments from authority, yet you don't even have the credentials to back them up.

From experience backed up with explanations, and you're dismissing them based on yours with no explanation.

> Literally no anti-spam situation protects against this, so it's irrelevant, and I don't know why you're bringing it up.

Incorrect. Looking at IP addresses suddenly behaving unusual is a great signal compared to the aggregate of a domain's total.

> Then these need to be fixed. Google indiscriminately blocking smaller senders isn't a fix.

They absolutely should be, but it's difficult and in the end you're not alone on the internet. Until people will, IP's continue to be used for spam classification.

And no, it's not indiscriminate.

> Exactly - that's what Hashcash does [...]

I think you misunderstood again and let me rephrase. Spammers spend more effort to lower effort spent per unit of spam. So in the end they spend less effort per letter than legitimate small senders.

> This conclusively illustrates that you have no idea what you're talking about. Spammers do not get resources for free

Your sentence conclusively illustrates you're clutching at straws. Declaring that something stolen hasn't been gotten for free is a shaky foundation to build your argument upon and pedantry at worst. The cost to acquire those resources is there anyways, some extra CPU they have to steal is not that significant.

> Hashcash directly increases the cost of spending spam email by making it compute-bound on top of needing to acquire domains, IP addresses, and machines. > Normal users don't have to do that again because they don't regularly burn their acquired reputation.

You're going in a loop again. We're comparing relative effort/monetary cost per email here. If one side has lowered the cost of effort by parallelizing things it's unfair to the side that hasn't. Consider the scenario where a new online store owner starts sending email or someone's script is compromising yet another poorly secured blog, IoT device or SMTP account.

Now a legitimate sender has built up all the reputation, after a few flags by recipients it was spent in an instant. Everything will become slow and expensive for the actual owner, once again. For spammers it was the tenth one that day.

Such a massive extra cost for the spammers. /s

That was just one example. Again it all boils down to the fact that regular users can't parallelize their actions in the same way spammers can - PoW would make it *relatively* more expensive (time+effort+money) to legitimate users per email.

> The fact that you can't actually counter these arguments about Hashcash sounds almost like a spammer who's worried about the deployment of an effective system based on Hashcash.

So many ad hominems, so little substance.

I read this entire tree of comments and I really don't understand why you continue to engage in this condescending manner. If you're some kind of subject matter expert on the topic then by all means, please elaborate on the intricate details and challenges of dealing with spam, from start to finish, as well as any proposed solutions to make self-hosting viable when dealing with providers such as Gmail.

These patronizing "you obviously know less than I do" type comments that occasionally drop a few bits information aren't adding anything to the conversation.

> I read this entire tree of comments and I really don't understand why you continue to engage in this condescending manner.

If you haven't noticed, I heavily copy phrases from the people I reply to. If you don't like your tone being matched, unfortunate.

> Please elaborate on the intricate details and challenges of dealing with spam, from start to finish, as well as any proposed solutions to make self-hosting viable when dealing with providers such as Gmail.

I have, and you included have ignored it as "edge cases" and alike. What can I more explain when someone has taken the stance of being dismissive?

Hashcash is an interesting idea and very creative. But I do think you're right. It may impose reasonable limits on a single computer. But if an attacker has a botnet they can still send out a crap load. Botnets and spam practically go hand-in-hand as-is, so I don't see how hashcash helps there.

Still... it is a creative idea.

> Who do you think has to and can spend more compute or money, well-trusted Google or a small player? Far from a silver bullet.

i'm not even a big fan of hashcash, but i don't think you understand it, or how it could fit in as a component of a broader spam mitigation system. (for better or worse)

google wouldn't need to compute hashcash for their outgoing emails, because they have DKIM and a solid reputation. nor is its computational power somehow at odds with that of smaller players.

> i don't think you understand it, or how it could fit in as a component of a broader spam mitigation system.

It's not that I don't understand, I know how it won't.

> google wouldn't need to compute hashcash for their outgoing emails, because they have DKIM and a solid reputation.

Great, so what's the point? Small players would have to pay Google to deliver mail :D

> computational power somehow at odds with that of smaller players.

As I said in my other comment, either botnet devices can send tens if not hundreds of spam letters, or it's going to be slow and/or expensive for all the legitimate small senders. You really can't have both.

> It's not that I don't understand, I know how it won't.

You really don't understand it, because you're unable to provide counterarguments to specific explanations of how it would fit in with a larger anti-spam system, and because you say things that are clearly false to someone who does understand it.

> Great, so what's the point? Small players would have to pay Google to deliver mail :D

...such as this, which is false. Nobody's paying anybody anything - Hashcash is a computational proof-of-work challenge with zero money transferred.

If you're talking about a metaphorical "payment" of the smaller players having to do some kind of work - that's a feature, not a bug, because it allows the smaller players to invest effort into convincing Google that they're legitimate, and if coupled with a well-designed spam system that then takes that reputation and associates it with DKIM signatures/IP addresses, then the smaller players burn n cycles and then stop having to do so because they now have reputation with Google, while spammers burn n cycles continually for every message that they send because they keep getting flagged as spam.

> As I said in my other comment, either botnet devices can send tens if not hundreds of spam letters, or it's going to be slow and/or expensive for all the legitimate small senders. You really can't have both.

Yes, you really can, because you don't understand how anti-spam systems work. A system with Hashcash would start out assigning most IP addresses+hostnames/DKIM signatures with a reputation of 0, and in order to accept a message, would require the sender solve a Hashcash challenge inversely proportional to their reputation. If the message is marked as spam, the reputation takes a hit - if it's not, the reputation increases slightly (along with all of the other factors that an anti-spam system uses).

The legitimate servers quickly build up reputation and stop having to solve challenges, while the spammers pay the computational tax until the end of time, making it unprofitable for them to send most spam to most targets. The end.

> because you're unable to provide counterarguments to specific explanations of how it would fit in

I have provided specific cases where the previously proposed use-cases wouldn't work.

> they now have reputation with Google, while spammers burn n cycles continually for every message that they send because they keep getting flagged as spam.

> The legitimate servers quickly build up reputation and stop having to solve challenges

So the spammers send a few warmup mails. It's very naive of you to think spammers can't imitate legit sender behavior.

It's also clear you haven't seen or analyzed any significant amount of spam. The end.

> So the spammers send a few warmup mails. It's very naive of you to think spammers can't imitate legit sender behavior.

It's clear to anyone who understands the basics of spam and Hashcash, and can use basic logic, that "So the spammers send a few warmup mails" literally doesn't change the effectiveness of Hashcash.

Let me state it again: every new sender has to go through a reputation ramp-up process, but spammers are uniquely penalized by Hashcash because they have to repeatedly because after they start sending spam mail, they lose their reputation and have to start that process again. Normal users do not have to do this - they start a new server, solve a bunch of challenges, and then they don't have to do that again because they don't regularly burn their acquired reputation.

You say

> It's also clear you haven't seen or analyzed any significant amount of spam. The end.

...but you're unable to use basic logic on publicly-known facts (the vast majority of spammers send large amounts of spam; the vast majority of users send small amounts of email; reputation is hard to gain and easy to lose) to infer the above or generate a logically cohesive counterargument. Every time you can't give an answer, you fallback to an argument to authority.

> Yes, and sender reputation is part of it :D that's why they can handle it.

They aren't doing a very good job of using sender reputation. If you have a domain with a several year record of no spamming whatsoever, and with every outgoing message using DKIM, they will still start blocking you if other senders who just happen to have an IP address close to yours start spamming.

IP block reputation should only be used to set the default when dealing with new senders. If they have seen enough DKIM signed messages from a sender to know that the sender has a good reputation, IP block reputation should have weight 0 when receiving mail from that sender.

Google could do this without any increase whatsoever in the amount of spam that shows up in their customers' inboxes. All that would change is that their false positive rate would go down.

> They aren't doing a very good job of using sender reputation.

That's your limited perspecive, sorry.

> they will still start blocking you if other senders who just happen to have an IP address close to yours start spamming.

Absolutely, how would they know how that provider assigns those IP's? A lot of spammers use entire /24's.

> If they have seen enough DKIM signed messages from a sender to know that the sender has a good reputation

A lot of spammers have DKIM, it's not a good reason to allow mail from a suspicious subnet.

Pick a provider that deals with their spam complaints. That's the harsh truth.

> A lot of spammers have DKIM, it's not a good reason to allow mail from a suspicious subnet.

You keep missing the point. Nobody is suggesting that mail be allowed merely because a domain is using DKIM.

What is being suggested is that if a domain has a proven history of not sending spam then blocking the domain's mail just because there are spammers in the same subnet results in blocking non-spam. Not blocking the domain does not increase the amount of spam that gets through because the domain is not sending spam.

Where DKIM comes in is that it allows receiving sites to distinguish between mail that is really from domain X and mail that is forged to appear to come from X. The latter can safely be automatically classified as spam. The former can be used to build an accurate history for the domain to determine if it sends spam or not.

> merely because a domain is using DKIM.

> What is being suggested is that if a domain has a proven history of not sending spam

That is merely because of using DKIM! Everyone can create a 'history of not sending spam', including spammers.

With your solution someone can rent a subnet, warm up a bunch of domains, but then blast spam from the entire subnet without immidiately affecting the entire set of spam domains.

What you're proposing wouldn't allow trusting even signed mail from an already suspicious subnet. Be the domain with or without reputation, so are the spammers'.

> Not blocking the domain does not increase the amount of spam that gets through because the domain is not sending spam.

They don't know that before they deliver your mail to inboxes and someone does or doesn't mark it as spam.

> Yes, and sender reputation is part of it

Sender reputation where you block smaller email providers indiscriminately without weighting DKIM signatures or identity (or just blocking based on IP) is not a way of "handling" it. Your comment is akin to someone saying "the government can handle poverty" and you saying "yes, by killing all of the poor people" - it's very clearly a non-solution.

> Who do you think has to and can spend more compute or money, well-trusted Google or a small player?

I don't think that you understand how Hashcash is supposed to work, or what it's supposed to solve. The computational cost is imposed on non-Google players - both smaller email servers without a reputation (until they build one up, obviously), and spammers. I mean, sure, if everyone implemented the protocol, then non-Gmail servers could technically request that Gmail do Hashcash challenges in order to receive email, but nobody's going to do that, because nobody believes that Google is going to send spam mail.

> Though, I can promise you that on average SpamChimp would get to send a lot of spam for much less "HashCash" than a small player.

No, you really can't promise me this, unless you're literally in control of Gmail servers and intentionally adjust the difficulty settings to fulfill this condition.

> > is in fact incentivized to not accept email from non-Google domains.

> Everyone is.

I don't see what your point is here. The fact that Google is incentivized to not accept email from non-Google domains is not a good thing - it's a bad thing. And, really, the above is false - everyone is incentivized to accept legitimate, non-spam email from all of their customers, which is not the same as what you said.

> Sender reputation where you block smaller email providers indiscriminately without weighting DKIM signatures or identity

That's not true.

> because nobody believes that Google is going to send spam mail.

This sentence indicates you don't know much about who sends spam or not.

> The computational cost is imposed on non-Google players

You're going in circles. Do you not see how that especially is unfair to the smaller players? Spammers have just as much compute, it will just impose an additional burden on the legitimate senders.

So in your scenario a small startup has to wait tens of minutes (bunch of letters in the queue already) to send email confirmation letters. The spammer on the other hand does the same, it'll infect a few more devices, they've exeeded the startup's sending rate.

Alternatively, both the spammer and the hypothetical startup build reputation first. They'll warm up the domain, both get to send more in the same timeframe. So... we're at the beginning again, everyone warms up their domains and IP's like they do right now. PoW hasn't improved the sitation.

> I don't see what your point is here.

Because you haven't seen enough spam. On average everyone is incentivised not to accept mail from new or unknown domains.

> everyone is incentivized to accept legitimate, non-spam email from all of their customers

No shit, but also not the topic. Received mail is not perfect, that's what's the actual topic.

They hurt more than competitors. If anyone from gmail is reading and gives a crap about the collateral damage from policies like these, they have set me back at least a few months in my attempts to adopt a child. The agency my wife and I had been getting licensed through dropped us, citing lack of responsiveness in completing paperwork. But I've been e-mailing them for months with questions clarifying how to fill out unclear forms and they've been ignoring me. I presented them with a history of all the sent e-mails they never answered, and even gave my e-mail address in person last time I was there, asking them to add me to an allow-list, but it didn't make any difference.

It's particularly disappointing because even just getting the process started was delayed when no one answered our initial inquiries of interest for nearly a month. That was when I first figured out it was because my e-mails were getting routed to the agencies' spam folders. At this point, I'm probably just going to have to give in and get a gmail address for the sole purpose of completing an adoption, and then stop using it and go back to my real address.

Who'd have thought this would be the thing preventing me from completely de-googling?

My loss isn't as big as this but I've gotten some Paypal disputes because those users didn't get any of the welcome emails (for a rather costly purchase). My issue has been more with Apple's mail for some reason though.

Thankfully all my customers have been super understanding and were kind to take the disputes back once they were able to locate the emails in the Spam folder. Also I finally decided to just use a "reputed" smtp server to fix my problems (feels like extortion but what can you do).

I've written this before, but it's so sad that something so important and vital like email is so broken and we are the mercy of corps like Google/Apple/Microsoft and everyone has their own secret policy.

Hiding messages in a spam folder is essentially ghosting. Email servers should reject messages that they consider spam.

I've run into countless problems because of my messages being marked as spam. The worse is that now it as become socially acceptable to ghost people, I never know who ghosted me, the mail server or the person! I've run into "I ignore your email, get the hint" a few times when thinking I was marked as spam and tried to contact people through other means.

Too many false positives to reject everything they consider spam.

Right, and this is exactly why they need to reject. At least the sender would know and could try to contact the receiver to let them know Gmail/outlook is blocking them.

Right now, when you send an email, you don't know if it was received or not and if the lack of answer is a conscious decision or because of filtering.

You can blame the spammers for that.

Do you remember the bad old days of spam?

Left unchecked I think it might have destroyed the e-mail ecosystem entirely.

Regarding the block vs reject, that unfortunately gives a big tip to the people whose working hours are spent trying to defeat three blocks and send you more spam.

Yes I remember, but this was a different world, it was a world without reputation networks, without ip black lists, with SPF/DKIM.

There is not reason at all to suspect that a message from a small server with proper SPF and DKIM that has never sent spam (for real or spoofed, see SPF), is spam. The other half is from small servers that sprung over night and have no reputation.

Half of the spam I receive is from google/outlook/amazon. They don't block each other, do they?

I think everyone always looks at it from the perspective of the hobby mailer, when instead gmail looks at it from the recipients point of view.

It’s really really trivial to automate buying a domain, get some IPs and setup DKIM, SPF and DMARC. How can a provider determine intent?

Gmails approach is to mostly use user signals, which in my opinion is the best way.

At a company I’ve worked for, they sent some mail using sendgrid to a poorly made mailing list and we’re in gmail spam for ages.

My personal ran email mostly stays in inbox.

From the recipient's point of view, having incoming mail that is not spam dropped is a fucking problem.

It’s only dropped if the relay/server sending it is not behaving nicely, otherwise it’s there in quarantine (spam) awaiting user signals to train it.

I think it’s such a typical mindset that people with very limited domain knowledge think that people with a lot more domain knowledge are just being stupid.

I don't care for the idea that any private network should be legally required to accept connections (let alone accepting data) from any other network if they see fit not to. No sender should have the power the force another server on another network to accept whatever they spew out to them.

If google is too aggressive about dropping mail and gmail users can't reliably get the email messages they want, those users should move to a platform that doesn't block those messages. Problem solved.

I've seen several articles in recent months about Google (and other providers) blocking too much, but they are nearly always about the senders and the burdens blocking and sender reputation places on them.

As much as I'd agree that senders have to do a lot to get mail accepted maybe this is working as intended more often than not and the fact that people aren't dropping their gmail accounts in droves seems to suggest that the people whose inboxes are directly impacted by Google's filtering are largely happy with how things are.

Most email is spam, and for the few senders who aren't just bitter that they can't force spam on the most popular email providers with impunity, I freely admit that this is an extremely frustrating experience for them and that it does help degrade the utility of email and contributes to the consolidation of useful email providers. The current situation isn't perfect.

The idea that we could send mail from any routable IP address, or send bulk email easily to anywhere, or even set up our own personal SMTP servers on a whim and still send email to any other network in the world with no deliverability issues is beautiful, but not practical.

Given the decades of scammers and advertisers abusing email, I'd much rather leave it up to the recipients to decide when a service has gone too far in blocking what ends up in their inboxes.

FWIW I run a service that sends a decent amount of email to GMail users. While it was a bit slow to get started (messages being marked as spam) once I had sent for a month or two and had a few users that marked the messages as not-spam I don't appear to have any problems. I say this sending from a Digital Ocean IP address that occasionally changes. It appears that Google highly values domain reputation and that once I have got onto their good list I am doing OK.

Disclaimers: I make sure to do everything else right. I have SPF and DKIM and my DMARC policy is to reject 100%. I also don't use IPv6 (DO Kube doesn't really support IPv6 well).

I have found other major providers to be much worse. Microsoft seems to rely almost entirely on IP reputation and marks everything as spam, even accounts that have marked messages as "not spam" many, many times. Apple outright blocks the IP range.

My experience is that all of SPF, DKIM and DMARC are almost completely ignored by Gmail -- one day I simply stopped DKIM and DMARC and Google keep happily accepting emails (and to this day I still send emails without). In fact they will happily accept emails even when the SPF check didn't pass and the policy clearly says strict reject aka -all .

While on the other hand I fully agree with TFA: I have _never_ been able to send an email to Gmail from a IPv6 address and have it not end up as spam, not even to accounts where I already whitelisted previous attempts.

I don't think it's a reputation issue, since my IPv4 addresses likely have much worse reputation. It's as if they just handicap all IPv6 addresses.

GMail definitely respects my DMARC reject policy. But IDK how missing one of SPF or DKIM affect its spam decision. But with DMARC reject and both missing or invalid it will bounce the message every time I have seen.

Automatic mail forwarding without altering the From address will cause DMARC alignment for SPF to break. This is a common enough legitimate setup that most providers seem to effectively downgrade the DMARC policy applied when they see this (usually reject becomes quarantine, quarantine becomes ignore).

> sends a decent amount of email to GMail users

That's why your experience does not apply to the GP.

My experience with digital ocean was, that their IP subnets have a horrible reputation for email. Google is easy, but try to get unblocked from Microsoft (office 365 or hotmail). In the end I switched to another provider.

The only thing I found that seemed to help with Microsoft is for the recipient to find the mail in the spam box and reply to it.

DigitalOcean definitely does have a terrible reputation (for good reason). IIUC they used to let anyone send mail and I don't think their current process of requesting unblocking is too strong of a prevention against spamr either. However ideally this would only hurt "first contact" for a domain assuming it is authenticated. In practice some providers will barely consider domain reputation even after a long history.

There was a study somewhere once that concluded that digital ocean is the biggest single source of spam emails. And that the are responsible for around >20% of spam emails worldwide.

I noticed that some providers completely block digital ocean IPs on their firewalls. So no way to get unblocked.

Back when domains were easy to spoof (I could setup a server and send mail as ycombinator.com easily enough) it made sense to track the IPs, but now that you have DKIM and SPF links to cross-check, you should be able to use the domain reliability as a strong indicator. Sure you would have to catch people buying a "good" domain that expired, but that shouldn't be an insane hurdle.

The real story is nobody cares.

> The real story is nobody cares.

Exactly. Why should the major players care about those too small to matter?

> Sure you would have to catch people buying a "good" domain that expired

Adding to my earlier point, Google and Microsoft are well-enough connected to the domain registrars to know when that scenario has happened as well. If they put any effort into it, they could reliably determine whether a new IP address for an established domain is legitimate or a fraud. But as we've said, why put any effort into it when the only people complaining are not important?

And it's even worse - if we theorize a email competitor appearing out of nowhere to rival Gmail, and people complaining, all that would happen is Google and Microsoft would special-case that provider, and the underlying issue wouldn't be solved.

a class action lawsuit could make them care.

as far as how to solve this problem technically, I think a reputation system based not on domains or ips but on email certificates is the real answer here.

DKIM is certificates, so I'm guessing you're talking about sender certificates?

How would that help? Spammers can get certificates too. Maybe it cuts down on some of the misconfigured http email senders, maybe, but not enough to matter. Scam sites run https these days.

You can't use like age of activity of the cert to help because a) things get compromised, b) you need to rotate your certs frequently anyway.

Shouldn't the authority (or key) used to sign those certs be long lived? The certificates themselves should be rotated frequently, bit not the key used to sign them.

There's only S/MIME, TLS and BIMI+VMC that use actual certificates.

DKIM does not and DKIM keys should be rotated once in a while, but few do.

Being able to tie together letters and senders, knowing who sent what, would help.

It wouldn't help to fully trust, nothing would, it's a human problem, it would help to trust more.

If the email is DKIM signed, it's expected that the sender was authorized to send the message.

Any wide spread certificate program will just have the email address as the identity, and it will be authorized by establishing control of the email (just like the majority of certificates used for https are domain control only, no organizational verification, not that organizational verification means much anyway). Anyway, identity is hard; there are many people with my name, including a Pulitzer winning author.

Using DKIm you don’t actually have the problem that people can buy so-called good domains, because the system works with private and public keys, so not only do you need a good domain, you also need the private key for that specific DKIM in signature

I wonder how many systems actually track DKIM signatures over time, beyond just checking at the moment of email receipt.

Most, check results are usually kept in the final stored letter.

You wouldn't need to.

At the point of receipt, when verifying via DKIM that foobar.com did indeed send this email, then update your spam statistics for foobar.com and you're good.

Isn't the problem more about how to treat brand new domains the first time you encounter them? In order to be friendly to small/new email servers, you would presumably need to initially grant new domains a sufficient reputation for them to send mail reliably. But since domains are essentially unlimited, a bad actor can trivially circumvent your reputation system by spinning up endless domains. This seems like a fairly textbook example of a Sybil attack.

Spinning up endless domains is something that can be detected perfectly well. Very few entities can do it in a way that interferes with other people.

Domains aren't free, they're limited.

Keep in mind that there are a lot of domains out there without SPF records, there's really no lack of domains to abuse.

Not to mention all the websites that get hacked or the uber-cheap registrars.

Indeed, although I think there is a little more rationale than simply not caring. The hobbyist/small mail server is historically more likely to go unpatched and pwnage is more likely to go unnoticed, so it’s unfortunately a good defensive move to penalize them reputationally by default.

PS yahoo is the worst for assuming poor reputation from a sender. In my experience they just introduce massive delivery delays at the drop of a hat even when the sender has a stellar reputation.

> The hobbyist/small mail server is historically more likely to go unpatched and pwnage is more likely to go unnoticed, so it’s unfortunately a good defensive move to penalize them reputationally by default.

But to reiterate my earlier point: Google and Microsoft have the computational and storage capacity to have a detailed history of all domains' authoritative mail servers behavior. They will know whether a domain has a history of patch negligence.

No, I think the far simpler explanation is that they just don't bother tracking reputation by domain. Or if they do, it's largely overshadowed by the weight given to IP-based reputation.

Oh, I’m not saying you’re wrong. They absolutely could put the effort in to solve this problem.

> I think there is a little more rationale than simply not caring.

You're right in one more way: both Google and Microsoft offer email hosting solutions for small businesses. Their main (and, if you've got more than 8-10 accounts or so, the only) selling point is that it makes managing email hassle-free. Making it as painful as possible for small businesses to host their email server helps these services tremendously. If they had real interoperability (either of their own accord or because it were forced upon them through regulatory measures), the biggest cash cows of these services -- companies that are well into "enough accounts that your own server would be much cheaper" territory but not large enough to afford or risk large infrastructure changes -- would evaporate pretty quickly.

they are basing their filters on the reputation of the netblock that your individual ipv4 /32 (or ipv6 equivalent) is contained in. And very often when you are hosting with a low cost VM or VPS provider the historical record of other IPs operated by the same hosting company in the same /24 or something is very poor.

while I concur with all of the points you make, there is a logical and statistically accurate reason for some of these spam filters.

even if your no-open-relay, rdns, spf, dkim, dmarc, SSL/TLS and other configuration is absolutely impeccable on your smtpd, your only recourse in a situation like this is to change to a new ISP that does not have a poor IP space reputation in all the adjacent IPs.

The netblock thing has happened to me. My Linode that I’ve had sending mail for 6 years gets blacklisted now purely for being on the netblock that it’s on.

Yep. I've encountered the exact same issues on multiple Linode servers in multiple data centers. I've also experienced it with an AWS server. It's often almost impossible to get it unblocked because the support teams at the email providers (if you can even contact them) don't realize that's what's happening. I didn't understand it myself until I got a call with an engineer at Microsoft who realized it in the middle of our call. He unlisted us on the call and we were fine for a while, but were eventually blocked again and haven't been able to get unblocked. It's really frustrating and I'm slowly ending all of my email services (both professional and hobby) as a result.

It’s frustrating. I had outbound smtp throttled right down and a script running that shut smtp down completely if the queue got too big. That triggers an alert and everything gets sorted. But no, we don’t like where you’re from so jog on. Monopolistic.

> If a domain's authoritative mail servers have a history of never sending spam, when that company or hobbyist (for any reason) has to switch hosting providers and gets a new IP address, Google/Microsoft should recognize that the domain, and its referenced mail servers, has never been malicious.

That algorithm doesn't work. The internet is filled with parked domains that have "never been malicious". This just creates a new market for clean domains that you can use to evade protections. It makes spam a little more expensive, but it still gets through.

None of these tricks work. There are no tricks. All rules can be gamed. The only thing that can't be easily faked is reality: if Microsoft knows you're a big org with a well-managed IT group running your output email setup, then they know they can (probably) trust you not to spam their customers. If you are too small to prove that to MS in a scalable way, no amount of heuristic trickery is going to help you.

I'm sorry, but I don't buy that argument.

Google--a technology giant with algorithms and heuristics running most operations--is not up to the task of improving algorithms for email server reputation? No, they are definitely capable of improvement. It's simply not of interest to them.

I can hardly blame them because the cost-benefit analysis clearly says, "why bother?"

Of course anything can be gamed, but magnitude matters. I've had the same domain for 24 years, with many hundreds or thousands of email conversations between users of my server and those of the major email players over the years. I had to switch my mail server's IP address two years ago. Immediate zero reputation from many big players.

> is not up to the task of improving algorithms for email server reputation? No, they are definitely capable of improvement. It's simply not of interest to them.

Incorrect, if they stopped spending all that money and effort to keep up, their users would get flooded.

> Of course anything can be gamed

Sending spam/marketing e-mails is a multi-million industry. Both on illegal and legal markets. It's a constant race.

Parked domains also don't have a record of sending good emails either.

Tough love: neither do our vanity domains (and yes, I have one too, and feel the same pain trying to get gmail to take mail). No one is going to pay someone to dig through our decades of mailing list activity trying to figure out whether to let a half dozen replies through.

This isn't the 1990's anymore, recipient domains aren't just homes to a few hundred undergraduates or a dozen programmers. Email hosts manage communication for hundreds of millions of customers.

The really annoying thing these days is google blocking zip files as spam requiring you to use google drive.

Zip files are being used to infect computers with malware which would explain why they're blocking them

I guess they're not allowed on Google Drive either.

I could bet that most of mails from new IPs are spam because there are much fewer people willing to set up a mailserver than there are spammer operated mailservers. It is cheap to buy a new IP(comes free with many $2 hosting plans). The problem of identifying if a new IP is used for spam is not easy to solve.

> The problem of identifying if a new IP is used for spam is not easy to solve.

How is it not easy to check a reputation database for domains when evaluating a mail server?

Using DKIM records, Google could cross-reference the reputation for the sending domain and, where applicable, recognize that the domain in question has never been malicious or negligent. And in that case, extend a probationary reputation sufficient to allow the IP to establish its own new reputation.

New IP means it is new and not part of any database.

I'll quote myself:

> How is it not easy to check a reputation database for domains when evaluating a mail server?

The point I'm making in this thread is that they have the ability to maintain a database of domains that have been proven to be trustworthy. I am not talking about a database of IP addresses, and I am not sure why that is being raised here.

> The problem of identifying if a new IP is used for spam is not easy to solve.

Huh? With dataset of ~100k classified hams/spams I get like >99% precisison in identifying spam/ham with just bogofilter and 0 heuristics whatsoever (my mail server accepts everything, and I just use bogofilter client side).

I guratanee you MS has a dataset with > 100k emails, lol.

It's not like this is unsolved problem.

Right. While at the same time as they clumsily block any IPs with unknown reputation, they seemingly cannot reliably flag a poorly formatted, obvious and well-known fake invoice for "Norton 360 renewal" as spam (and really, it's worse than spam, it's a scam).

> It's not like this is unsolved problem.

My god, are you joking or what? Spam, a solved problem? Far from it.

Why can I identify spam at a glance, with greater than 90% accuracy, from the subject line alone, and to Google it's an unsolved problem?

Google reads and indexes every email that they deliver to a gmail inbox. They probably have the worlds largest and highest paid staff of ML experts, and they resort to auto-flagging based on IP address?

Fix it.

> to Google it's an unsolved problem?

It's an unsolved problem for anyone.

> Why can I identify spam at a glance, with greater than 90% accuracy

I don't think you understand how terrible even 99.998% accuracy is at their scale.

> They probably have the worlds largest and highest paid staff of ML experts, and they resort to auto-flagging based on IP address?

You think they aren't using them for this? IP addresses very likely go into their model as well, when determining something is spam.

> Fix it.

They are and adversaries are breaking them again.

> I don't think you understand how terrible even 99.998% accuracy is at their scale.

What does scale have to do with it, am I missing something? Wouldn’t a spam detection rate of 99.999% mean that only every 100000th spam mail would get through on an account basis, how’s that terrible?

> What does scale have to do with it, am I missing something?

The person I replied to said they can identify spam with 90% accuracy with a glance, my point was that it's abysmal as accuracy. Even orders of magnitude better it'd still be bad.

> Wouldn’t a spam detection rate of 99.999% mean that only every 100000th spam mail would get through on an account basis, how’s that terrible?

I didn't specify if it's per-account or a false positive rate, it's also actually not relevant to the point I'm trying to make.

When you receive billions of letters a month small errors start to matter and they're doing a better job than the person I replied to thinks they do.

You can identify the spam because it's your inbox; you know what your goodmail is supposed to look like. For that reason you can use locally-trained naive Bayes on the client, and get good recognition (you don't need ML).

Honestly, the best filter in my experience is Spamhaus Zen; it spots most spam with very few FPs. It's an IP blocklist.

If you use something like rspamd, then doing ML is not terribly hard. Then it becomes a problem of providing a proper teaching set. Combining RBL's (be it URI, IP or domain WL's or BL's) will help with that.

In the end it's still a hard problem, so takes a bunch of work.

Yes, spam classification is a solved problem. Perhaps even at the level of "this is new IP address with no reputation, is it sending 80% spam or 80% ham?" kind of questions with some defined precision at scale, but ceratinly at personal mailbox level, if you own and train your own clasification engine with thousands of samples of ham and spam you actually receive. That's very effective, and much better than my past experience with gmail.

> small and hobbyist mail server operators

From the perspective of an email receiver, I WANT things to be difficult for this group. Truly free email available to all would be unusable owing to spam. Oligopoly, I believe, results in the best experience for everyone.

Reply on news.ycombinator.com