Hacker News Reader

Comment by rsync | Hacker News Reader

There's a lot to say here from both sides - people running their own mail infrastructure (like I have for almost 25 years) and big mail providers dealing with brutal, unrelenting spam.

But there is one piece of this that's ridiculous, broken and almost cruel: silently dropping messages marked as "spam" with no notification given back to the sender.

Why does this practice exist ? Who believes that this is decent or acceptable behavior ?

If gmail doesn't want my inbound message - for any reason - that is just fine.

If they drop it on the floor without telling me that is totally shitty.

View on HN

Mailservers drop mail on the floor because they are doing spam-filtering AFTER accepting mail for final delivery. Once it's been accepted for final delivery, it can no longer be rejected.

The delivery server doesn't generate the bounce message you were expecting; that's generated by your own mailserver, on seeing a REJECT status code from the delivery server.

Mailservers do spam-filtering after accepting for final delivery because spam filtering can be processor-intensive. Sometimes it's farmed-out to an appliance or whatever. To have the SMTP process suspended while Spamassassin goes through it's contortions multiplies the consumption of server resources on the SMTP server.

The delivery server CAN'T (and shouldn't) send you the desired bounce message, because it doesn't really know who you are. It can't rely on the From: address, because you could be sending on behalf of someone else.

In my view (and the view of the RFCs), if a server says "200 OK Accepted for final delivery", then it MUST deliver the message.

There's an awful lot of the kind of server-side spam-filtering that does actually involve delivering: the kind that filters mail into the recipient's spam folder. That mail hasn't been dropped on the floor. It's been delivered, just not to the inbox.

You've described how my mailserver should work.

I'm willing to stipulate that this is correct and would, in my case, be a difficult problem to solve.

But nobody is losing job offers or missing kids' schedules or breaking their summer plans because of my mailserver.

I am talking about gmail. I am talking about MS (whatever it is). I am talking about yahoo.com.

Their spam heuristics are, in many cases, laughably bad - they are demonstrably, clearly broken. If I email my wife twice daily for 15 years and then one of my responses to her emails gets put in the gmail spam folder ... what words to even use for that ?

They need to fix this. I don't care how sticky of a problem it is.

$employer uses MS for email hosting. MS recently started dumping every single email from every single Apache mailing list in the spam folder. These are mailing lists to which I've been subscribed for a decade, to which I regularly send emails, and which probably have thousands of subscribers. There is no option for me to whitelist the mailing list, only individual senders, of which there are hundreds.

Everything about megacorp spam filtering is broken.

> they are demonstrably, clearly broken

Well, gmail, MS and Yahoo have their own ideas about what "broken" means. Google in particular forces changes to standards by simply implementing them in their own services. Those changes never make it easier for small-fry postmasters; so I conclude that Google would like all small-fry mailservers to disappear.

Discrimation through spam-filtering isn't unthinkable, and it would be hard to prove (especially if they claimed there was "AI" involved in the filters). Google used to have really good spam filtering; I can only suppose that the reason it's got worse is that they want it worse.

if gmail wanted to, they could provide some SMTP extension. eg. they could advertise that the sender server can send a callback url where gmail will post back the results

there's already some kind of feedback loop mechanism, but mostly available for large senders.

Responding gives the sender the ability to better explore the detection rule space and find a way to get through.

Exactly what a legitimate sender wants and what a provider would not want to give an adversary. Now the adversary has to also incur a cost to determine successful deliver vs open/engagement rates make it just that much harder.

I think it's a false economy. At best it is security by obscurity.

Google tell you why things are spam and often they even return quite detailed error emails when they don't accept stuff it doesn't benefit an attacker that much. Any decent attacker already knows that they should sign stuff and make identifiers align and can do so trivially easily.

People like Yahoo are the opposite and are completely opaque as if they are doing anything that clever. All they can realistically do is check originating IPs, message content, alignment etc. just like everyone else.

Since I can still a lot of very decent SPAM in my inbox, their lack of transparency clearly doesn't work so they might as well help legitimate senders to deliver stuff properly.

"Google tell you why things are spam and often they even return quite detailed error emails when they don't accept stuff ..."

Where are you seeing these details ? I have never seen any bounce messages from messages I send into @gmail.com that end up in a spam folder ...

In case of gmail you can have an account there and see what goes through.

I would think returned spam would have to be checked for being spam by servers receiving the returned messages.

If they don’t do that and decide whether checking is needed based on presence of some mail headers, or using heuristics on the subject line, spammers will start faking returned messages.

I think that makes it expensive to return spam. https://99firms.com/blog/spam-statistics/ says about half of all e-mail is spam, so returning that would increase e-mail traffic and spam checking by 50%.

A bounce message can be very small, specifying only three fields: To, From, Date to identify the rejected message. Compared to this spam messages would be much bigger, consisting of html, css, javascript, pdf attachmens, zipped executables etc. The message could be sent to a fixed address, like abuse@same.server; how to check messages on that address is a different story as it's not user messages, but technical messages.

> If they drop it on the floor without telling me that is totally shitty.

No, it's the ONLY reasonable thing to do when something like 98% of all SMTP traffic on the Internet is spam.

If mail administrators bounce back an explanation for every "bad" message:

1) Their outbound mail volume would go through the roof.

2) The host sending all the bounces would look like a spammer to _other_ automated spam-classification systems.

3) In the unlikely event that a spammer actually reads bounches, they could use the feedback to tweak their systems to avoid the spam filters.

What a recieving mail server should do is reply immediately after the DATA command with “That mail is spam and I refuse to acknowledge recieving it”. This would be fair, as the sender would know that the mail was not passed on. What it should not do is to simply say “Yep, I got this mail now, I’m good!” and then later, silently look at the contents of the mail and decide that it’s spam and delete it.

There are sane email providers who do exactly this, for example Posteo — see question "Is there a spam folder" here: https://posteo.de/en

> If mail administrators bounce back an explanation for every "bad" message:

Nobody is talking (I don't thing anyone is) about sending back a bounce message, that would indeed make no sense.

A responsible email server should:

1) Reject the email during the SMTP conversation if it's going to do that. Then the sender knows it didn't go through because it got the error code. There's nothing to bounce back.

2) If it accepts the mail during SMTP conversation, then always deliver to the recipient.

2a) Some disagree, but I think it's totally fine to deliver it to the recipients spam folder if post-processing determines it might be spam. That's not wonderful, but it still got delivered and the recipient can go find it in the spam folder. Most people are used to looking there regularly anyway since many of the larger providers (coughgooglecough) have such terrible false positive rates. The important thing is to never lose email.

What's never ok is to accept the email during SMTP and then silently file it in /dev/null.

Does this really happen? Because I imagine that, if Gmail really did drop some super-spammy incoming messages, it would includes quite a lot of my current spam folder[0]. Maybe they only drop spam when it’s an IP [range] ban at some firewall level?

0: https://cdn.discordapp.com/attachments/468851579487780905/96...

I run my own mail personal mail server, and do my own spam filtering. Things that are likely spam get delivered into a spam folder for the occasional review, but things that are extremely likely get rejected entirely (with a "spam rejected" message).

I guess my feeling is: if I want to quarantine suspected spam without telling the sender, that's my prerogative -- why does the sender get any say in this?

I don't think he's saying you can't quarantine spam if you choose. What his complaint is, I send you an email, your mail server says it accepted it, the email is then just discarded entirely (not put in a spam folder).

I guess the volume of spam-reject messages would dwarf legitimate mail.

Reply on news.ycombinator.com