Hacker News Reader

Comment by c0l0 | Hacker News Reader

Having been a part-time postmaster for more than a decade by now, I fully agree, and would even go further: Ingress spam is pretty much a solved problem if you play your cards right. ChatGPT et al. might change that again - but the mechanisms you can deploy today are very effective against the current UBE landscape.

The _real_ problem is reliably getting your 100% legit mail into your consenting recipients' inboxes.

View on HN

I've been doing it for well over 20 years now.

Ingress - out of the box rspamd is pretty decent and it is rather configurable.

Egress - DNS (A,AAAA, PTR), (E)HELO, SPF, DKIM, DMARC. "IP Reputation".

There are absolutely no shortcuts and yet most of the problems I diagnose regarding email delivery will find a missing PTR record or a miss-configured (or non configured) HELO. You cannot be lazy when it comes to email. SPF + A + AAAA + PTR and (E)HELO is a minimum.

There is one thing that you cannot generally, personally mitigate and that is being on a deny list due to your IP address. This one is a bit more tricky to deal with. You might have to use a relay. Another mitigation might involve IPv6.

> Egress - DNS (A,AAAA, PTR), (E)HELO, SPF, DKIM, DMARC. "IP Reputation".

Any insight into why this is a giant ball of complexity? I’ve had to setup spf, dkim, dmarc a couple times now and man… design by committee?

There is no design committee, they just keep adding stuff over and tell us is definitely the one that will fix security/phising/privacy/spam issues SMTP suffers.

I think it’s a miracle E-mail still works at all this far into web whatever-point-oh we’re in.

That is not that complex...

They solve different things. Email is very old so obviously things are going to be tack-on rather than baked-in.

The standards are fine in terms of achieving their goal, the hard part is getting the ecosystem to follow and make everyone's life easier.

Sorry for naivety, what is HELO in this context? I've been running a mail server with the other aforementioned things set correctly and tested; I've been able to reliably deliver mail to all of the big senders for years now, albeit on a small scale.

Am I missing something not knowing what HELO is ??

HELO (or more usually EHLO) is the initial greeting of SMTP protocol which includes the mail server's address.

AFAIK plenty ofservers will outright reject all mail if you only have an IP address and not a domain name there.

As a user, I'd rather my receiving email provider (eg. Gmail, iCloud, Outlook) be less restrictive in their filters if it means individuals can host their own mail servers and we can have a more egalitarian internet.

Extremely filtered emails is practically the same as a social network determining what I can see at this point.

Only it’s not a social network, not even close.

Web standards have advanced to a level that walled gardens are replacing email. Email itself is still stuck on IE6 level standards. And SPF, DMARC and DKIM are a confusing mess to deal with.

Email is showing its age, but are SPF, DKIM, DMARC really confusing?

I've had a company tell me the other day that you can't do DMARC without using their services. It's just too complex. Why is that - we don't run our own mail servers and we use Google mail for our organization. I'd think Google would do that?

If a company tells you that it must be true. /s

Sounds like a salesmen who doesn't understand DMARC himself was trying to sell it by labeling it too complex.

From a purely technological point of view DMARC isn't that complicated. It just specifies how to treat DKIM and SPF results (with a bit of rather simple configuration). SPF is basically a list of ip's you own published on your domain (if the email was sent by that ip, you vouch that it was sent by you) and DKIM signs the email with a private key and you publish your public key on your domain so everyone can verify that this email was signed by the domain owner. SPF might fail if your mail gets proxied (as now its a different sender ip that you didn't vouch for) and DKIM might fail if the mail got modified including headers (because the signature can only be verified for exactly the original headers+content). So if you're sending email for someone else it gets a bit tricky, but for your own emails it's certainly not "just too complex" and boils down to a few line long configuration file, a list of ip's and a private/public key pair for signing emails.

Yeah, I need to do more research on it, thanks for your advice. It seems like this must surely be possible to set up on our corporate Gmail admin account.

> The _real_ problem is reliably getting your 100% legit mail into your consenting recipients' inboxes

It's amazing that having someone in your address book isn't enough in many cases. Like, why?

Because the from address can be forged, probably.

Not really a problem in the age of DKIM, _if_ you want to solve it.

Even with DKIM, all you need is the recipient of one email from one user on one domain (I have hundreds of domains) of your mail server to file a spam report, and WHAM you are blacklisted. So yes, it is a problem even with DKIM. If you have a solution, I would LOVE to hear about it.

If you’re blacklisted, the mail usually doesn’t even reach the spam folder.

Correct.

One of my clients is a construction firm specializing with churches. It is not infrequent for them to be communicating about a project with a church - often to a role account (e.g. info@church.org), which is step 1 towards being filed as spam - where the role account is shared with a dozen or more people. The building manager will check the email on Mon, Wed and Fri and every other day there will be a number of well-intentioned volunteers at least one of which will click on the spam button dooming my client's email into the abyss. Weeks later we find the church has gone with a different contractor, as they "could never get a response" from my client.

We will go so far as to donate a new IT/Cloud system to the church (small $$ compared with the project) just to ensure reliable communication. But then they think we are just trying to buy them.

Now my client has a blacklisted domain/mail server IP, and bids they send out are rejected as spam by the providers to other churches.

Makes no difference if we are hosting the mail server or if it is a 3rd party mail service. As soon as a customer is sold, we try and move them off email into a web-application framework for ongoing legitimate communication. Again a lot of resistance.

Does this happen even if you move your client to Google Workspace/Office 365? Using one of those two should eliminate the problem. Was that your experience?

Life is so easy if you just hand over your personal agency to monopolies and billionaires who don't actually care about you /s

If the alternative is that your construction firm fails and you and everyone else are out of a job - well your email hosting choice is a weird hill to die on.

(the '/s' means 'sarcasm' and that wasn't the user you replied to)

Doh missed that, reading is hard!

No. Workspace gave no apparent protection. Why/how would it?

Presumably most of the firms clients are themselves hosting email on Workspace or 365. And Google/MS might treat email originating from their own systems differently than inbound email originating elsewhere.

Also I'd expect all the firms competitors to be using Workspace or 365, so if using them gives no apparent protection, presumably they should be suffering from this as well?

I file this under “you can’t have a technical solution to a social problem”. We can do all we want to protect e-mail, but when it comes down to it, someone is going to figure out a way around it and ruin it for others.

The current situation is that we have technical solutions for authenticating smtp sending domains. But there will always be someone who flags an email too quickly or just wants to spite you. Or someone that will send spam regardless (or hack an account, etc...). And so we’re back at square one.

This is especially true for situations (like email) that aren't subject to market forces. Because sending email is so cheap, the return rates can be very small to still justify sending spam. There just isn't any market pressure to keep it contained...

Hashcash was originally proposed to add some form of cost to sending email. Something similar could be a great way to get mail from legitimate people through. Spam wouldn't scale but the average person would only pay in a bit of CPU time/cost.

“spam wouldn’t scale” - unconvinced on this: spammers already mostly use other people’s compromised machines to do the sending; there is no cost to them here.

If you charge some cost per mail (whether that's CPU time or actual money), users/teams would check their spend and optimize accordingly. They'd notice runaway spend and act on it. The only reason why mail servers become compromised and nobody notices is that bandwidth for mail is generally too cheap to meter. On any 10s of megabits connection, sending a deluge of spam is trivial.

Not even that, you can become the victim of a "noisy" neighbor if someone on the same IPv4 /24 sends too much spam (some of the common blacklist providers will do entire netblocks if they get enough complaints)

Yeah, it may be a combination of “not everybody uses DKIM” and “too few users actually use their address book”.

from can be forged, but spf is there so only valid servers can send mail (or at least non valid can be filtered).

SPF can break with automatic email forwarding (though DKIM usually shouldn’t).

But to exploit this, an attacker would need to have the address book of every account they are trying to spam.

That's relatively trivial. Send N^2 emails, pay attention to the bounce backs.

There's almost no cost to send an email except time so even when n is large, this does not prove nearly as intractable as one wants it to be.

I’d guess the main issue is, the big guys don’t care about you. Like, at all.

If they pass their log through a scanner, come up with a few dozen small servers that look fishy (e.g your IP is contiguous to some other spammer’s IP), and yours is rope in as false positive, you’re banned.

And from there you’ll have to convince Google that you’ve done nothing wrong.

> And from there you’ll have to convince Google that you’ve done nothing wrong.

Imagine the conversation with their support when you ask why bob@domainx.com's email was put in spam folder of jan@domainx.com's email, as both are hosted by Gsuite and presumably entirely within the Google network.

The answer, after two levels of support, was, "We don't understand why, but we can open a ticket with the developers." In the business world, you don't have time to wait for that, especially when non-technical business users are making mistakes or losing business because they can't function properly.

I've gotten emails from old friend accounts that were totally taken over for spam purposes. Usually a low tech friend who just creates a new account.

I moved a Gsuite client to Zoho because once or twice a week, an email from one person at the custom domain to another person in the same custom domain (company) would have their direct email get put into spam in Gmail.

You would really think that Google would whitelist emails within the same custom domain which is paying for Gsuite service. Maybe that has changed, but it was definitely a problem 5 years ago.

And we're not talking about an email which had some copy/paste of spam, we're talking about a one or two line sentence giving an instruction or asking a question to a colleague.

https://support.google.com/a/answer/2368132

> Bypass spam filters for internal senders

Is this what you are looking for?

Shocked that an article is required to explain what should be solved in the product.

It is solved by the product. You tick a box, and internal mail is bypassed.

The issue here is that developers and IT folk seem to think that email is easy because the know IT. Email is a complex, and old protocol that has many nuances and, believe it or not, phishers are smart; end users are naive, and often reckless. Spammers, be they benign or otherwise, can be incredibly lazy too, but don't think for one second that you, as a postmaster, are cleverer than they are. Spam and phishing are getting harder to beat. Businesses like Proofpoint, Cisco and Mimecast, along with the likes of Google and Microsoft, are investing heavily in trying to beat these guys, but the reality is that they are always at least two steps ahead.

Check box or not, my point is this should be default behaviour, I.e., don't mark same domain as spam, and instead make that an option to enable.

Drives me nuts having to leave a UI to google (ironically) the issue to find a doc to enable a check box.

At scale, that’s not an option. If you have external services that send mail to your Workspace tenant, there is a risk of compromise to the sending service, especially if it is outside of your control. The sending service could have SPF records that permit sending mail on your behalf. That mail needs to be scanned. There is also the threat from bad actors inside your perimeter. Sure if you’re a small operation, or using Workspace as a personal mail host, this may seem like overkill, but I can assure you that the majority of orgs that use Workspace (it a business too after all) would prefer the status quo.

Don’t know about GP but this is what I was looking for. Thanks!

Google's own emails about expense reimbursements were systematically flagged as SPAM on gmail and they knew it.

They still didn't care at all!

That would demand maintaining an address book. Most mails I receive are from services which I have never written to, and I don't see a reason to extra maintain their data when I already have their mails.

For sure. I’m contemplating a scenario I’ve had multiple times “No I didn’t get it” “Okay let me try again, did you check spam?” “Yeah” “Maybe add me to your address book” “Nope still nothing”

Well, ChatGPT could write a mail server too, so what are we even trying to achieve any more?

Just let ChatGPT send emails to itself, using mail server and client software that it wrote itself, and the rest of us don't have any need to handle email ever again.

I believe what the parent meant was that SPAM blocking relies heavily on keywords, and strings. With the advent of chat gpt we go beyond a/b testing and push it to into the every infinite way of someone conning you to click on something through email.

Reply on news.ycombinator.com