Hacker News Reader

Comment by susam | Hacker News Reader

I posted this two days ago. I believe a moderator must have rescued this post today to bring it to the front page.

By the way, I posted this link because I've been running my own mail transfer agent (MTA) for my personal vanity domain to send/receive emails for about 10 years now and I often forget why I need to edit /etc/mailname while setting up the MTA, so I posted this link to HN as a bookmark for myself. (Who else uses HN submissions as bookmarks?)

I use Exim 4 as my MTA on Debian GNU/Linux. During its setup, among many other things, I need to configure /etc/mailname with the domain name I want to see in the "From:" field of outgoing security notification emails, e.g., email notifications for failed sudo attempts.

If you want to take a look at my MTA setup, see this Makefile: https://github.com/susam/susam.in/blob/master/mta/Makefile

Related Show HN post: https://news.ycombinator.com/item?id=22299169

View on HN

Is it possible to run own SMTP server these days and send emails successfully? I thought GMail will classify it as spam. How do you tell the big email providers to not classify email originating from personal domain as spam?

It's not difficult - most distros have email servers you just install. Most of those use that same email server internally for their mailing lists. It's not much harder (or easier) than setting up any other server.

GMail does not classify an "own email server" as source of spam. Quite apart from anything else how would it know it's an "own email server" as opposed to some companies email server or even a distro email server. They all user the same code.

To get your email server classified as a spammer you have to send spam (mostly). That won't happen if it is just forwarding emails you create, but if it is the work horse behind an email list server or something most of what the list will see is spam and not relaying that to GMail or something becomes very hard indeed.

The reason for the (mostly) above is occasionally somebody will have finger trouble and mark some good email coming for your server as spam. As it happens GMail seems fairly immune to this problem. I guess they must need quite a few people to do make it before it kicks in. Microsoft's servers seem to make the mistake occasionally, and some middle wear boxes can be bloody horrible about it. Fortunately they all seem to provide web pages to fix their mistake you just have to find it.

In a universe where the SORBS DUHL and its ilk never existed, this may be true. In this universe, however, they did and do exist, and running things on one's own machines renders one a third-class citizen.

* https://news.ycombinator.com/item?id=14719504

I'm not sure what evidence you have for this, but I'll provide my own: I run over 30 mail servers on normal ISP links. I can categorically state that's not a problem.

FWIW, I agree was a problem years ago. I suspect it isn't now because ISP's can not longer say what an IP Address is used for, or maybe it's because the DU in that RBL's name stands for Dial Up and this is 2020, not 1999. There is one caveat: you do have to get the DNS PTR records set up.

A good way to avoid this is to setup dkim and spf for your email domain.

https://www.sparkpost.com/blog/understanding-spf-and-dkim/

I follow the usual best practices and set up DNS TXT records for Sender Policy Framework (SPF), DomainKeys Identified Mail (DKIM), and Domain-based Message Authentication, Reporting & Conformance (DMARC). Only DKIM requires configuring the MTA for it because the MTA needs to know where the DKIM private key is to sign emails and what the DKIM selector, domain, etc. should be in the DKIM-Signature header. From my Makefile[1]:

  # --------------------------------------------------------------
  # Enabling DKIM ...
  # --------------------------------------------------------------
  echo DKIM_CANON = relaxed >> /etc/exim4/exim4.conf.localmacros
  echo DKIM_SELECTOR = s1 >> /etc/exim4/exim4.conf.localmacros
  echo DKIM_DOMAIN = $(FQDN) >> /etc/exim4/exim4.conf.localmacros
  echo DKIM_PRIVATE_KEY = /etc/exim4/exim.key >> /etc/exim4/exim4.conf.localmacros
  openssl pkey -in /etc/exim4/exim.key -pubout -outform DER | base64 -w 0 > /tmp/publickey

After that, it is just a matter of adding the necessary TXT records for SPF, DKIM, and DMARC to the DNS zone. The relevant section in the Makefile[2]:

  # --------------------------------------------------------------
  # Generating DNS records ...
  # --------------------------------------------------------------
  @echo "Add the following TXT records to DNS:"
  @echo
  @echo "@ TXT v=spf1 mx -all"
  @echo "s1._domainkey TXT p=$$(cat /tmp/publickey)"
  @echo "_dmarc TXT v=DMARC1; p=reject; adkim=s; aspf=s"

Having said that, I have also found in the past that major email providers would accept emails sent by my MTA just fine even without these measures. For a long time, I did not have SPF, DKIM, or DMARC enabled and I did not observe any issues. However, it is a good practice to use these measures to make it harder for a bad actor to spoof and send spam that appears to originate from your domain.

[1]: https://github.com/susam/susam.in/blob/7cff99b/mta/Makefile#...

[2]: https://github.com/susam/susam.in/blob/7cff99b/mta/Makefile#...

It is.

I setup a simple new virtual-server yesterday, initially outgoing mail was rejected at SMTP-time by Gmail with an error referring to reverse-DNS for the IPv6 address.

Fixing a couple of things like that (add/update a SPF record, setup reverse DNS for IPv4 & IPv6) and mail was successfully delivered to gmail.

Biggest pain there is that some virtual-machine providers don't let you change RDNS for IPv6 addresses (e.g. scaleway). So you get the choice of disabling it, or changing provider.

I never had any problem with gmail, but outlook et al I can no longer send mail to, since I switched VPS providers.

My IP range is on a blacklist and it won't come off.

Outlook has a manual way of removing a single ip. There should be a URL in your mailer daemon response to do so. And yes, f u outlook.

Yep, but as far as I understand, the VPS provider should use the URL. And they (supposedly) did, twice and nothing came of it.

If you have the reverse DNS set to something within a domain you own, they'll usually let you do it yourself (the process involves them sending an email to postmaster or abuse at the parent domain of the server or at the AS which controls the IP address, but you get to choose which one).

Still doesn't seem to stop them randomly flagging emails as spam, sometimes mid email chain.

The MS support team are also responsive in my experience.

I've never had trouble sending to gmail.

I've never had occasion to send to yahoo mail, but recently sent a test message to an account I had there. Yahoo said it was spam. I marked it as not spam.

After that, sending to yahoo was fine.

I also had never sent to outlook mail. I made an account and tried sending mail from my domain. It went to the spam folder. I told outlook it was not spam.

Tried another message...still spam. Told it that it was not.

I think I'm up to 10 or so such messages now, and outlook is still saying they are spam.

Totally possible. Run SPF, DKIM, and DMARC, have correct forward and reverse DNS, have reliable DNS servers for every lookup, forward with ARC, deliver with a valid certificate, prefer IPv6, and have all associated IP addresses and sender (SMTP envelope) domains with a very good reputation. This last part takes months if not years to build, it’s not hard, but it’s not fast.

I use separate sender domains for a newsletter than for transactional mail.

You can monitor reputation with Gmail and Outlook postmaster tools, and there are various sites that will mass query RBLs; worth checking occasionally. I have a monthly reminder in the calendar. Sign up for the JMRP and SNDS.

Then just behave like a responsible sender. Don’t send unsolicited mail. Make it easy to unsubscribe, with RFC8085 headers and easily found unsubscribe links. Honour unsubscribes instantly. Honour spam reports instantly with a suppression list. Make the first statement of your communication one that describes the purpose; the second, the unsubscribe link. Make it easier, both functionally and emotionally, for recipients to unsubscribe rather than to report you as junk.

Run your emails through SpamAssassin before sending, and respect and understand the scoring. Read the Gmail sender guidelines (https://support.google.com/mail/answer/81126), they are good advice.

Include a plain text part.

Do not trust anyone claiming you can pay to improve reputation. That is snake oil.

No.

GMail is arbitrarily blocking ip addresses or subnets and there's nothing you can do about it.

One year ago I've purchased a VPS with a clean IP address (checked against all DNSBLs), set up mail server with SMTP and IMAP server, set up SPF, DKIM and DMARC and was able to successfully send emails to Gmail for 3 months.

After 3 months GMail stopped accepting mail from my server saying that my ip address was used to send spam. I'm pretty much sure it wasn't me. My VPS OS and mail server are up to date, there was no suspicious activity in the logs, so it's very unlikely that I was hacked.

There're only three possible explanations for it: 1. My VPS was hacked indeed 2. Some bad guy from the same IP network sent spam (although GMail error message specifically states that spam was sent by this exact IP address, not an address from the same network). 3. GMail just randomly blocks access to any little guy trying to setup independent mail server just because they can.

I'll never know the truth but it was very disappointing experience. Also, lost couple of semi-important emails this way.

> 2. Some bad guy from the same IP network sent spam (although GMail error message specifically states that spam was sent by this exact IP address, not an address from the same network).

This is the most likely situation. If a VPS provider is sufficiently cheap, it will attract the wrong people, who will then get the entire VPS provider's IP blocks blacklisted.

It's a shame, but it's the reality. There are lots of discussions around this subject on various mailing lists, where people from the likes of Gmail have chimed in stating as much.

I would reckon it takes as little as 10% of the addresses in a /24 (so ~25 IPs) to get that entire netblock blacklisted.

If you want a higher probability of clean IPs, you need to be using a provider that is sufficiently expensive to not attract spammers.

Either that, or you need your own IPs.

> Either that, or you need your own IPs.

Or just use IPv6, which will probably fix this.

Why is this considered a useful signal by Gmail? Isn't it cheaper for a spammer to get throwaway IPv6 addresses?

Most spammers aren’t competent to provision an IPV6 service correctly, especially one with matched PTR and SPF(TXT) records, and using throwaway v6 addresses makes SPF much harder. It’s also easy to spot, at which point you can tag the sender domain with a bad reputation.

It’s no surprise that Gmail et al are more interested in sender domain reputation, and also the graph of related domains, than they were a few years ago.

This'll stop being true at cheap VPS providers as soon as spammers cotton on, and then VPS providers ranges will be blocked in /48s (even if you only get a /128 or /64 from your VPS provider) because it's not worth the hassle of doing anything smaller.

It sounds like someone on your provider's network got the subnet blacklisted.

I've been running mail servers on OVH (which is known to be a cheap host) for years and I successfully deliver to GMail and other providers without any issues. I'm pretty sure delivering over IPv6 also helps a lot since spammers will never bother setting that up.

Not sure why you said “no”, because the whole of your remaining comment simply reinforced that reputation matters, including reputation by association.

Association extends to sharing an IP allocation. It may even extend to sharing a nameserver. Gmail isn’t randomly targeting a “little guy” for the Internet Death Penalty. This isn’t arbitrary, and an entitled victim mindset isn’t going to get your email delivered.

Like I said, prefer IPv6. For now at least, bad senders don’t choose it. That’s the something you can do about it. Pro tip: most VPS services are a swamp.

Incidentally, if your server was rooted, don’t assume that this would show up in the mail logs. There are plenty other ways to use a compromised host as an open relay, including lying to your admin tools.

Ultimately, I think that if you’re unwilling or unable to run a MTA to the maximum level of reputation, then skip doing so, and use Postmark instead.

Reply on news.ycombinator.com