What's an SPF Record?
blog.ohmysmtp.com
blog.ohmysmtp.com
I see e-mail as another Internet service that is slowly centralizing, and if you operate outside of the big tech companies- for example, if you don't use Google/Yahoo/M365 e-mail, you're probably going to be in for a headache, even if you run perfect e-mail infrastructure.
On the side of centralization is a market worth unimagined billions, and perhaps seven providers that already routinely lobby to get what they want.
On the side of decentralization? Little guys who are already being bullied and don't have the resources. Politicians who have other extremely dangerous problems currently, including and especially those in their own industry. Don't expect their help for a long time. Smaller service providers who only want the chance to switch sides. And large independent organizations (businesses, schools, governments) who are really only interested in a cheaper bill for the same service quality.
Look, we have struggled to fix the MITM/plain text problem and spam/origin problem for well over 20 years, and the Internet Age is maybe 30? It's extremely likely that the only way to fix the core design problems with email is to have the service consolidate to a small consortium of services, have them improve the base design, and then fracture the monopolies. That's going to take decades.
And mail is still such a pain in the butt for a user document database. And the user experience is a joke! Asynchronous delivery, best effort failure? I mean, what? The average web page is orders of magnitude larger now! People complain about FAX and FTP.
A new, chat-styled mail protocol can be designed to address all the problems above while reduce the risk of abuse. I wonder why nobody standing up to do it.
Unfortunately, we are back to the age old question. How can we have an ecosystem where anybody can email anybody without it being taken over by spam?
If you still have a google account that has the “calendar” feature enabled, this will lead to google sending invite acceptance emails.
Especially if you have a business account (which was my experience,) where you verified the domain.
The solution for me was to disable “calendar” as a service on the entire account.
Realize that “google calendar” and “google mail,” are technically distinct services. However that doesn’t seem to stop “calendar” from sending emails on your behalf.
https://dmarcly.com/blog/spf-dkim-dmarc-set-up-guide-for-g-s...
I doubt google will send out spam on your domain.
Even using SendGrid, it will be a week or two and a few hundred emails to 'warm up' one of their mailserver's IP addresses and get reliable delivery. Like you, I found VZ to be the worst at trusting new servers.
The days of firing off an email from any ol' SMTP server are completely gone.
I'm not sure... I agree with the general vibe in this thread however my experience (within the last year) is that you can fire up a fresh one and send email to gmail at least, without SPF, DKIM or DMARC.
However as soon as you start doing any mass mailing of any kind you need all three of those things, SPF to not get outright blocked and get into the spam, and the other two to have their spam detection filters to even bother parsing your email content for the chance of getting into the inbox.
This is my experience from setting up systems that send automated status emails not marketing, so fairly low volume but frequent enough with same message content to be immediately classified as mass emailing.
The problem is the other provider. Yahoo, AOL, Microsoft. They're all a pain in the neck and much, much stricter. And sadly, they still amount to a not insignificant portion of the market.
Just when I relax content that things are in control I get a folder full of files which contain spaces in the file names that need to be processed.
The only place to fully self host your own email these days is on a more boutique isp where it's not possible for anyone with a credit card and a pulse to sign up as a customer in a fully automated process.
The first RBL showed up around 1998, operated by Paul Vixie of DNS fame. I was using a small ISP in the East Bay at the time and they hated the MAPS RBL, especially the idea that they had to deal with some third party to resolve mail transport issues. But, from 1998 until the early 2000s, RBLs proliferated and lots of mail services subscribed to one or more. The one thing they had going for them was that they were easy to query, so it took no effort to find out if you were on the list, and if you were, there was usually a living, breathing human being somewhere that you could contact to get the matter resolved. It was a pain in the butt, but it could be handled.
During this time, the responsibility for ensuring that a message was received shifted from the recipient to the sender. Beleaguered systems administrators soon found themselves in a situation where they were supposed to be responsible for both ensuring that no spam reached their customers and that all of their customers' email reached the intended recipients.
Gmail came along and decided that, because they were operating "at scale", they didn't need to play in the same ecosystem. Over the years, ensuring that a message lands in a Gmail user's inbox has turned in to an infuriating game of trial-and-error. Gmail can do this because they now manage between 40% and 60% of the internet's email traffic.
AT&T/SBCGlobal/Yahoo/whoever they are now seem to have recently penalized all of Linode's and DigitalOcean's IP space. I deployed several mail exchanges and didn't have any luck reaching any addresses managed by AT&T's network. And, again, there's nobody I can kibbutz with to resolve it.
AOL has been such a hot tire fire that I ended up blackholing any outbound traffic to them. I know that sounds drastic, but anytime a single AOL user clicked the "spam" button on an email from one of my customers -- who weren't spamming, newslettering, or anything else remotely skeezy -- it would generate an automated complaint from AOL to Linode, and Linode would threaten to suspend my account. I'd have to dig the relevant traffic out of the logs and respond back to Linode with a polite "AOL's full of shit, please stop listening to them". I explained the situation to the few people that were impacted by it and everybody got on with their lives.
Microsoft's outlookprotection.com filter has been a gigantic pain recently too. It's annoyingly capricious and, again, the tools just aren't there to resolve it.
Email delivery has been a bit tricky for several years, but the last year especially it has become impossible for small services. You have SPF, DKIM, DMARC, great, but it turns out that Gmail also dings you if you don't have ipv6 records arranged right or if you aren't transporting mail over ssl or or or or.
Email was designed as a cooperative system but the BigCos have carved it up and are working hard to ensure that if a message doesn't come from one of their networks, then it's immediately suspicious.
Sendgrid isn't much better in this regard. My network received a flood of spam from them, with legitimate traffic mixed in. I couldn't block them without complaints and I couldn't not block them without complaints. I wondered if I was the only, so I signed up for their service and routed some mail through them for a few days to see what it was like as one of their subscribers. Turns out that Comcast and half a dozen other service providers hate them just as much and deliverability was around 79%.
The things you and others are experiencing, and that I experienced, are going to keep getting worse. The bigger networks are going to keep squeezing customers out of the smaller ones, breaking email bit by bit in the process. I hope Fastmail is able to grow quickly enough to keep sitting at the table.
Even with my private IPs I've had random anti-spam services just decide to list all Sendgrid IPs.
Spam filters have significantly advanced beyond simple IP checks.
My newer devices get their email proxied through a proxy I host. It was a dumb mistake on my part to tie myself to so tightly to a SaaS vendor.
As far as I can tell, it simply is not possible to buy a new domain, pay for a email service and have mails delivered straight into people's inboxes. After all, if you would be able to do that then so could the spammers.
Ramp up volumes slowly. Make sure the recipients actually want the mails you send, and for all that is holy, educate your users that the "Junk" and "Trashcan" buttons are two very different things. People who repeatedly mark your mails as spam need to be blacklisted, no matter if they're paying customers. Period.
Oh, and handle abuse complaints right away. Don't be afraid to tell people they're not supposed to mark your mails as spam and to explain the consequences.
we have 97% reputation on sendgrid, and the only mail we send are a) customer requested password reset and b) customer requested notification for completed workflow jobs.
I don't know who that 3% is, but it is enough to get sometime filtered and I don't think we can ever win this
The 3% may very well be ignorant users who are using the Junk button as a way to remove the mails from the inbox (yeah, people actually do that), but for good measure, things like signup forms and password reset forms really should be protected with a captcha, as any form able to send any form of email whatsoever will be abused by bots trying to send spam.
well we do have a user existence checks
so is gmail by the way.
I'm in favour of anti-trust action against Google and Microsoft for their outright anti-competitive anti-spam practices, but that's going to be a long and hard up-hill battle for anyone attempting it.
Meanwhile, the rest of us will have to accept things for what they are and deal with them as well as we can.
If all we had was a few giant corporations with proprietary apps and proprietary protocols, that's not the Internet.
As I post every time on these email discussions, the difficulty of running your own email infrastructure is vastly overstated on HN. The more people do it, the better off the Internet world is for all of us.
I have spent many days debugging the issue, to no avail. Every other operator accepts my email, and my domain was never used to send out spam (I set up DMARC to make sure I am aware of all outgoing email). I imagine the IP might have been the issue, but I cycled through a few hosts without success.
Everything was configured correctly AFAICT, I could send email to all major providers, except microsoft. SPF, DKIM, DMARC were all set up. I tried many different configuration testers, and they all returned green on my domain.
Since migrating to fastmail (keeping the same domain), I haven't had any problem.
The thing is, I agree with you, the world would be a better place if everyone could host their own email. But at the end of the day, I need the ability to contact users using the major email providers, and so do I suspect many HN users.
I'm running an email server for myself since 1998 and in addition some low traffic email servers for a few small companies that I administrate and we have had no delivery problems so far, not even with Google, although I have heard from other people that run their own email servers, that Google occasionally puts delivered emails in the Spam folder.
I always keep an eye if some of our IPs may appear on any of the well known abuse lists, but so far that never has happened. I would guess if once there's coming spam from your mail server IP, it is negatively branded forever.
The thing with gmail is that this is true even even for legitimate email from long-known connections when the path is gmail to gmail!
Even worse, it can be true within a gmail-hosted corporate space, so email from your boss ends up in spam!
So it's nothing specific to whether you host it or not, it's just a reflection on gmail's spam false positive rate being particularly bad.
Start with a good IP address (static, not residential ISP, not on any block list). Search for sites that will help check this. https://mxtoolbox.com/ has many useful bits but there are others, try them all. Set up DNS correctly, forward and reverse.
Set up postfix. Tons of guides out there, here's just one list: https://www.linode.com/docs/guides/email/postfix/ Read the various guides but most importantly read the postfix documentation in detail. It is very good and has everything you need to know. Configure postfix to enforce all sanity checks that it supports during the delivery connection phase. This alone will cut off nearly all spam!
Install a local MUA (mutt would be my favorite) and you're up and running and should be able to send and receive, so test that thoroughly. Probably ending up in spam at this stage but should work. Mark it not spam in recipient. Note that if you had to register a new domain name, it'll likely be penalized as too new for a good while.
Set up a cert (Lets Encrypt) for postfix.
Configure SPF correctly. Read online guides but also read the RFC. Again, use online tools to validate your work. SPF is great.
DMARC, DKIM are more controversial and less clear-cut useful than SPF. Read about them. I do set these as well. Set the notification email so you get reports (which aren't that useful but still). Use the online tools to validate.
Set up Dovecot to enable IMAP.
Very common issue.
It sounds like this is a case of buggy implementations. You can have TXT and even SPF records longer than 255 bytes. The fundamental issue is that TXT records are composed of segments of up to 255 bytes each, where each segment has an 8-bit length header. As Resource Records (RR) individually, as well as DNS packets as a whole, have a maximum length of 65535 (because of 16-bit length headers), DNS supports TXT RRs of approx. 256 segments, 255 bytes each.
Both RFC 4408 (https://tools.ietf.org/html/rfc4408#section-3.1.3) and it's successor, RFC 7208 https://tools.ietf.org/html/rfc7208#section-3.3, require SPF implementations to concatenate the segments and process them as a single string. This means you should be able to publish a single SPF record of approx. 65025 bytes (256 x 255).
It's not too surprising implementations might be buggy in this respect. It's a rather esoteric aspect of the TXT RR specification. (If only because few people, including developers of DNS-related tools, bother actually reading the specification!) And unfortunately neither the popular rfc4408-tests.yml nor rfc7208-tests.yml test suites seem to have a test case for long TXT records. Even if a test had been included it likely wouldn't have caught many violators as it's actually a matter of the low-level DNS library providing the proper interface(s), many DNS libraries are deficient in how well they expose DNS record parsing or packet injection interfaces, and most SPF libraries would have likely integrated the test suite at a higher level that didn't involve actually making queries using a low-level DNS library, which can be difficult to do.
That said, RFC 4408 says that,
> SPF implementations MUST limit the number of mechanisms and modifiers that do DNS lookups to at most 10 per SPF check, including any lookups caused by the use of the "include" mechanism or the "redirect" modifier.
IME experience this limit is rather low and many, many SPF policies require more than 10 queries. (And if you need SPF policies larger than 255 bytes, you're probably skirting this or other similar limits.) When I was writing and deploying my own implementation I very quickly realized I needed to increase the minimum maximums, but choosing that value is a total crap shoot.
While this will mean that and mail sent will pass SPF, it doesn't play well with DMARC. The SPF pass domain won't align with the From header domain, so the SPF authentication will be ignored.
"But SPF alone doesn’t completely solve spam, we also need DKIM, and to a lesser extent DMARC. More on those later!"
Realistically, SPF doesn't do anything to solve spam, nor does DKIM or DMARC. All of these technologies address spoofing and give a greater confidence of the sender responsible for the email which can then be used in reputation engines.
Passing SPF/DKIM/DMARC say nothing about the message contents.
* no space between directives: "v=spf1 ip4:192.0.2.0/24 include:example.org-all"
* space inside a directive: "v=spf1 ip4:192. 0.2.0/24 -all"
* bad mechanism: "v=spf1 ipv4:192.0.2.1 -all"
* no mechanism: "v=spf1 192.0.2.1 example.com"
* = instead of : "v=spf1 include=example.com"
* Unicode, mostly dashes (but I've seen zero-width spaces too): "v=spf1 —all"
* Two SPF records for the same domain: "v=spf1 mx:example.com -all", "v=spd1 include:example.net ?all"
I recently wrote a short article on those, a tldr for setting up basic email authentication and anti-spam: https://www.usertrack.net/blog/stop-others-use-your-domain-e...
>Hostgator -> cPanel -> Email Authentication
...is ignoring 99% of the complexity and long term maintainance that DKIM implies.
And why bother? It's not like the mega-corp walled gardens care if you have the perfect mailserver implementation. They'll mark you as spam or blacklist you anyway, because they can, because it's easier, because it's a job someone's doing for profit and you don't effect that.
No. An SPF record is enough for determining if mail is legitimately from it's real mailserver. DKIM is cargo cult nonsense for most cases. A sacrifice to the google/microsoft/apple gods to hopefully earn their favor, but one that is almost always ignored anway.
In my case, the mailserver is only used to use a custom email address and forward the email, so deliverability was not an issue but other people using my domain name to send phishing emails. By making the fixes mentioned in the article the phishing attacks were stopped.
DKIM allows you to prove a message originates from your domain via a cryptographic signature. So any mailserver with the private key can send messages from your domain, it's not limited to a predetermined subset.
In different situations one can be more useful than the other. It's also recommended to implement both, as one or the other might fail for various reasons.
It's also worth noting you really should implement DMARC as well, which lets you spell out a policy for what recipients should do if SPF and DMARC fail, including if they should notify you and how.
No chance of getting through to their tech support, especially since these types of email are usually from a no-reply@ address and the people manning the phones are not the mail server admins anyway.
Basically DMARC needs either DKIM OR SPF to pass to consider a message passing. Being no SPF record will ever include (forwarders aren't allowed to originate messages from the domain) that means DKIM has to pass.
DKIM works by signing the message body and selected headers with a private key, while the public key is published to DNS. This allows a receiver to fetch the public key from DNS and validate it was really sent by the sender. It should be clear at this point that you'll be able to forward a DKIM signed message as much as you want as long as you don't modify the message when sending. If on the other hand your forwarder modifies the headers (including subject) or the body DKIM will fail.
In other words, if your going to use a forwarder, make sure that forwarder will forward the message as is. You can add headers, but don't remove or modify any existing headers and don't modify the message itself.
DKIM asserts that the message contents have been sent by a particular domain (or domains), that domain does not necessarily relate to either the envelope sender domain, or the domain the user sees in the From address. Again, an email from potus@whitehouse.gov can have a perfectly valid DKIM signature from phisher.com
DMARC is a policy layer over the top of these, which ties the domain the user actually sees in the From address back to those used in SPF and DKIM. This is where the potus@whitehouse.gov email starts getting blocked. Neither the SPF nor DKIM of the email are "aligned with" whitehouse.gov, and we can start to do something about the spoofing.
Or does spam appear in any popular messaging system anyways because it is such a strong selective pressure?