Self-hosted email is the hardest it's ever been, but also the easiest
vadosware.io
vadosware.io
After self-hosting my email for twenty-three years I have thrown in the towel - https://news.ycombinator.com/item?id=32715437 - Sept 2022 (628 comments)
Email Done My Way, Part 0 – The Journey - https://news.ycombinator.com/item?id=32719956 - Sept 2022 (10 comments)
Almost none of the things linked there are run by serious isps that are responsible for other peoples' mail in quantities of tens of thousands of inboxes or more.
Resist the urge to reinvent the wheel just to be a contrarian.
Edit: yes, I see the issue, the mention of postfix+dovecot is jammed together in the same small paragraph as the oligopoly. It's as if they are being discussed as the same thing. The oligopoly is clearly google, office 365, etc. One hundred percent in agreement there.
> Almost none of the things linked there are run by serious isps that are responsible for other peoples' mail in quantities of tens of thousands of inboxes or more.
Software doesn't have to be run by a large ISP to be worth running for people. Even at tens of thousands of inboxes (which is something post people don't need).
Good software is out there for running self-hosted email and people should know about it.
[EDIT] The post must have been unclear, I've edited it so hopefully no one will think I'm bashing postfix/dovecot.
I've even tried to make them easier to use in the past[0], but gave up when I realized that there was better software out there that completely sidestepped the problem.
I’ve had my own domain since the 90s and self-hosted at one point out of my dorm room. I continued for years after, but I gave it up before I ran into issues with deliverability. It was largely due to issues from the other side of the equation: Too much spam in my inbox and not enough time and motivation on my part to keep fighting it. I noticed my throw-away gmail account had pretty good spam filtering, so when moving my Mx to gmail became an option, I threw in the self-hosting towel.
EDIT: Again, with perspective from the other side of the spam fight, I wonder if anyone self-hosting on a non-residential IP that is having almost ZERO of their emails go through successfully has their DMARC set up properly. I use gmail for my own personal domain, but Microsoft 365 setups I manage for companies all use the strictest filters for emails without a tight spf rule (and a pass) and not using DKIM. Those are just not optional these days, unfortunately.
Then it failed to keep up in the war against spam and it became completely irrelevant and postfix took its place as the sendmail alternative/replacement.
Postfix was considered robust but reliable without security holes.
And then at some point postfix took over qmail as the default.
Spam was never the issue for us - tieing things in with filtering was not a problem. I think the bigger problem for qmail was that because of the licensing, you pretty much had to patch qmail, which meant you needed developers happy to effectively maintain your own version or at least patch sets. If you were building your own mail infrastructure (we were - we ran a webmail provider until we sold of that bit), that was fine and it was a good scaffold while you customised what you needed, but if you just wanted to run a mail server and it wasn't your core business it very quickly stopped making sense to deal with that hassle.
It's a real shame, but a good example of how license choice can affect things...
This is - indirectly - exactly what I was referring to. Most of the patches people had to apply to qmail, in order to be a good netizen, had to do with spam either directly or indirectly.
First because the design of qmail made it very difficult to stop backscatter by immediately failing undeliverable mail - out of the box it wanted to send a bounce for even the most obvious of errors. None of the patches for this were great. This made even running qmail as an incoming mail server only difficult and risky.
And then of course the various anti-spam tech came along like rbl and the now-static qmail source had no support for it, and it was another place where you had to reject early.
And this would all probably have been fine if the licensing hadn't let it sit for so long without a real fix to the fundamental architecture, with competing patches and so on.
Since the licensing issue was resolved a long long time ago (more or less) a string of maintainers have popped up to keep it alive with another name for a year or two at a time but I can't imagine standing it up fresh now.
Absolutely agree on the strengths though, it's almost certainly why it was so successful at scale for a while. Postfix feels clunky to me by comparison.
The whole point of self-hosting is precisely so that no entity runs tens of thousands of inboxes. If it's good enough for your personal use and the few people you're willing to help, then it's perfect.
Both are terrible for anything but personal or maybe even small business use; they're easy to set up and run, but inflexible compared to a manual install. I highly recommend them for running a personal mail server without fuss though, unless you're interested in manually maintaining your email config.
If you look at Haraka in the list, it is used by craigslist to deliver a lot emails every day, they switched to it from postfix to save cost.
I know Haraka is smtp only, however by using ZoneMTA like what wildduck is using, it will be a complete email solution.
On recommendations here, I switched to it 2 or 3 months ago from Gmail, after Google announced they were going to start charging for Gmail on custom domains. I'd no previous experience at all with running my own mailservers and had always been put off by how complicated it seemed.
Even with a non supported configuration setup [running the mailserver for my domain on a separate VPS from the domain itself] MIAB setup was pretty easy. I had it installed and was up and running in no time. I then spent a couple of days setting up the various DMARC MTA-STS policies etc. [most of that time involved looking up what they were and why I needed them!] and [after testing with an unimportant account] migrating the rest of mine and my family's and friends' emails across from Gmail. I was able to import half a dozen accounts from Gmail with about 8GB of emails between them, using ImapSync [1]
So far [fingers crossed] it's been almost painless. Giving the lie to the commonly held belief that running your own mailservers needs constant hands-on maintenance. My MIAB setup runs quite happily on a $5/month Linode [I checked the IP was not on any blacklists, before I started] and the whole thing has been pretty much 'fire and forget'. MIAB sends me weekly reports on what's been happening. Software updates are a one-liner in the terminal and I've had no trouble whatsoever with Google, Microsoft etc. as regards the mail not getting through.
If you've always hankered after running your own mailserver but [as I did] thought it was too difficult and required too much 'hands on' tinkering, I heartily recommend you give MIAB a go.
DISCLAIMER: I run a couple of dozen email addresses on 3 or 4 domains on my MIAB. Probably sending and receiving a few hundred emails a week between them. YMMV if you're a heavier user than this.
It is great for having a project have a completely different identity. I want to get a used to a few upgrade cycles before I use it for a "main" email address - but I concur the ease of set-up and use.
If it is for a separate project - normally only receiving emails are vital, sending has been ok even with low volume behind a residential IP.
You said - > I then spent a couple of days setting up the various DMARC MTA-STS policies etc. [most of that time involved looking up what they were and why I needed them!]
Can you tell me how you did this? It'll be a bit help!
>Can you tell me how you did this? It'll be a bit help!
https://www.mailhardener.com/tools/mta-sts-validator
EDIT: have you tried the MIAB forums? They're quite a helpful bunch:
Full details here -
https://discourse.mailinabox.email/t/how-to-unblock-from-mic...
I believe the difference between people who say "email self-hosting is dead" and people who say "email self-hosting is trivial" is probably the volume of mail sent from their domain(s).
At my busiest, I was sending dozens of emails per day, but they were all from my work account. My personal account is pretty close to recv-only and probably averages an outgoing message count in the single digits per week. How can I reasonably keep an IP/domain reputation score fresh/warm if I send such a low volume? The answer is I realistically can't.
Self-hosted email remains an extremely difficult and time-consuming endeavor unless you happen to have some good luck, it seems.
Unless you can get hosting at an ISP that does not sell low budget, commercial virtual private server, virtual machine or dedicated hosting to random people with $20 and a credit card, this will be an ongoing problem.
I'm open to recommendations for clean IPs/providers.
Run your own email servers, but relay through there for better delivery?
Maybe this is what we need more of -- A class of mail system participants who exclusively maintain trusted IPs and do the legwork of trying to get through the gnarly systems set up by the other large email providers.
[EDIT] - "forward" -> "relay" for clarity
I don't know the solution but I know it needs to be discussed.
Sorry if it's confusing (email as a whole is kind of confusing at first glance) -- relaying SMTP through a different server is kind of like using a VPN or a proxy.
Services like ImprovMX and Mailgun can be used for aliasing emails, but they can also just be used to deliver your emails for you (no aliasing to a new address).
The idea is to send all your email (like web traffic through a VPN) to those services.
The big problem with this, is that gmail considers all messages forwarded to it as coming form the forwarder, not the original sender, and therefore spam harms the reputation of the forwarding domain.
But I don't understand how to configure mailgun to relay messages to gmail instead of forwarding them? Can you explain?
Sorry unfortunately I don't use Mailgun -- I can't explain the setup (I didn't suggest it)... But if I search I see this:
https://www.mailgun.com/products/send/smtp/free-smtp-service... (Ignore the "free" bit I'm sure it ends up costing something!)
https://help.mailgun.com/hc/en-us/articles/360012360833-Why-...
I'm more thinking in terms of emails going out -- not incoming email.
Cf. https://www.dmarcanalyzer.com/arc-is-here/
Postscript: Neither the Improvmx nor Mailgun sites have easily found information about ARC. It's a relatively complex and not widely used technology; nonetheless, the point I made about its adoption being in their interests stands.
So far people have suggested:
- ImprovMX (https://improvmx.com)
- Mailgun (https://mailgun.com)
- Postmark (https://postmarkapp.com)
Commenting for visibility: as any others come in I will update this comment (until I can't, anyway) and the article as a reference.
The only issue I have with a Postfix setup is providing an easy-to-use out of office auto-reply facility - I have tried the manual approach and also some scripted solutions but they don't seem to work reliably and, inevitably, I get comments from colleagues (it's a small, family-run business), that they just want to be able to turn on an auto-reply in Outlook (which is used as a front end for multiple inboxes on different mail services) and for it to 'just work'.
For example large part of IT professionals which contacts our support (https://poste.io) don't get difference between SMTP envelope and from/to headers.
Heck, like 95% of open source fails in that regard. Hell, anything Linux fails (even though progress has been made, it falls halfway short still)
You'll usually only learn about this after getting bitten by a bug in an auto-updated dependency and at that point you'll learn to manually pin your dependencies and use commands such as `npm ci` instead of `npm install` in your build pipeline.
As such, navigating around the NPM world is anything but easy. There are razor sharp edges and footguns lying around everywhere, just waiting for you to use them.
That's a dubious remark. Even proprietary software most of the time fails at being easy. Easy takes great design and most software out there falls into the average.
Take Blender which was terrible for years and years, but changed happened because the projects they did allowed them to work with professionals which could point out shortcomings.
Or GIMP, its UX is terrible still afaik.
Or lets install Linux, as long as it takes a considerable expertise and doesn’t come preloaded, people won’t switch.
ProtonMail is about the single worst experience I’ve ever had with Big Email. I really do wonder how many people they’ve harmed but get away with it because the happy path works for most people most of the time.
The worry I have with something like Gmail or Outlook is that they can just take away your access to your inbox without reason or recourse, but it’s never actually happened to anyone I know. It happened with ProtonMail though - I set up an account for a family member, updated some of their online registrations to point to it, then a couple of days later it was locked out permanently.
Thankfully they hadn’t got to the point of using the account for anything they cared about yet, just a couple of big retailer mailing lists (they’re not the most tech literate so move very slowly with things like this). Literally it was receiving mailing list emails from two well known, non-shady retailers and that’s it.
Password not forgotten, No way to have the account unlocked, no way to find out why it was locked, no way to have the account deleted, no way to get access to repoint the accounts using that address because now ProtonMail owned those other website accounts (not your email not your account).
Pretty much just [you can’t have your account back, you can’t find out why, you can’t appeal, and every email that ever lands here in the future belongs to us now so go fish].
They are by a long way the worst experience I ever had with a provider of any online service, so much so that I’ve since moved my own e-mail that was there back to outlook before some arbitrary spam caused the same thing to happen to me. Never again.
- No details or evidence to back them
- Happened to you more than once
Sure. We'll just believe that ProtonMail locks accounts randomly, for no reason. Or was spam the reason? If that was true, everyone would be locked out because, well, everyone is getting spam.
https://www.reddit.com/r/ProtonMail/comments/ic19q4/locked_o...
A quick Google search returns tons of results from ProtonMail's subreddit of people complaining that ProtonMail locked them out of their account for unknown random reasons.
This is a curated subreddit.
Fair enough. You can google though. There's a wealth of complains of people seeing their accounts locked.
If you don't have time to run a Google search, you are free to read ProtonMail's own docs, specially how they describe how they lock accounts based on heuristics and how they recognize the occurrence of false positives.
The evidence would be the e-mail account which PM now hold and aren’t going to give over.
I didn’t say it happened more than once, if anything I was saying once was enough and it was a risk I didn’t want to take ever again.
Also, I hadn’t realised this before, but a reply further down the thread has linked to many instances of this happening on reddit.
> They are by a long way the worst experience I ever had with a provider of any online service, so much so that I’ve since moved my own e-mail that was there back to outlook before some arbitrary spam caused the same thing to happen to me.
Well, I misunderstood you here. Wrongly assumed you meant it happened to you again.
All of that said, if it says nothing else it at least suggests that larger providers seem to have a better grasp of how catastrophic being locked out of one's own e-mail can be - PM really don't seem to understand the responsibility they bear being a provider of such critical infrastructure. If this were happening in this same way at larger providers like Google and Microsoft, I'm quite certain there would be heavy regulations by now.
The account was just.. gone. That's what was so shocking about it - I realised pretty much immediately at the time that there would be no way to recourse a locked account because any means by which you could prove the account would be yours (other than having the password, which we still did, but which was not acceptable to unlock the account according to PM) is inside the inbox itself. More than that - they weren't asking for more information. They simply said it was permanently locked and nothing could be done.
This has happened to others too, as has been posted elsewhere in the thread.
I don't think it's meant to solve e2e encryption.
Edit: or do they perhaps not store sent messages (that is, they have to queue them, and attempt resending on failure, but beyond that), and/or encrypt incoming ones with the user's public key upon arrival?
Edit 2: apparently the stored messages are indeed additionally encrypted by ProtonMail [1]. That looks useful.
Edit 3: now I wonder why not to do that with OpenPGP too.
Edit 4: looked around, apparently some do that. [2]
[1] https://proton.me/support/proton-mail-encryption-explained
[2] https://jnphilipp.org/posts/auto-encrypt-all-incoming-email-...
I don't consider them "Big Email" just by literal meaning (they're very small)
> The worry I have with something like Gmail or Outlook is that they can just take away your access to your inbox without reason or recourse, but it’s never actually happened to anyone I know. It happened with ProtonMail though - I set up an account for a family member, updated some of their online registrations to point to it, then a couple of days later it was locked out permanently. > > Password not forgotten, No way to have the account unlocked, no way to find out why it was locked, no way to have the account deleted, no way to get access to repoint the accounts using that address because now ProtonMail owned those other website accounts (not your email not your account). > > Password not forgotten, No way to have the account unlocked, no way to find out why it was locked, no way to have the account deleted, no way to get access to repoint the accounts using that address because now ProtonMail owned those other website accounts (not your email not your account).
This is pretty terrible, but I'm coming at it from a oligopoly-break-up angle, and hadn't heard any such stories about ProtonMail.
I'll try to update the post to serve as a better example, but I hesitate to do so without at least suggesting something (other than self hosting, obviously) that is a similarly easier option. What do you recommend instead? Fastmail?
FastMail and Mailbox spring to mind as the premiere ‘alternate’ e-mail hosts. I’ve heard decent things about Tutanota but have never used them so can’t speak to it.
Yeah, Australia's policies regarding the internet are fully foobar'd. Removing them from the listing. Nothing against the people of course but it's off my radar as even a place to visit for this reason (not that they need my tourism).
I’ve no experience of FastMail, but this same thing could happen just as easily with any provider. You’re basically left to rely on the opaque internal policies of the email provider to prevent your online identity being revoked, which is one of the many reasons why so many people would rather self-host in the first place.
Sorry I can’t help you out with my two cents here. I’ve never heard of such things happening with FastMail, but then I hadn’t heard about it happening with ProtonMail either until it did.
Self-hosting is the end-goal but it looks like the in-between is still murky as well.
Maybe what it really takes is people springing up to fill the gaps of a closer-to-self-host solution (so you have full control) w/ deliverability auto-solved (automatic purchase of some external relay or something). Like a service that purchases a VPS in your name, sets it up, then essentially disconnects (or offers to manage the service for a fee).
Of course there's stuff out there like Cloudron etc so maybe that's part of the way forward as well -- but I wonder if Cloudron could ever really reach a completely non-technical user.
It's not obvious so I'll add that too.
I pay them for a service, they provide said service. I think there have been a handful of outages, but I only noticed them afterwards from the HN front page :D
The Masked Email system alone they added a while ago has been a life-saver, I sign up to all mailing lists and weird online shops using it.
I'm sorry this happened to you, but I find it weird that you make this out to be a problem specific to Proton. It happens to people on all major services, it sometimes even makes the front of HN. Do we have any evidence that this happens at a significantly higher rate at Proton than at other providers?
It’s like saying “it’s easier than ever to build a car! But you can’t drive it on the street”.
I would say deliverability is the most important feature for email.
You're not wrong, but the thing is, the deliverability problem is not completely broken.
I don't have a problem sending emails with my self-hosted stack with DKIM, DMARC, and SPF set up. That said, some people do.
I lightly monitor my domains to make sure they're not on blacklists and have figured out how to get things working over time.
The battle isn't lost yet -- some people have problems and others do, but discouraging people from self-hosting is not the way to fix it.
My main point is that while deliverability can still be hard, many of the other things that made sending email hard (configuring postfix, dovecot and DNS correctly) have become drastically easier.
I self hosted my email for more than a decade but I gave up when the time spent was just too much.
Oh not much now -- to be honest with you 99% of the time when it's an issue it's because I changed something on my side.
Maybe I got lucky with the IPs I have (I use dedicated servers), and I've held them a while. What was jarring about the original post by Carlos is that he did all that AND he's been doing it 23 years and it was still a problem.
It made me recall that one of the things I struggled with a long time ago was just setting up Postfix/Dovecot correctly. These days no one should be running into that.
> I self hosted my email for more than a decade but I gave up when the time spent was just too much.
Unfortunately I don't doubt this assessment. I just want people to know that if difficulty/technical work was a blocker, it is now mostly not (with the news software out there).
As other people pointed out, there are services for improving deliverability that could fix the other (bigger) issue, and hopefully people could consider doing that going forward.
At the end of the day people have to do what's right for them of course but I just want people to know that things HAVE gotten easier. It's the big companies that are making it hard now.
Deliverability really depend on solid reputation of IP address blocks, including reputation of your neighbors. Ideally you want an IP near other IPs who also send high volume of non-spam.
I stopped a couple of years ago, and moved my mail to my ISPs servers (they let me use my own domain). I switched because of the hassle of keeping the thing up, backing-up the mailstore and so on; and because I was moving home, so I wanted a service that couldn't be knocked-out by the move. Not because of any deliverability issues. I haven't had deliverability issues since about 2010, and those were probably because I couldn't be bothered with DMARC.
My ISP is a niche outfit, targeted at nerds; their mailservers are Postfix/Dovecot, which suits me. They support Sieve, and they have a control panel with various knobs and buttons. As an ISP they're a bit pricey, but for email hosting they're cheap.
> At some point your IP range is bound to be banned, either by one asshole IP neighbor sending spam, one of your users being pwned [emphasis mine], due to arbitrary reasons, by mistake, it doesn't matter. It's not if, it's when.
In other words, at least some of the time, the one of his users' accounts had been compromised, and as a result, his servers were actually sending out spam. But rather than take responsibility for detecting and blocking outgoing spam on his own servers, he's blaming The System for doing exactly what it was meant to do.
The point being that outgoing spam detection won't save you, by the time you are pwned it's too late.
My solution is to have a script that scans the mail server log files and aggressively block ips of failed authentication attempts that are not quickly followed by a successful authentication attempt, and get immediate notification when there is a successful authentication attempt from an unexpected country (I think it's not the same bot that scans for vulnerable accounts and sends spam so you have maybe an hour to react).
That works well for a small user base.
I've encountered a few obvious login attempts like that, but since they come from a broad pool of IPs it's not something fail2ban can easily handle without collateral damage.
I've been running a non-profit ISP for about 23 years with a few friends. We've always been doing this from our own IP Space (/20) in RIPE. Even when we had problems sending spam (compromised user accounts, compromised php websites) and we ended up on a blacklist, we usually could get removed pretty fast.
For a few years our mail system would sometimes generate late-bounces, that is accept a mail on the incoming MX only to then figure out that it actually cannot be delivered later on and generate a delivery failure notification mail. Not a good situation.
That got us into some trouble here and there. But even that could easily be unblocked again.
When we finally managed to set up a new mail infrastructure (2 year project cause it's a hobby) we set up new outgoing SMTP servers which cycle through multiple IP addresses. There was exactly one ISP (Deutsche Telekom T-Online) that was not accepting mail from some of these IPs. One mail and a turnaround time of abour 12hrs later this was fixed. Gmail or Hotmail/live.com/Outlook never had any problems with deliverability. Even with a few users forwarding all their email to their gmail accounts including the spam that slips through our filters. That might mean that a single mail would not be deliverd, but other users never suffered as our outgoing IPs are not being blanket-banned.
There's one residential ADSL provider that has a blanket ban on one of our outgoing IPs. There's no way to get that resolved because their mail infrastructure is unmaintained and nobody is reading their mail. Common problem with that one ADSL provider, googling their name shows other people have the same problem. shrug We just use a different outgoing IP for them.
No DMARK or DKIM setup at all for outgoing mail.
So I wonder, what really makes the difference in experience? Is it just the fact that we have a decent sized IPv4 Network in our name as PI space?
:)
I don't track the time but it's within a rounding error of no time at all.
I set up SPF for my domains when it became popular many years ago, took maybe a few hours.
Some years later I set up DKIM, that took a bit more time over a weekend. Some time later I set up DMARC as well.
I haven't had to make any changes since then, it's been at least four years. No problems of any kind. It's absolutely worth it to own my own email, no question.
But not accessible at all to non technical people.
Not much, maybe on hour per month at most. I've been running Postfix on a dedicated server for 20+ years and don't need much time dealing with it.
But I've made sure to keep the same ip address when upgrading servers (possible with Hetzner) and besides configuring postfix to reject various connection attempts I do run spamassassin. Resulting in about one or two spam mails per week (moved into their own box by procmail in the filter chain) which I handle once a week by training spamassassin (takes just a minute).
That's only because true it's the one feature the oligopoly is withholding from self-hosters.
It's like saying "the most important feature of an operating system is the ability to multitask." Well, yes... but I'm glad we don't have to worry about that anymore when comparison shopping.
I'm personally hoping the magic of free market competition drives down these costs (at least), and in the meantime we can know how to steer clear of the centralization.
And the free market competition will ensure that those providers make stopping spammers (at the source -- signing up for an email service) their priority.
Spammers still use SES! Just the other day I found an SES IP blocked by Spamhaus. I'd argue smaller providers can be even better than AWS at preventing spammers from signing up -- they just need to play like smaller providers and actually vet/accept through referral/etc.
Smaller providers also must be much more careful about accepting spammers, so this helps keep them sharp -- and hopefully we get to a bigger smorgasbord of choices.
[0]: https://vadosware.io/post/even-amazon-ses-has-ips-blocked-by...
I usually think of just using SES whole sale but these days they have receiving too... Might be a good way to hide from the other big clouds that are a bit more entrenched in the email space.
Life is hard so let's surrender and use WhateverMail service? That's a very pathetic way, that's how corporations thrive and screw us later.
You shouldn't stop doing something just because someone did it for 20 years and failed. Some people learn - others are just getting older. That's defeatist mentality.
I've personally selfhosted mail for 20+ years as well and see no reason why I would stop it. And yea, I use postfix + dovecot. 50+ domains. 2M messages a day on some domains. Everything works and costs $0. You can calculate yourself how much services like Postmark will charge you for similar volume.
It's not a brag but attempt to show something is absolutely doable as contrary to someone who "did it for 20 years and gave up".
I love ideas as much as the next guy and serverless email is kind of floating out there:
https://medium.com/schibsted-engineering/building-a-serverle...
https://github.com/arithmetric/aws-lambda-ses-forwarder
https://github.com/0x4447/0x4447_product_s3_email
It's possible to build it, but the problem is that you still have the same problem of deliverability. Obviously it works fine/great for receving emails though.
> Idea is how cheap can it go for personal inbox with all the features denied by the superlative pricing plans
It could get really cheap, but would people buy it? I always wonder if price is really the limiting factor for self hosted emails.
Zoho is already QUITE cheap: https://www.zoho.com/mail/zohomail-pricing.html
Maybe this would work as a business, but it's a bit questionable to me.
Arguably, a http based email protocol would be a good thing. Not only would it enable serverless, it would also enable all the features of http/2&3, like compression, multiplexing and better pipelining. Someone would just need to define it. The downside would be that VPS providers can not limit mail spam from their servers by just blocking outbound traffic on certain ports, so you'd find all their IPs to quickly be blacklisted. So the protocol should ideally somehow also offer some form of spam detection/prevention that works better than simple IP blacklists to prevent that.
Great to hear they're a solution for this though -- will update my article to reflect them next to ImprovMX.
> I still want to use a different provider to send email newsletters, why can’t I do that?
You can. There is no requirement to use Ghost’s built in newsletter delivery feature. Before we released this feature, thousands of people sent their newsletter using all sorts of other services such as Mailchimp, Sendgrid, Convertkit, and many others. You can easily sync your members database to an external newsletter provider via Zapier, or by following our detailed integration guides.
I don’t know anything about self-hosting, but I’m curious to know if there’s no place for self-hosted email servers when it comes to integrating with Ghost.
Yup, "transactional" emails can be sent via a variety of methods, but "bulk" email can't.
I think the differentiation between transactional and bulk email is also bullshit (and a result of a broken underlying ecosystem) but that's a different story.
> I don’t know anything about self-hosting, but I’m curious to know if there’s no place for self-hosted email servers when it comes to integrating with Ghost.
Just to be clear, transactions emails are fine, but bulk email is not.
I personally think Ghost just didn't want to unleash a wave of under-prepared people into the difficult problem of email deliverability.
Letting people use SMTP servers would immediately lead to people using their own postfix setups, GMail, transactional providers that offer SMTP, etc -- probably instantly leading to many people being banned for unintended use (there's that bullshit again), and people either being angry at Ghost or filing issues against Ghost about email not working.
What's unfortunate is that they didn't really shim it out so it's easy to replace Mailgun... Which is why I'm embarking on this quixotic quest at all. It may end up being helpful down the road though, API aggregation has been a common theme for my ideas lately.
To be fair, this was over 5 years ago. Maybe they have improved their ability to keep spammers out of the free tier service.
1) Clean IP addresses. This can be hit-and-miss so you might need to move service providers since it seems like some (OVH, Linode) seem more likely to have been used by spammers before. We have also held the IPs for a while. We have servers at 3 different VPS providers. 2) Making sure you setup DKIM/SPF properly. We see plenty of mail from others that doesn't align properly - accidentally or by ignorance - and it can mean very different things: GMail is quite forgiving but Yahoo (and their children) are very strict about alignment 3) Using the various tools available like mxtoolbox and the various blacklist checkers. We had a problem the other day because the parent domain was linked with Vidahosts (an old email provider) so even though the subdomains were clean, one domain didn't care and strongly greylisted us (some got through!) 4) Making sure your postfix installation is up-to-date and not fiddled with too much. Postfix includes features to retry greylisted mail and to backoff if it gets a "Too Many Requests" response from the other end. 5) Use https://www.mail-tester.com/ to check a mail sent from your domain, it can reveal all kinds of things you might not have set up correctly. 6) Understand your DMARC policy - even having one is a good idea even if it is p=quarantine
Most of all, before complaining about the large providers, think about how you would deal with SPAMmers and you might understand why they do certain things. Blocking ranges of IPs might seem harsh but is a much quicker way to deal with higher volumes than trying to filter each IP; more email can be better than less. Sending very low volumes doesn't allow you to build up a good reputation, especially since you can not trust by default on the web; greylisting (retry after 1 minute) is a reasonable way to block a lot of lazy SPAM attempts while allowing properly configured servers to get through.
The idea of specifying "bad actors" is a little suspect but is already how the system works implicitly (and explicitly)... Requiring fair email delivery from standards-compliant businesses with a clear path to escalation at the government level might be amazing.
For all of GDPR's warts and criticisms it's almost surely a net positive for consumers, so maybe this could be another piece of legislation in the same vein.
Of course I'm not so naive (or rather I'm too cynical) to believe that governments are simply doing the bidding of the people, but I don't think I mind them mining this new revenue stream a bit in an enemy-of-my-enemy-is-my-friend kind of way.
email systems are retarded. They throw all kinds of acronyms, apps, subapps, etc at you. Why isn't there just some damn app that you can to to a website, click on download and install, and it is up and running? Everywhere I read, I was cautioned it was difficult because you always have to upgrade security with new patches, for example. Why are they even bothering me with this knowledge? Why would I have to do it? Can't people doing email apps figure out how to continuously update everything so I don't have to think about it?
I simply do not have the time to figure it all out. I have other responsibilities taking up my time. I have a computer science degree, and have worked on all kinds of apps, but I can't figure the stuff out, how is somebody else going to do it "easily"?
I've always thought someone could automate the entire thing, but no.
And so, I got tutanota. Less than a minute for getting an email up and running. It should be that way to self-host, like any other app in the world. I didn't choose proton mail because it was WAY more expensive. I have multiple emails, and don't want to use aliases, heck with that. So buying multiple emails is WAY more expensive on Protonmail.
If there is a simple one-step email hosting app, they sure as hell hide it very well. Because believe me, I looked.
For those who want to explain why, I have a computer science degree. I can sort of guess why. But the reasons don't interest me. I just need a one-step self-hosted service. End of story.
I'm not mad or bitter. I'm just being direct at outlining my experience and frustrations. But I'm ok now, not frustrated anymore. Except when I think back on it....like now. :)
Somewhat related:
This is not new, by the way. Preventing spam was the original application for PoW way before the blockchain existed. First proposed in 1997 with https://en.wikipedia.org/wiki/Hashcash
The downside is that it can be tricky to correctly tune the difficulty factor for a wide variety of e-mail sending hardware with wildly varying computing power available.
EDIT: I'm assuming here that, if you use e.g. gmail, you would be suspicious of any unsolicited attachments and not sync them to your personal devices.
EDIT: If you're not worried about this happening once, what about if it happens once a month for several years?
But if you host your mail on your cloud server, or even in your home, wouldn't you feel especially worried if you saw that you had received such illegal material? What do you even do? Do you wipe the server? How many logs do you keep just in case? Do you keep them for 5 years? 10?
If you feel even slightly differently about those two options, that's the concern that is still unresolved for me. If you honestly don't, after having thought about it, then I would consider that a valid point of view here.
One reason I would feel safer with larger providers is they would be better-equipped to block the content, better-equipped to deal with the authorities, and liable for the infrastructure itself. If the cops come to confiscate or investigate some of the hardware, they're going to be dealing with Google. They may also deal with you, but if you're savvy enough not to download attachments you don't expect to receive, you may have SOME buffer there. If it's clear you are hosting your own mail, you have absolutely no buffer. It's very likely to get all your home/cloud computing equipment seized. Maybe THAT's the part that's not realistic?
Why would you be worried? In Law the main thing that matters to make a case against someone is demonstrating intent, and if you had no intent in receiving the material, not sure why you would have to care about that kind of risk.
And re: client devices, with a provider you don't control, you at least have the option not to download or access content you are not confident is safe. If you self-host, just accepting the message is storing the content on your device.
I can certainly recommend it should you want to self-host from home.
It's real simple, if you don't own the hardware, if it's not physically available to you to go whack with an axe, then you don't own anything, you're just renting and entirely at the mercy of whoever you're renting from.
Also, "owning" and "being able to walk over and physically whack it with an axe" are not the same thing. Servers can be leased, and there is something called "server housing" which breaks your analogy
Saying you own your leased server is the same as saying you own your rented apartment or leased car.. It's not yours if it's someone elses, in the same way the movies on netflix are inherently less yours than the movies on your laserdiscs, dvd's or blu-rays.
1) Free email provider (you have not recourse for anything)
2) Paid email provider (a little better)
3) Free/paid email provider with your own domain (gives you disaster recovery without the provider's cooperation)
4) Running your own MX but doing delivery via another provider (perhaps a bit more privacy)
5) Running your own MTA/etc. on a managed host, e.g. a VPS
6) Running your own MTA/etc. on a colocated host
7) Running your own MTA/etc. on a host on your own premises
Even with the strictest definition of self hosting, i.e. (7) you still are subject to your ISP recording who you communicate with (and more since SMTP is not consistently encrypted) and are still subject to law enforcement accessing your data. (6) and (5) mean you do need to trust your hoster to not look at the data on your instance but at least for now that seems to be a reasonable asumption unlike for (1) to (4) where at least automated scans you don't control are the norm.
I won't go as far as to say that, unless you're your own ISP, you're not self-hosting, but I'm leaning in that direction, from having experienced the increasingly firm hold that ISPs have around my neck.. It was not always like this. There was indeed a time when a connection to the Internet meant that you got to participate on the same terms at everyone else. I want that level of participation back for everyone. Participation by every connected device should be the default, not the exception.
What webmail options are recommended to use with maddy? Some users want webmail as well, not just IMAP.
there is also things like roundcube which is fairly popular
The user is actively seeking the email, spam filters won't matter, and its easier to get out of spam once other email providers recognize its a legit service.
I host email on a Hetzner dedicated server, and it arrives in inbox on gmail, not Outlook though.
But that didn't affect anyone really, its OK.
I use postal, but Maddy looks interesting.
- ProtonMail - FastMail - Mailbox*
Anyone uses or used to use Tutanota? That’s the only privacy-focused email provider I know of aside from ProtonMail.
And why is having a data center in Germany a selling point for some of these email providers?
I guess having data center inside EU could be a plus for EU citizens?
The EU, and Germany especially, have strong privacy regulations and laws to protect their citizens. Having a datacenter in the EU does not mean that your data can't be accessed by authorities at all, but it's harder than in other countries. At least in Germany, a judge-signed request with a specific reason is needed to get data from any email provider. Mailbox even publishes a transparency report[1] with details about said requests: in 2021 they just received 65 requests, 15% were incorrect. 61 of those requests were just about the contact data of the account owner, not even emails or any other data stored.
This could quickly be changing with proposed mass-surveillance "Chat control" legislation which would force all providers to scan the contents of all messages, emails and other communications.
https://www.patrick-breyer.de/en/posts/messaging-and-chat-co...
For a nominal cryptocurrency fee, my email server won't reject your inbound email. If I respond, you may even get it back.
The more you send me, the higher priority I'll flag it.
Must be using one of those incorrect regexp or something...
As a side not, taken who the author is, I had hope to have something responding to French requests :(
Most clients now special-case Gmail because they have all but deprecated POP and IMAP.
Push notifications, essential for seeing mail "in time", are a proprietary, inconsistent mess.
Receiving servers use all kinds of voodoo to determine whether a message is acceptable to them, much of which is powered by proprietary software and semi-secret block/allowlists.
Attachments are barely usable because Outlook uses a format that many other clients don't understand.
Large providers have hardcoded logic for marking messages from "trusted" senders like banks and online shops, so some senders are more equal than others.
Meanwhile zero meaningful progress in decades to address the protocols' basic shortcomings like lack of encryption, replies that don't involve sending hundreds of pages back and forth, ...
Let me guess, email is dead? RSS is also dead right? Literally no one ever uses them anymore. You probably think that all businesses should just all switch to TikTok and communicate with memes and dance routines.
The OP was not giving a take on whether that's a good thing or bad thing, let alone saying that everyone should "switch to TikTok" or "communicate with memes", but rather simply commenting on the above hard to dispute fact.
You contradicted yourself.
> and either way you're still wrong.
Ironic. I guess you want me to take your word for it.
Unfortunately for you, your mockery doesn't hold up to any kind of scrutiny. Libsyn, a podcast distributor which I associate with more nerdy podcasts and competent users compared to e.g. iHeart, released user-agent stats across their entire field quite recently, back in 2021: https://thefeed.libsyn.com/193-alexa-play-the-podcast
Mobile apps not named Spotify, Stitcher, or Apple claimed a total download count of 12.6%. That 12.6% is largely composed of:
* 2.3% Google Podcasts
* 1.8% Overcast
* 1.3% Podcast Addict
* 1.2% Castbox
* 1.0% PocketCast
To the best of my knowledge, all of the above are centralized and normally fetch the RSS feed on a server instead of the app functioning as the user agent.
And so we can infer that at maximum, mobile downloads originating directly from RSS-fetching user agents represent 5% of the market for Libsyn. Desktop as a whole represented <15% of all downloads, and a huge chunk of that is going to be (once again) Apple Podcasts.
Furthermore, there are a ton of extremely popular "podcasts" (though they barely deserve the name) that are entirely centralized, being available only on one platform. This is a large portion of the market which the above figures don't reflect at all.
And so, on the basis of the evidence that is actually available, I pronounce RSS dead.
Email is dead, RSS is stone cold dead and buried under 100 ft of permafrost.
All mainstream browsers have removed RSS support. Virtually no major website still offers RSS feeds. When you see an RSS/Atom icon on a page today, it's either an old Wordpress theme that noone bothered to update, or some stubborn ideologue who insists that RSS is still a thing because there is a document somewhere that specifies it.
The dream of open syndication is over. Wanting it to be otherwise doesn't make it so.
This claim is as bold, as it is baseless, as evidenced by dozens of emails I send and receive each day, and by the fact that I stumbled upon this thread via RSS, which I use daily to receive almost all my subscriptions.
Yeah, except tiny sites like The Guardian, NY Times, CNN, BBC News, Stackoverflow and pretty much every German major news paper...
If you go around in a city asking people what a RSS feed is you will definitely will get more blanks than asking them about e-mail or facebook.
E-Mail is used a lot still, don't see that going away anytime soon either.
Sadly, yes. Businesses, and even many hackers, moved their email to the oligopolists; ditched RSS readers in favor of Twitter; switched from IRC to Slack; many people in organizations can't even be called directly, they're on MS Teams.
There isn't any such thing as "in time" for email, and there's no provision for "push notifications". I imagine parent is describing a web UI; webmail interfaces generally poll an IMAP server. Just like my desktop mail client.
> replies that don't involve sending hundreds of pages back and forth
That's not a problem with email; that's a user-education problem. Trim and contextualize.