How we send 22k emails every hour | Hacker News Reader

How we send 22k emails every hour

jitbit.com

114 pointsjitbit54 comments

Haha, this made me chuckle :)

> Here's a cute completely irrelevant story, feel free to skip. The phrase "Because Australians!" has actually become an inside joke/catchphrase in our company. See, Aussies and Kiwis (New Zealanders) wake up before everyone else (even before Japan). Basically Australians are the reason SaaS businesses don't have an outage/maintenance window. And a couple of times when we were about to put a server down for maintenance on a late Sunday evening (EU time), someone would shout "Wait! Don't do that!" - "What... Why?" - "Because Australians!" So after 4 or 5 times this phrase is now used as an ultimate "42" answer why everything has to be up and running 24/7. I even have a tiny map of Australia in my office. I (nor anyone from the team) have never been to Australia or New Zealand, but hey, we're always thinking about you mates [wink]

Australia is a good, almost-habitable place and the time difference is quite a trip. Sometimes talking over video chat with my buddy it's daylight in both the US and Australia and we both say "huh?"

FYI there is generally 3 hours overlap of working hours between the East coast of Australia and the West coast of USA. 8am our Melbourne time is 3pm San Fran time (the day before).

https://www.timeanddate.com/worldclock/meetingtime.html?iso=...

22k e-mails per hour averages just six per second — even the stated peak of 30 per second isn’t anything to write home about. If you go strictly by the book you are writing email to disk and flushing the buffers when the smtp server receives the email for relaying, and only acking reception when that has completed - and that gets into iops, channels, spindles, and other fun considerations that are often obscured by the cloud these days. But an easy optimization for a bulk sender is to forego that and use a less resource intensive method of tracking what hasn’t sent successfully and needs to be retried. But whatever method is used I don’t think it would really get interesting until you started sending thousands per second or higher.

I have to agree.

I built a custom email solution for a small business similar to the one presented in the article, with a bespoke newsletter creation/mailing list management/engagement reporting platform on top of it. It handles far more than 22K emails per hour, which is a trivial volume for even very modest modern hardware.

However, I do agree with the article that building a system like this can be highly cost-effective. At one point, my aforementioned client was spending nearly $75K per year to outsource this. They've been using the system I built for more than half a decade now, with the only fixed monthly costs being a few virtual private servers.

I think the main value add with the outsourced solutions is not having to deal with all the RBL issues, complaints, staying on top of the latest smtp requirements from Google etc. That does seem to be a full time job and the 75k is probably cheaper then staffing that directly. It’s not always about just the raw infrastructure costs.

I think you would be surprised at how many outsourced solutions don't deal with all the RBL issues, complaints, RFC compliance etc.

May I ask what the main purpose of the e-mailing is? Notifications for a web-app, or is it marketing outreach, some combo thereof? Thanks

sums it up 100%

This is the hazard of putting concrete numbers in a blog post like this. Among your readers will be people with experience sending this amount of traffic every second.

Then don’t read this article and write about what you think is interesting.

So it makes it interesting that you can either buy service, which seems expensive according to his experience, or spend engineering time (salary money) to build it once and pay almost nothing to keep it online. I think a lot of us would build it just for the experience, but many shops have a habit of buying things that are not their core competency.

> or spend engineering time (salary money) to build it once and pay almost nothing to keep it online

Every time an engineer says this, somewhere a puppy dies.

Especially with an email service, keeping up with the spammers, scammers, blacklists, AV false positives, delivery failures at large companies, etc. cost far more per year than building the thing.

Operations is never free.

i concur, sending 22k an hour isn't very interesting nor difficult. I think the highest i've tuned a system to was about 1m per hour sustained for days and this was back in 02-03 era. we definitely burned out a few parts and upset the hands in the datacenter a few times. would be a fun project to do with modern hardware, but i have no reason to send that quantity of email anymore. removing atime was probably the biggest single speed gain, along with understanding the MTA very well.

This is interesting to compare with this 2004 paper on managing a large email for the University of Cambridge: http://people.ds.cam.ac.uk/fanf2/hermes/doc/talks/2004-02-uk... managing 300k messages per day.

> The system’s back end Cyrus message store comprises 16 single-CPU PCs, each with 3GB RAM and 7 72GB SCSI disks providing 350GB of available storage after RAID and filesystem overheads. The front end redirector ppsw is 6 dual-CPU PCs each with 1GB RAM and 4 18GB SCSI disks providing 50GB of available storage. The webmail system is another PC with the same configuration. The terminal-based Menu System is still running on the old Hermes systems at the moment, until migration to the new system is complete. It will then move onto a PC with plenty of RAM and CPU. The backup system is attached to a disk shelf containing 16 250GB SATA disks providing 3.2TB of usable storage, and a tape robot containing a 17 cartridge magazine and two LTO drives

These days each of those would be a "small" VPS.

Edit: just found an even better example, https://github.com/Exim/exim/wiki/Q1002 "On a PII 400 with 128M of RAM running Linux 2.2.5, I have achieved 36656 messages per hour (outgoing unique messages and recipients)"

It's not really the same... You are talking about hosting big mailboxes; the article talks about sending and receiving e-mail. Sent e-mails are probably just discarded and incoming mails go into a processing pipeline, which is handled externally, which removes the need for a web interface to browse messages, search them and maintain backups.

For reference Back in 1984 Huge was a cluster of 16 or so prime 750 super mini's over two Data centres - this was Telecom Gold.

Wish I had saved some of the printouts of the billing monitor when it was running (the monthly map reduce billing system) as it listed the DASD and memory for each member of the cluster.

But we had to learn everything about email. Every damn bit.

SMTP, relays, MX-records, SPF verification, DKIM signatures, spam filtering, reverse-DNS, blacklisting, "Internet Message Format", throttling, MIME-encoding, email-antiviruses, bounces, headers, log-parsing, rate-limiting, greylisting, RFC standards (and that no one follows them)... And of course - about IP address reputation (more on that later).

Are there any open source tools to tackle some of these? Given the maturity of email marketing I would expect some pretty robust free tooling available.

I'd love to see if they did the math of the engineer time spent to learn all this vs paying another service. (Although outsourcing this doesn't necessarily mean you don't have to learn a lot of these things).

There must be an inflection point where it becomes more economical to host your own, but I doubt 22k an hour is that point? Maybe it is.

Sendgrid is pretty spendy, going by their public pricing and assuming the aveage usage is half the peak usage, the Sendgrid bill could be around ~$2500 usage.

So, instead of spending “several thousand a month at this scale” which isn’t a lot of money for a business, you wasted how many engineering hours or worse founder hours that could have utilized building features or signing-up new customers. Developer; I have to pay money, nahhhh I’ll just build it complex.

It looks like they're helpdesk software. Email would pretty integral to their business model so I don't blame them for wanting to do it themselves, although it wouldn't be my first choice unless I needed some feature missing from all of the ESPs.

Yeah:

o SMTP = postfix/exim/sendmail (hint don't use the last two)

o dkim, spf & reverse-dns are just DNS records. Spf is a list of IPs that your domain "owns", dkim is a hash that proves you own the originating server.

o Log parsing is standard log interrogation stuff. You are better off writing a custom metrics exporter that talks to the queue than parse logs routinely.

o Greylisting is a configuration option for email relays where they reject the first attempt of unknown servers attempting to deliver mail. nothing more to it. (https://www.greylisting.org/implementations/postfix.php)

o rate-limiting is again a config option: https://www.vooservers.com/technical-blog/rate-limit-outboun...

o spam filtering: https://docs.iredmail.org/enable.dnsbl.html is a good start

> We looked at paid PaaS providers like Mailgun, Sendgrid, Amazon SES... And all of them would cost several thousand a month at this scale. And since we're not a sexy funded startup from California, but a boring customer-funded business, we can't afford this bill

I'm a little curious about their reasoning for buy-vs-build. They say they "can't afford several thousand a month", but their list of clients includes Adobe, VMWare, GE and many other impressive names. I find it hard to believe that with a list of major enterprise clients, they can't afford several thousand dollars a month?

It seems more plausible that they can indeed afford it, but figured building it in-house would be cheaper and produce more profits. Developer costs in USA run about ~$5-10k/month, and that's assuming you're not trying to compete with companies like Google. Would be curious to hear about the number of dev-weeks they have spent learning all the stuff they mentioned, building it, troubleshooting any issues, and keeping the system running and up-to-date. I'm also curious about the effect building this project has had on their bus factor.

> I find it hard to believe that with a list of major enterprise clients, they can't afford several thousand dollars a month?

They can probably afford it, but if every piece of your stack needs that kind of money you run out of money fairly quickly.

> Developer costs in USA run about ~$5-10k/month

They are based in the UK.

> Would be curious to hear about the number of dev-weeks they have spent learning all the stuff they mentioned, building it, troubleshooting any issues, and keeping the system running and up-to-date.

It's just an SMTP server, not rocket science.

Hi, OP here.

Well, to be embarrassingly honest... We suck at pricing. We were offering "unlimited" plans to everyone until recently. And the "impressive names" like you mention, well, they mostly pay us around $250 a month - which used to be our "Enterprise" pricing plan with unlimited everything (users, storage, agents etc.)

So I guess the real answer is - we suck at positioning and we suck at marketing. As the result - profits were REALLY low (Lesson learned - don't compete on pricing).

P.S. Couple of years ago I met Thomas from "FE International" at some conference, really experienced guy, who told me "dude, this is crazy, dump the unlimited plan like right now" so we did. So I guess technically we can afford a PaaS now...

That's nuts. Millions and millions of dollars left on the table. You have to A/B test the hell out of pricing (and methods of delivery). I've been a long time user btw. Great product.

Whoa, small world.

1st of all - thanks for being a customer. Just so you know - we are terrified to lose you since we're not VC-backed.

2nd of all - you're totally right. Pricing has to be A/B-tested like crazy. But it's much more exciting to play with a Postfix server in your backyard, aye? Evading the businessy/salesy part for as long as you can #sarcasm

Yea this is super impressive but also a little crazy. I’ve never heard of a company (yes, even bootstrapped ones) say that SES or Mailgun were too expensive - if anything, it’s that they’re dirt cheap.

Cost aside, all that time hacking your own ESP takes away from investing in your core business.

+1 on pricing - you provide an awesome service and you should charge what your product is worth. I think it’s natural to start out cheaper to establish yourself but when your margins are so tight you need to hack your own ESP, time to examine pricing :)

I’ve tried both methods in terms of email delivery: outsourced vs self built. I think it’s great you’ve decided to own another part of your stack. The rationales you presented are the same ones many of us who have gone down this path have concluded as well. When a component of your app could potentially send unlimited emails, it’s better to be able to control this cost from the get-go.

Your self-hosted plans look very cheap to me. $1699 to $4999 including 'Unlimited lifetime technical support'.

Unless you can invest those revenues at some very high rate of return, the annual return on the money you put aside to support future support costs won't pay for much.

e.g. if you put $1699 in the bank and earn 2% interest, that $40 in annual interest might buy you an hour of someone's time to provide support to that client.

Of course, I might be wrong, but I reckon you could offer free support for the first year only, and charge ~20% per year for upgrades and support.

And my point might be totally irrelevant if all your clients go with the SaaS option.

Very good article, that proves the common belief that self hosting emails is not possible.

I do the same that the person in the article, sending 20k emails per day with the same setup and same result: it works

s/proves/disproves/ right?

Article summary: To send large amounts of emails get a VPS and use Postfix. Watch out for IP reputation.

Seriously, not to rant or anything, but isn't this "email server 101"?

but still there's probably enough companies out there paying half a remote sysadmin in fees for such a service ;).

For you it is. For me it is. But for a lot of people the first thing that would come to mind is: lets see what (insert: SAAS) can do for us. And that's a good thing, because a lot of people just shouldn't be running their own servers.

I am literally about to migrate from the mailgun API to my own email server, so this is a really helpful overview. Thanks!

> Setting it all up is a one-time expense...but this is knowledge that stays with you (and the company) forever.

That sort of depends on how well documented the entire system is, I'd think. And even then, I'm not sure this is quite as good as it sounds.

Presumably this system is going to be humming along smoothly for quite some time. But when something breaks or goes wrong, is it going to be easy to go find the problem and fix it? Probably not. Whoever built it - if they are still with the company - will likely have forgotten a lot of the little details. Even if it's documented well, that documentation needs to be reviewed, understood, and (re)learned.

So yeah, it's pretty low maintenance, sure - until something goes wrong. And it inevitably will.

I'm not saying this is the wrong approach. Thousands of dollars a month for a PaaS is nothing to shake a stick at. So this very well could be a great approach under the circumstances. But there will be hidden costs down the road. It's just a matter of when and how much trouble they cause. (Will it cause your systems to go down for days while a bug is investigated, for example).

how do you deal with spam ? do you have multiple "prepped" ip addresses that you switch if some of yours end on the spam list. at your scale it probably makes sense to take care of it, but i had problems with office365 simply sending ours to spam folder, with mxmonitor showing everything in green.

for me it was important that every email is delivered (i have 99.9% now)

I once had to send out personalised emails for a large client in Portugal. We couldn't handle sending them out one by one, but we found out that there was a 50% chance for at least two people to have the same name and surname. So we bundled together people with the same names and then managed to overcome the bottleneck.

"before that the inbound email is scanned for viruses (ClamAV)"

We moved away from ClamAV because it detected less than a tenth of virus samples we tested in practice. Simple things like EXEs, ISOs, VBAs and JS attachments.

VirusTotal uploads showed the same single digit detection rate for ClamAV.

I thought ClamAV was a better scanner, have things stagnated?

Did you use anything else than stock rules?

Nothing other than stock settings and the latest signatures, same as VirusTotal's ClamAV I think?

I use the free version of sophos for linux plugged into rspamd. Hopefully it has a better detection rate since Sophos. I did it this way since I read about the poor detection rate of ClamAV. Tbh we've blanket banned exes and their ilk so I'm not even sure if we need it anymore.

Not sure if this article is serious or a joke. 22k/h is quite small and most large enterprises handle this volume easily. Very very small scale. I personally, built a system that routinely would send between 10-20 million daily and doing about 3k/s. I know of systems that send 10x that ie. 100M/day.

> I know of systems that send 10x that ie. 100M/day

Is that a spam system?

No, e-commerce transactional and marketing.

>> Is that spam?

> No, [...] marketing.

Excuse me while I lol.

(It might be completely wanted marketing, just amused me; almost all spam is marketing for something.)

I doubt more than 1% of marketing emails are desired. They may as well be spam. Forgetting to check a box, or to uncheck a box, or accidentally checking a box, doesn't make them wanted. Not wanting them anymore but also not bothering to go disable them doesn't make them wanted. It's fair to class them with spam.

[EDIT] I mean 1% of marketing emails where there's any relationship between the sender and the recipient whatsoever, of course. All others are spam, clearly.

So still spam, but falls outside the legal classification of impermissible spam?

"tiny AWS instance running Postfix under Ubuntu. It has 2 Gigs of memory, priced around $9/month and the CPU load rarely goes beyond 15%"

The title should have been "How we send 22000 emails every hour for $9/month"

16 million emails for $9 is actually pretty good.

I'm really happy the answer ended up being "boring, old tools on a smallish (virtual) machine" since my immediate reaction to the headline was "that'd be slightly impressive I guess... with limited resources, 20 years ago".

If you are thinking about doing this, you get a lot of discouragement from the conventional wisdom.

https://www.digitalocean.com/community/tutorials/why-you-may...

It's nice to see that a small-to-medium-scale mail server is still a possibility in this time of Gmail/Hotmail bans, IP blacklists, and the like.

On that note, I wonder what happens when a scam caller dumps a SIP number and a legitimate business unwittingly picks it up when they open. Now a hundred thousand people have blocked it in the phone and it's blacklisted on several apps.

@jitbit - you might want to add “Feedback Loops” (aka FBLs) to your list of “Everything” you needed to learn.

Reply on news.ycombinator.com