Hacker News Reader

Comment by swombat | Hacker News Reader

I'm not convinced about the need for SendGrid. Learning to set up an email server isn't that hard in this day and age, and we seem to get very decent deliverability with our own SMTP server - and don't pay a penny for any emails sent.

I'm planning to build an email subscription feature for swombat.com soon, and was looking at options like SendGrid, but I just can't justify the cost. When you send people a sign-up confirmation email, they expect it right now. If it's not there in a couple of minutes, in my experience, even the least tech-savvy have learned that they need to check their spam folder.

View on HN

I was talking to the guys from mailgun.net recently, which provides a similar service to SendGrid, but also makes it easy to receive e-mail via webhooks. They told me a ton of interesting things about how this stuff actually works.

Getting high deliverability is actually incredibly complicated. You need a dedicated IP address to send mail from, which you need to be whitelisted by the big ESPs (E-mail Service Providers). To get it whitelisted, you need to "warm it up" before you start using it properly. You need to be very careful about setting the right headers. You need reverse DNS configured correctly, you need SPF and DomainKeys set up, you need to make sure the IP/domain you are sending from hasn't previously been used for spam... and even after all that, you probably need to set up a relationship with AOL/Gmail/Yahoo etc to make sure your stuff gets through. You also need to watch for their bounce / rate limit notifications and dial back your sending rates to match.

There's a whole lot more to it than just setting up an SMTP server.

All of this apart from the relationships with ESPs is relatively easy to do if you're a competent sysadmin.

As far as the ESPs are concerned, all of them appear to accept our emails just fine, except for Hotmail. We've largely given up on Hotmail, but then everyone who uses hotmail seems to be aware that it marks everything as spam and so they check their spam box as often as their inbox, from the looks of it.

It's a nice convincing argument. It just hasn't borne out in practice for us. Email is critical to us, and it gets delivered just fine the vast majority of the time. We've had about 2 cases in 3 years where we needed to intervene manually to talk to some kind of ESP to get us unblocked, because they'd randomly decided to blacklist our IP. That's not worth $80/m.

The trouble with hotmail is they sometimes just blackhole what they classify to be spam, instead of filtering it into the Junk folder. It doesn't even bounce, it just disappears.

Gmail does this as well. Our outbound mail to Gmail accounts was just disappearing until we fixed a reverse DNS record.

Really? I've come across loads of people who have had the disappearing email problem with hotmail (including myself on a couple of systems), but this is the first time I've heared of anyone having it with Gmail.

I also think it's a good idea to get two separate IPs. One for signup/notification emails, and a separate one for marketting. People will report marketting emails as spam, even if they've signed up for it.

Getting high deliverability is actually incredibly complicated.

Urban legend.

You need a dedicated IP address to send mail from, which you need to be whitelisted by the big ESPs

That is nonsense. Nobody reaches out to the ESPs to have their IPs "whitelisted" in advance. You reach out when, if and after you've been blacklisted, in order to get unlisted.

To get it whitelisted, you need to "warm it up" before you start using it properly.

Nonsense.

You need to be very careful about setting the right headers. You need reverse DNS configured correctly, you need SPF and DomainKeys set up

Yes, there's a few things to watch out for, but it's all pretty well documented nowadays and you only have to do it once. SPF is pretty easy with http://old.openspf.org/wizard.html

you need to make sure the IP/domain you are sending from hasn't previously been used for spam

A quick check at a few of the largest blacklists (spamhaus et al) takes all of 5 minutes. Another 5 minutes that you have to spend exactly once.

[..further marketing pitch for MTA SaaS snipped..]

Well, to sum this up.

It seems many people have a strong misconception of what SendGrid, MailGun and friends actually do. You don't magically get higher deliverability only because your mail is passing through them. They don't have mythical "whitelist" slots at the ESPs. They're not even on the radar of the ESPs.

What you get from SendGrid et al is a (hopefully) properly configured MTA and DNS, to push your mail through. From there your outbound mail is suspect to all the usual blacklisting and scrutiny at the receiving endpoints. And that's all - no black magic at work here.

>>> "You don't magically get higher deliverability only because your mail is passing through them. They don't have mythical "whitelist" slots at the ESPs. They're not even on the radar of the ESPs."

That's absolutely right, you don't. But email deliverability turned from a massive mystery and headache for us, into a sure-fire deal, the day after we implemented SendGrid. We develop translation/localization related apps, not email apps. So for us that's cool :)

If they're a good deal for you then there's no arguing with that.

I didn't mean to say they have no value proposition at all; the API and stats they provide can sure be useful for some.

All I was saying is that their raw deliverability is no different to running your own server. And your own server doesn't charge a few pennies per mail...

Sure, I think it's a valid point. No beef there.

> That is nonsense. Nobody reaches out to the ESPs to have their IPs "whitelisted" in advance. You reach out when, if and after you've been blacklisted, in order to get unlisted.

That's nonsense, lots of people do that. Only you don't usually go direct to the service provider, see following links:

http://www.returnpath.net/commercialsender/certification/

http://www.spamhauswhitelist.com/en/

http://www.certified-senders.eu/csa_html/en/266.htm

http://dnswl.org/

That's nonsense, lots of people do that.

Lots of people also believe in astrology. Anyone is free to do what they deem good for their karma, you just shouldn't expect a measurable impact towards your deliverability. The big ESPs don't look at these lists.

> The big ESPs don't look at these lists.

Some of them do.

AOL uses GoodMailSystems.

http://postmaster.aol.com/Postmaster.Whitelist.php

http://www.goodmailsystems.com/

Hotmail uses ReturnPath.

http://mail.live.com/mail/services.aspx#Safelist

Yahoo seems to run their own. I remember them using one of the big ones but I can't find any mention in their help pages now.

http://help.yahoo.com/l/us/yahoo/mail/postmaster/bulkv2.html

Google only asks for best practices, but it's possibly they look at some of the whitelists for more data.

https://mail.google.com/support/bin/answer.py?answer=81126

Oh my, do I have to put it even more bluntly to get the point across: These commercial whitelists are effectively extortion scams and affiliate rings.

Their sole purpose is to extract money from you. They're businesses, that is their business model.

The value they provide in return is rather minimal - unless getting e-mail out is at the very core of your business and you need to pull every last string to absolutely maximize it.

Otherwise there are cheaper ways to get reasonable deliverability even at AOL and Hotmail. The default deliverability is already quite good, which is what services like SendGrid largely bargain on. I firmly doubt SendGrid signs each of their customers up for the aforementioned whitelists. They probably offer "assistance" with the process, which probably amounts to sending you a link to a nice HowTo document, and which probably yields them a small affiliate kickback when you actually follow through.

> Oh my, do I have to put it even more bluntly to get the point across: These commercial whitelists are effectively extortion scams and affiliate rings.

You made a few points.

First is that nobody reaches out to get themselves whitelisted in advance. Lots of people do that and some ESPs even offer their own whitelist request forms.

Second is that none of the big ESPs look at third-party whitelists. AOL and Hotmail say they do.

Third is that even if the above is true, the impact on deliverability is minimal. I would guess you are right about that but I'd love to look at some data.

Intermediaries like SendGrid would be well within reason to be paying for such services. I don't think they have to signup their individual customers on the whitelists. They just say they are intermediaries and promise to vet their customers.

I think anyone sending low volumes of email is much better off using an intermediary. They get to use well-configured mail servers, IPs with good reputation, etc. Doing it yourself, IMHO, is not worth all the trouble compared to paying $20/month.

Not sure how many emails you send a day, but I signed up for sendgrid almost immediately after I learned it existed.

ISPs have lots of arbitrary policies. For periods of time my emails have been blocked from AOL and AT&T accounts, which make for a lot of headaches and support emails to deal with. (I'm just sending out forum registration, activation and thread subscription emails. no mass mailings.) For a while, we just told people to use non-AOL addresses.

Also, I haven't noticed a delay in email deliverability. It seems to deliver quickly.

Edit: If it matters, for reference, my forum sends out 6000-7000 emails a day.

I use SendGrid, too. It was a no-brainer: $80 a month is cheap, I am expensive. I was quite expensive indeed dealing with persistent minor clusterflops like, e.g., automated forum spambots for the forum I host for college debaters (an old hobby) caused the BCC mailserver they were freeloading off of to have enough of a spike that a large multinational advertising company decided BCC "Thanks for paying me $30 Cindy, here's the software you just bought" emails needed to be consigned to the seventh circle of hell sandwiched between Viagra spam and their own customer support requests.

For us it's just another thing that's not our core competency which we're happy to have outsourced so that other people worry about security/maintenance etc... Similar rationale for using Spreedly [1] for subscription billing.

In the future it may be the case that we move that in house, but right now it's working out really well.

We're using PostMark [2] which amounts to $1.50 per 1000 emails, which is cheap enough for our uses. It's one less thing to worry about so we can focus on more important tasks such as finishing the v2 we're about to launch!

  [1] - http://spreedly.com/
  [2] - http://postmarkapp.com

you get decent deliverability now - what are you going to do once that stops happening?

one of the points of SendGrid is that you pay them to worry about that, so you can concentrate on your core competency.

one of the points of SendGrid is that you pay them to worry about that

Where are you getting that from?

I see nothing on the SendGrid page to suggest they track your blacklist status or feel obliged to help with unlisting when it happens.

We use Postmark, (we've sent a few million emails through them) and they indeed do this for us.

http://sendgrid.com/deliverability.html

I don't think you're wrong — I just think that for $80, we get a lot of peace of mind once we were sending thousands of mails per day. Particularly because we're sending job notifications to translators too — so if they don't see them, they might not think to check spam.

I dropped the idea of using Postmark after I read this blog-post: http://blog.postmarkapp.com/post/1431120311/oddities-in-gmai...

In there they admit that mails sent through Postmark still end up in Spam-folders or black holes. That was basically the only added value I'd get from using them.

This isn't a technical question, it's an economics question. If you would rather put in effort than dollars, then do it yourself, and it will work great for a while. But if your business takes off, you will accumulate email-related problems like bacteria, and suddenly that $20/month starts to sound reeal worth it.

My company uses it because we run our web front ends on various cloud providers, most of whose IP space is blacklisted. I don't even bother to verify that they are anymore and just assume that they are. Sendgrid gets rid of this entirely, which is a pretty big win.

Reply on news.ycombinator.com