Hacker News Reader

Comment by adambenayoun | Hacker News Reader

Sendgrid customer here.

Since many services rely on SMTP providers like Sendgrid, they should have a way to notify customers when their server go down and transactional notifications may be disrupted.

I shouldn't be notified by someone who know we're using Sendgrid and happen to read HN.

View on HN

Yeah -- outage notifications to a service's users should go out, rather than just being pulled.

Also seems like best practice would be to have SPF/etc. entries in place for multiple ESPs, even if you routinely just use one, and be able to switch, for just this reason.

We do have SPF entries set up for another provider, the main problem is that some services may need to warm up an IP prior sending lot of emails through a backup provider.

What it means is that basically you should be using 2 providers at the same time and send 50%/50% on each provider in order to keep your IP warms.

While I think big services should account for such scenario, I think 95% of Sendgrid customers are using them to avoid setting up that kind of redundancy.

Main question is should I create the same redundancy for my hosting provider, DNS services, CDN etc...

Most of the really heavy CDN users I know do have multiple CDNs constantly in operation, for contract negotiation reasons if nothing else.

DNS is interesting, especially if you do anything location-based. I currently use just CloudFlare, but am not convinced they have enough internal redundancy on DNS, so investigating using Dyn, or self hosting again, or Route53, or some combination. It gets a more complex if you want to use multiple providers doing anything beyond simple DNS though (the DNS protocol itself is totally fine for this, but most of the config management is provider-specific, and using normal dns zone transfers/notifies doesn't really work in this model)

What do you mean by "to warm up an IP"?

If an IP address does not have a history of sending [large amounts of] email it is considered "cold". If that IP address then suddenly begins sending huge numbers of emails, many providers will assume the worst (that the email is being used for SPAM) and either block the IP or mark messages coming from it as SPAM. "Warming up" the IP means sending a smaller number of emails (per time period) and slowly increasing the volume so that Email providers can properly "score" the IP as legit and okay in their systems.

Update: For some reason I couldn't reply to your response kstrauser so here is my reply: That works if and when their is trust. First you have trust the buyer isn't going to abuse the system and wreck the IP [reputation] and you have to trust the buyer isn't giving you stolen info. Obviously things one can work through but again, it involves trust. Also the provider has to have warmed up IPs to give out. Which, having warmed up IPs would actually be a great valued add upsell for those that need them!

Update 2: Thank you FfejL and symfoniq for explaining it better.

That's when you physically call your sales rep and explain why your using their services and ask them to whitelist you. I say this from experience. If you called my employer (and any of our competitors, I'm certain) and said "I need to host a mail relay that's ready to go from no traffic to tens of thousands of emails per second, here's my company website, and here's our credit card", we'd be happy to make it work for you.

I'm not saying this to advertise but to offer a suggestion: call someone and ask for help. This isn't an unusual need at all and any reputable provider will be quick to help.

Short version: you don't have to "warm up an IP" if you do a little advance homework.

That's not what 'warming up an IP' means.

If Google (or any other ISP) suddenly sees an big spike in email from @example.com on an IP that @example.com hasn't used previously, Google is much more likely to mark those emails as Spam.

So senders need to 'warm up' an IP by sending a small amount of email first, usually for a few days at least.

So if you're a SendGrid customer expecting to send 2,000,000 emails today, you can't just switch to a new ESP and send those same emails. Spam rates will go through the roof.

jrs235 is referring to IP reputation as measured on the receiving end of the email. In other words, Yahoo or Hotmail might reject your emails if they see unusual activity from an IP address. Until an IP is "established" as having sent a lot of email with low bounce and spam rates, getting bulk email delivered from that IP address will be a frustrating experience.

Thanks for the clarification! I'll pass that along to our products department. :-)

Who do you work for?

Mail me at kirk@strauser.com . I'm not trying to be evasive but I like to keep work and personal separate, especially since this story has become more political than I was expecting. I just came in to read about a service outage.

They've been tweeting about it for an hour or so: https://twitter.com/SendGrid

I would expect them to send emails to customers in the event of serious issues though, expecting your customers to learn about problems through Twitter isn't that much better than learning about them from HN.

Critical notification should be sent to me and not pulled by me. Nice addition would be to have an addon to newrelic that would basically poll services/API like sendgrid and allow you to setup notifications for failure or degradation.

> I would expect them to send emails

Preferably not with their own SMTP service ;)

Aaand the PR slide for SendGrid continues. Perfectly valid point though.

This is a “PR slide” in the same sense that torching someone's store is PR.

"I shouldn't be notified by someone who know we're using Sendgrid and happen to read HN."

Was what I was replying to, not insinuating the fact that SendGrid's website going down is a "PR Slide"

how does one send outage notices if the mail is down?

Have an offsite cloud server with a mail daemon that's ready to send a few thousand messages to customers when your main server is down.

It might actually be worthwhile for a third-party status site/monitoring service to have a way to send this mail to users. It would be thousands of users per developer tool, but not millions like consumer services, so technically a much simpler service.

Letting users set "I must ack notifications of outages or keep pinging me" would make sense, too.

I'd probably consider mobile push notifications with confirmation and/or voice or SMS as well.

Basically like Nagios, but for third-party components.

Reply on news.ycombinator.com