Mailgun Down – SSL cert is expired | Hacker News Reader

Mailgun Down – SSL cert is expired

status.mailgun.com

65 pointstarr1153 comments

Ugh. Should be fixed now. Sorry to anyone that experienced disruptions due to this stupid error.

We've had issues previously with Rackspace and SSL certs expiring (webmail api certs, not mailgun previously). My app has been "broken" twice in the past due to expired certs, to the point that we now monitor all SSL certs used on third party api's that we rely on. (As well as monitoring our own certs of course).

We've got our eye on Rackspace's webmail reseller api's SSL cert, which now has less than 14 days till it expires: https://admin.webmail.us/excedentsoap/excedentsoap.wsdl

Pretty well unacceptable for a major vendor of SSL certs. Wouldn't you hope that Rackspace could help keep you out of this trouble if you bought a cert from them? I wouldn't expect it, since they can't keep themselves out of this sort of trouble.

To clarify. We don't buy SSL certs from Rackspace. I was referring to Rackspace allowing their OWN certs to expire, breaking api calls made to those endpoints. The URL I linked to was a webmail.us URL which is a Rackspaced-owned domain.

Edit: fixed grammar.

That's horrifying!

This is a really important operational point that deserves highlighting -- make SSL checks part of your monitoring!!!

This is probably a good time to piggyback an Ask HN: What are the best SSL providers these days with regards to price/hassle vs device/browser coverage?

I'm ashamed to say my company is still on VeriSign (ie. Network Solutions) simply out of inertia because by the time we realize the cert is near expiring there's no time to evaluate and switch providers.

They're all pretty much the same. Browser support for the big issuers is pretty near universal, even on mobile. Choose the cheapest reseller you like. No matter who you buy through, you end up in the same request/validation process on the actual issuer's site after purchasing.

I usually buy from Namecheap just because they already have my cards on file.

https://www.namecheap.com/security/ssl-certificates/comodo.a...

Same prices at SSLs.com (used to be cheapssls.com).

http://www.ssls.com/

An SSL cert costs between $5 and $10 per year depending if you're willing to prepay multiple years. Wildcard certs are around $100/year.

For noncommercial websites, https://www.startssl.com/ offers free certificates.

For commercial websites, the cheapest certificates I've seen are PositiveSSL certificates resold by gogetssl [1]. It's $4.55 for one year or $17.25 for 5 years.

Disclaimer: I have no affiliation with either site. I've never used gogetssl, but I will probably give them a shot the next time I need a certificate.

[1] https://www.gogetssl.com/domain-validation/comodo-positive-s...

StartSSL has a good deal for commercial websites too - for $120/year you get unlimited certs, including wildcard.

Only downside is they do proper identity checking which is a bit painful to go through (they will need to see official company documents, verify ID by phone etc) - but worth it in the long run IMO.

Yeah and sometimes they just drop the ball on the ID check. I didn't have a bill ready that has my physical address on it (none of my bills print this info) so they said they'd send me a piece of mail with a code in it. Never showed up, all attempts to contact them were ignored. Ended up just getting an $8 cert somewhere.

I will say that once you do get into StartSSL (which I have through a previous company), it's nice to be able to create/sign unlimited certs.

They also are located in Israel which can add a few days to verification if you can't do it online or via phone.

At work we use digicert -- good site with support for corp stuff like separate accounts for accounting/ops/etc, expiry notification, etc.

For personal stuff I use alphassl.

Another vote for Digicert for corp certificates - their support team is top-notch and if you opt to chain to Entrust you get browser support back through 1999, if you need to support older devices.

I'm surprised there aren't companies that look at the expiration dates of SSL certificates and try and get them to use their service to get a new one, like domain people do. Seems like it could also be a feature of New Relic / Pingdom / etc.

There are - I get spam from more than one of them. However, at least the ones I get are not from "brand names", but rather no-name resellers.

Many SSL providers have sales staff that try to contact their own customers prior to expiration.

We have this at our website monitoring and recovery service https://t1mr.com and it's free.

This looks quite handy! I've got an SSL certificate installed from StartSSL and I'm getting this error though:

    [Errno 1] _ssl.c:504: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

Hmm, seems I missed a step installing this certificate. No worries, I'm on it :)

Would you consider adding a check for particular string on page? That would be handy.

Also, if you check SSL cert validity, you should totally advertise that! I remember I've been paying for one such service where they would monitor SSL url, but would not alert you about SSL cert expiration. I found out the hard way.

Sure, checking for a string on the web page is also in the basic functionality. We also ping your site every minute, so you'll know about problems immediately.

The advanced (paid) functionality is a restart of your vps or other automatic ssh action trying to recover it when it's down.

Oh there are. Most seem to just be trolls, though.

My nagios has a check specific for expiration of certificates. First I get a warning 14 days before and an error 3 days before.

I've set up nagios alerts around SSL certs. It's free and easy.

I'm actually in the process of writing an app to do this. If you'd be interested in seeing the MVP when it is ready, feel signup to our mailing list: http://launch.renewalmonitor.com/

Had an expired SSL cert bite me in the ass yesterday actually. They auto renew on the billing side, but if you have a year SSL cert, you have to generate a new one each time. So I saw that the billing renewed, didn't think anything of it, then was reminded of the manual generation step by a bit fat warning on my app. Yay.

Oddly, their uptime charts still show 100%

well, technically, their system was up, the certificates were just expired.

Anything that makes a system unusable is downtime...

"Technically our site was up, just the database was down..."

Right, but if their uptime monitor is internal, and they are resolving from an internal address, nothing is wrong.

There is the problem.

Yeah, but if your status page is made to be seen by external people, the right thing to do would be to monitor it just like if you were using it externally. There are whole slew of problems that wouldn't be exposed correctly if your monitoring point is internal -- like this exact problem, DNS, etc.

RIght, but since their customers are not internal, having a monitor that is not outside of their network would make for a pretty poor uptime monitor.

The system would be useable. You just wouldn't be able to validate the certificate. That said, it's a fine line and I would consider the system down in this case anyway.

It's time for this old joke again:

A software engineer is firing a rifle on the shooting range, but not hitting any targets. Puzzled, he puts his finger into the barrel, and pulls the trigger. Seeing as his finger is blow off, he cheerfully reports to the instructor:

- On my end the bullets are flying out, so the problem is somewhere on your end, with the target!

Same story when they had a maintenance window, the status shown 100% uptime during that period.

I don't trust their status page.

They use statuspage.io, a great service, but most people use to manually set the times that the service was out - but I believe they have an API for that as well.

So I think MailGun just didn't indicate the service was unavailable when they put the outage details in.

If I had to guess, their StatusPage is only reporting on port 25, which wasn't impacted by this.

Disclaimer: I work at Rackspace and sit across the office from most of the MailGun team.

So, more or less bad than when Azure let their SSL cert expire last year?

http://www.zdnet.com/windows-azure-storage-issue-expired-htt...

My favorite is when Passport.com expired, taking down Hotmail, and a good samaritan renewed it for Microsoft.

http://news.cnet.com/2100-1023-234907.html

I hadn't heard of that episode. That's hilarious - particularly that it was a Linux consultant.

Expiring certs and domain names kinda haunt me. It's such a silly task and for most of us we only have to deal with every year or two. Yet if you forget, your site goes completely offline! All the work spent scaling and automating the site and one stupid renewal can undo it all.

I have all manner of alerts for these things, but still I worry about it and check from time to time to make sure nothing is expiring soon.

I use http://sslcertcheck.com to monitor these situations for me. You can do so for free, real simple service. Just gets tiring to deal it's these sorts of issues, and great to get a bigger warning before the service goes down as a result. Full disclosure: I'm friends with the owner.

Heh. I wrote 3 calendar reminders 60, 30 and 15 days out for our engineering team on Friday reminding us to do this when the time comes. Now, I will reserve the right to feel smug.

Yeah, I mean, if you're responsible for an SSL cert, why isn't this common practice. I'm going to join you in smugness, because I did this as well on a web app I was partially responsible for. It's super easy, and if your system is at all critical (mine wasn't even, really - it was an internal-only wiki with a self-signed cert from a different internal team), then there's no excuse not to.

There was a bit of jest in the smugness, but in my case it was because I inherited the SSL responsibilities from an outgoing engineer and randomly thought to myself "I wonder when these servers certs expire?". It's the type of thing that multiple people need to keep an eye out for, not just one person in the event of inevitable turnover. For something that can entirely take down your site/service for an indeterminate period of time that is entirely out of your control, it's a necessary evil.

protip: nagios.

Seriously - this is on my standard checklist for installing new certs.

More specifically, you can use the nagios check_ssl_cert plugin [1], which checks if the server is running and delivers a valid certificate. We use this plugin extensively and it takes the surprise out of an expired ssl cert. You will get multiple heads up, say 90, 60, and 30 days out (it's configurable), that your cert is going to expire. Seems like a no brainier.

[1] http://exchange.nagios.org/directory/Plugins/Network-Protoco...

This feature is built into the standard check_http with the right flags.

Specifically: `/usr/lib/nagios/plugins/check_http -H <ip/hostname> -C 60`

There is a free service called http://certalert.me that can warn you about expiring as certificates. Check it out!

I wonder what is the economic impact of this error.

Reply on news.ycombinator.com