Let's Encrypt certificate issuance was down | Hacker News Reader

Let's Encrypt certificate issuance was down

letsencrypt.status.io

141 pointsromseb51 comments

I'm going out on a limb and saying that "emergency response" is a thing of beauty. timestamped. To the point. Examplar of how it should be done.

These look like "canned" responses - one of the terror inducing parts of managing an emergency response is, in the middle of everything, choosing the right set of words to use when telling loads of really worried people - and it is dry easy to put it off and that lack of communication makes things bad

this smells like a tickbox on an incident response webform - the responder is just clicking "we have identified", "production database" (or network issue or ...) and the response handler does the rest

would love to hear more

One of the features of status.io is "Create reusable templates to streamline your incident and maintenance updates": https://kb.status.io/planned-maintenance/status-update-templ...

Yeah, we have our 'Shit's on fire' one.... generates a message like 'We have some unscheduled maintenance of databases to improve your customer experience'.

That's borderline insulting. When a service is down, it does not "improve customer experience".

Maybe it was intermittent or very slow before taking it down to fix the issue.

Bringing the database back up with a fix does improve your customer experience, however!

It's triggered during times when database response time increases beyond 1 second per query.

You can read more here:

https://community.letsencrypt.org/t/2019-11-17-autoincrement...

Once a post-mortem is done there will be more information.

Status.io is where we post status information, not long-form explanations. The responses aren't canned the way I think you mean it, they're just meant to tell you status information quickly and clearly - is it up, is it down, what services are affected, how long until they're back up...

Once we have a fairly complete picture of what happened to communicate, we communicate that on our community site.

I can assure you this was not a tick box operation.

Is there a detour here or is using letsencrypt reliant on these apis?

https://medium.com/enigma-shards/lets-encrypt-uptime-and-ope...

You need the API up; that's how it works.

However, should you fail to renew a cert, your script should just keep retrying until the API is back and you should allow enough time for it to fail. I believe most clients renew certs ~10 days in advance.

Obviously if you're relying on getting a new cert to bring up something like a preview environment or to hand out your own subdomains, this will result in a downtime/delay in provisioning, but most people would be fine with a single wildcard and never really experience a problem, as long as their script runs and they keep it reasonably up-to-date.

> I believe most clients renew certs ~10 days in advance.

The recommendation and the time certbot uses is 30 days.

I've heard 10 days before, possibly older documentation?

Based on the reminder emails they've sent me, they recommend renewing when one-third of the current cert's validity is remaining. That is, 1 month for their 3-month certs.

What is the problem with attempting the renewal every day?

It will predictibly fail until D-30,and then there will be 30 attempts to renew (in case something went wrong firth the 1st,2nd etc attempt)

Some of these can (and should) be written ahead of time, at least as templates.

Not only does doing so save someone from having to write them on the fly when they're worrying about other things, they can encourage you to communicate specific factors ("OK, we think we've found the problem, let's publish the message to that effect now"), remind you to think of certain things ("we think we'll be up within the hour" "we don't know yet how long it will take to fix" etc) and for that matter, if you have further failures, help someone look back to see how the process went.

It was down for 5 hours since when they posted the first update. And now it's fixed.

> November 17, 2019 19:33 UTC > [Resolved] We have resolved the issue that was affecting production certificate issuance. Service is restored.

Isn't it WAI that the Service Disruption API is unavailable?

It's quite confusing, but I don't think it's a "Service Disruption API", but instead a Service Disruption in the API.

A small outage should not matter at all in case of renewal, because clients are configured to renew days or weeks before cert expiration.

Yeah... my own setup for a hobby project of mine renew 30 days in advance but I don't think the way it is set up will try again tomorrow if it fails today.

I've got a systemd timer that tries to renew it once per day. It just aborts early if the existing cert has more than 45 days left in it.

Isn't that placing a lot of load on a free service?

With certbot, the check to see if certificates are close to expiry is done offline. So the service is only hit if a renewal is needed. The developers of certbot actually recommend that you schedule the cron twice a day.

EDIT: Actually I'm wrong. It also checks if the certificate was revoked via OCSP. However I can't imagine that it consumes much resources.

They're also different resources, and cacheable ones.

That free service has rate limits, which are presumably set to a sustainable level for them.

"Aborts early", in this case meaning before any network calls are made.

i also have a daily cron. One of the renewals somehow repeatedly reissued the cert for 5 consecutive days without me noticing. For some reason the certificate was also nowhere to be found, and letsencrypt woulnd't let me manually renew the certificate because it had exceeded the limit of 5 issuances/month (or something of that sort). Thankfully the limitation is only for the exact same cert, so you can issue a new one for a slightly different set of domains without waiting for the period to pass.

The default configuration for certbot in most distros is to attempt renewal twice a day

dehydrated is easy to set up this way, and has barely any dependencies.

Somewhere out there is a person writing am angry blog post about how unreliable letsencrypt is because they set the renewal date to last minute

Good chance one of the classic CAs will use this outage against Let's Encrypt though. Some classic CAs have been spreading a lot of FUD about Let's Encrypt.

Probably. What matters is the community response to the outage though. From what I saw there's been a ton of love for the team working through an entire Sunday.

And this is the reason why Debian (and its derivs) ship a cron and systemd timer for you with certbot. You shouldn't have to think about or worry about this stuff. Certbot handles it for you.

For some reason my cron task always fails to renew, and I have to manually renew at the last minute, when I notice the certificate has expired. This is for a personal site, so it doesn't matter.

I tried many things, and can verify the cron task renews when I test it, but months later it always fails.

Is your cron, by any chance, configured to run on every first day of month at midnight?

Is this on Debian? There should be a log file created in /var/log/letsencrypt on each attempt; if you could send me one at my username @debian.org, or ping me on Freenode IRC, I may be able to help you debug. I'm guessing it's because your renew.conf's trying to use the "simple" authentication method which won't work if you've got a webserver running.

Not a solution to your problem, but if you supply an email when creating a certificate, you will receive a notification plenty in advance (three or four weeks, I think)

You're welcome to join us on https://community.letsencrypt.org/ and we can help you debug this.

I receive a mail when certs are soon expired.

I also set up traefik with auto renewal so it's a setup and forget thing now.

Try CapRover?

It does your auto renews in the background.

Hey, it might be a bug that a "real" user could encounter, if you need any kind of motivation to actually look into it!

True, but still quite impactful for initial issuance being provided live to end users of things like Squarespace, Zendesk, Pardot, etc.

The Caddy server can be configured to request a certificate while the user makes a request. That means things hosted in that manner might have downtime from this.

I stopped using Let's Encrypt when they broke Apache servers not using virtual hosting.

Good for you? I guess? Not sure what to do with that comment of yours. Is it an attempt at venting or constructing a situation where Let's Encrypt is responsible for your server?

1. The certbot client is not a product of the Let's Encrypt CA and is not mandatory. You can use another client.

2. With the certbot client, instead of using a deep integration with your http server, you can use `certbot certonly --webroot --webroot-path /var/www/html/.well-known/ -d example.com` (or similar) and provided you told your web server to serve file from the /var/www/html/.well-known/ filesystem directory under the /.well-known/ url path prefix on your domain, cert issuance and renewal will work. This should be easy to do with any web server. certbot will not know which web server you happen to use and will not care. You can replace the web server and renewal will keep working.

Just don't use auto stuff. Be responsible for your configuration. certbot certonly --webroot works fine.

Certbot is from EFF not let's encrypt - you can actually write your own certbot if it doesn't do what you want - nothing to do with lets encrypts.

Lots of instructions for many common situations here: https://certbot.eff.org/all-instructions

Did you check that you were running it with the right command flags?

If certbot has a bug that breaks things in a situation it's intended for do file a bug or even better a patch to fix the bug.

You can use dehydrated [1] which is a single shell script that you can trigger via cron to do the whole let's encrypt stuff for you. No dependancies other than curl.

[1] https://github.com/lukas2511/dehydrated

Reply on news.ycombinator.com