Hacker News Reader

Comment by dylan604 | Hacker News Reader

They might average that, but we all know averages only work on paper. For example, AWS has tutorials that provide instructions on how to setup TLS in a LAMP stack running on Linux 2 EC2s. As part of the Let's Encrypt setup, they provide a crontab entry that runs twice a day with a copy option to paste. How many EC2s all hit the Let's Encrypt server at that exact time? Since EC2s default to UTC time, that means that servers are not offsetting those requests by timezones, so that means an even bigger spike.

View on HN

OpenBSD recently added the "~" random range separator to their crontab syntax. The manual for acme-client, the native Let's Encrypt tool, provides

  ~  *  *  *  *  acme-client example.com && rcctl reload httpd

as the example crontab entry. Though, for maximum friendliness to Let's Encrypt that should probably be something like "~ ~ * * ~", which would run the command at a random time once per week. I think you could accomplish something similar using systemd timer units and RandomizedDelaySec, assuming it permits a delay as high as 604800.

You probably should not run it only once a week. That means it takes up to a whole week to even attempt to automatically fix a problem, which is a long time. True that's going to be four attempts before, finally, the certificate actually expires (if the problem is just intermittent service availability), but it doesn't seem worth the small additional risk.

Now, sure, in principle you should have active monitoring so you'd know immediately if there's a problem status e.g. certificate with only 14 days left until expiry; revoked, expired or otherwise bad certificate presented by server; OCSP staples missing. But we know lots of people don't have monitoring, and I guarantee at least one person reading this HN thread is mistaken and isn't monitoring everything they thought they were.

Like Certbot acme-client does a local check before taking any action, so if run once a day (or indeed once an hour) it will not normally call out to the Let's Encrypt service.

Unlike Certbot acme-client doesn't do OCSP checks, so it won't even talk to the network to get an OCSP response. Even with Certbot this (OCSP) is provided through a CDN (so it's roughly similar cost to fetching a static image for a popular web site) and so your check is negligible in terms of operating costs for Let's Encrypt.

Its worth noting that running certbot / LE twice a day doesn't actually hit the LE server twice a day. It just checks the certificate dates locally and if they have been renewed in the last month it does nothing.

I guess you still have a peak time of 00:00 UTC every day though unless people are using servers set to their local time.

I believe modern Certbot also does OCSP. The intent here is, if for any reason your certificate was revoked it makes sense to try to obtain a new certificate even if it hasn't expired. Even if it can't, perhaps the operator will notice a logged reason from Certbot when they did not yet notice their certificate was revoked.

Examples of reasons your certificate might have been revoked:

* You re-used a private key from somewhere, perhaps because you don't understand what "private" means, and other copies of that key leaked

* You didn't re-use the private key but your "clever" backup strategy involves putting the private key file in a public directory named /backup/server/ on your web server and somebody found it

* You use the same certificate for dozens of domain names in your SEO business and yesterday you sold a name for $10k. Hooray. The new owner immediately revoked all certificates for that name which they're entitled to do.

* Your tooling is busted and the "random" numbers it picked aren't very random. (e.g. Debian OpenSSL bug)

* A bug at Let's Encrypt means their Ten Blessed Methods implementation was inadequate for some subset of issuances and rather than cross their fingers the team decided to revoke all certificates issued with the inadequate control.

* Let's Encrypt discovers you're actually a country sanctioned by the US State Department, perhaps in some thin disguise such as an "independent" TV station in a country that doesn't have any independent media whatsoever. It is illegal for them to provide you with services and you were supposed to already know that.

So that is a network connection, but not to the Let's Encrypt servers described in this story.

In practice OCSP is done by a big CA by periodically computing OCSP responses for every single unexpired certificate (either saying it's still valid, or not), and then providing those to a CDN and the CDN acts as "OCSP server" returning the appropriate OCSP response when asked without itself having possession of any cryptographic materials.

Certbot has a random sleep before it does anything when run non-interactively.

Reply on news.ycombinator.com