Letsencrypt Service Disruption | Hacker News Reader

Letsencrypt Service Disruption

letsencrypt.status.io

87 pointshw60 comments

I think Let’s Encrypt is great, but it is worth noting that other ACME providers do exist, for example, ZeroSSL. Worth noting since it seems like LE is quickly becoming a gigantic SPOF for the internet!

Recipe for disasters. Who designed this SPOF?

Nobody “designed” it and as the parent said, there are alternatives. It’s just that SSL certificates need to be generated by an authority, and LE is the authority chosen by most people that use LE’s technology.

A couple people. Working groups that decide internet standards. Industries and private companies that control how the internet works. Web browsers that determine how the world wide web works. Organizations made up of people from each of these categories. And of course, every single user, developer, vendor, etc that hungrily adopts and subsequently becomes dependent on everything they put out.

There is no God, but there are working groups of angels discussing Existence Standards.

Or a single point of entry for a powerful enough government.

Every root CA (your browser and OS have dozens built in, including those explicitly issued by governments, including ones hostile to you) is a point of entry for a government to compromise TLS.

There's no "single" one, hence all of the varied efforts (DANE, CT, et c) to reduce this attack surface.

Pinning and certificate transparency help a lot. Chrome errors out on TLS certificates that aren't in the certificate transparency log, so an attacker merely having a trusted CA is insufficient to silently MITM traffic.

Yes, it will show up in the CT logs, but so what? This assumes everyone is watching the CT logs and is able to distinguish expected from unexpected entries. And until the entry is discovered, they can still MITM traffic.

But with CT logs, why do we need CAs at all? After all, an ACME-capable CA cannot verify anything else than that I control the site's DNS record at that particular point in time (from the PoV of wherever the CA's servers are located)

But that kind of validation could just as well be performed by the CT log directly.

It seems to me, the real SPOF are DNS registrars. The authoritarian government (or some disgruntled employee of the registrar) could just force the registrar to hand over control of my DNS record to them, point it to their own server and ask LE to hand them a brand new certificate.

DNS is totally unauthenticated. It's the NSPs/ISPs you have to worry about, and they are already in bed with the covert spying apparatus of the state.

See also: QUANTUMINSERT

No, HTTPS with ACME certs protects well against that particular threat model: The CA runs the challenge from their own servers and contacts the registrar directly. That exchange completely bypasses my ISP.

A bad ISP still can intercept my DNS queries and point my browser to an MITM server - but it cannot intercept the CA's query and therefore cannot complete the challenge. So they can't obtain the certificate their MITM proxy would need.

DoH goes one step further (in theory) : When enabled, my own DNS queries are proxied to Cloudflare, so my ISP cant even intercept those anymore.

The downside of ACME is just that if someone manages to manipulate the record that is actually stored at the registrar, they get the ability to stamp themselves valid HTTPS certificates for that domain for free. Doesn't even matter which CA the site owner uses, they can alway get a Let's Encrypt cert that will be accepted by all browsers.

> The CA runs the challenge from their own servers and contacts the registrar directly.

No, they contact it via the internet, which is controlled by the national authorities in the countries between the CA and the DNS server (which you are calling the registrar, but isn't always).

It's not a direct connection; there are NSPs in between.

This is true, but the CA does not have to disclose where on the network their servers are located or which network path the validation requests will take.

Let's Encrypt sends queries from multiple servers that use different NSPs to make this kind of attack harder. (what they call "multi-perspective validation"[1])

To spoof a challange, an attacker would first have to find out where the servers are located, then compromise all relevant autonomous systems and intercept the reqests from each server.

I admit though, it's still a risk. An attacker can pick an arbitrary CA for spoofing - so even if Let's Encrypt does their homework, maybe some other free CA has a less secure ACME implementation.

[1] https://letsencrypt.org/2018/12/31/looking-forward-to-2019.h...

Wait How does that work, I often use a self signed (locally trusted) cert for intercepting traffic, and do not get any Error Messages, Does it check CT logs for only bundled certs?

Certificates issued are good for multiple weeks, and all of the automated tooling renews when there is over 7d of certificate validity left, so they'd have to have a very long outage for this to actually take properly configured sites down.

They're not in the request path, and aren't accurately considered a point of failure. This is how the PKI works, by design.

In this case it looks like their OCSP servers are down as well though. I think the validity period for OCSP responses is a bit shorter than 7 days.

Even ignoring that, 7 days is a rather short period of time for such a large portion of the web to migrate to another CA if that were to become necessary for whatever reason.

IMO having another free, independently operated ACME CA would be ideal, particularly if it could support having built-in fail-over added to ACME clients as a default setting.

> IMO having another free, independently operated ACME CA would be ideal

Did you see this comment? https://news.ycombinator.com/item?id=27885851

Obviously yes; it's in the comment chain I'm replying to.

ZeroSSL is good, but AFAIK its free tier[1] doesn't support everything Let's Encrypt does (like wildcard certs) which makes it difficult to use as a drop-in replacement for many people.

[1]: https://zerossl.com/pricing/

True, but my guess is that most certificates are not wildcards.

Their pricing page seems to be a little confusing, but [1] does state that wildcard certs issued via ACME are free of charge. The SSL.com [2] pricing model, OTOH...

> By using ZeroSSL's ACME feature, you will be able to generate an unlimited amount of 90-day SSL certificates at no charge, also supporting multi-domain certificates and wildcards. Each certificate you create will be stored in your ZeroSSL account.

[1] https://zerossl.com/documentation/acme/

[2] https://community.letsencrypt.org/t/is-ssl-coms-acme-really-...

Their OCSP servers have always been spotty. OCSP stapling helps guard against that, at least.

Let’s Encrypt ACME is a service that many depend on for their services, like Netlify, Synology, or Digital Ocean. It is true that existing certificates on existing hosts shouldn’t be impacted, but new certificates will be, and in my opinion that does actually matter. There may even be legitimate use cases for lazily grabbing certs in the request path; it’s pretty easy to do given that the Go ACME integrations basically do this plus some caching and scheduled renewal logic.

This is of course aside from the OCSP point that others are noting, which does at least make PKI downtime a little more sketchy.

> There may even be legitimate use cases for lazily grabbing certs in the request path;

Anything springs to mind?

If you're using per-user sub-domains on your service, I'd imagine you'd be using wildcard certificates.

Why use a wildcard? They're expensive and if compromised have a large blast radius.

LE limits the number of subdomains you can request on a given domain, so it is probably somewhat necessary to use wildcard certs.

How about this: a service that runs on a customer’s own domain, under a subdomain. Think blogging, GitHub issues, some kinds of SaaS products. Seems like something not covered by wild cards that would not be thwarted by LE rate limits and probably is useful in practice.

Spot on..! sure it's not great that you can't setup a new domain. But that shouldn't be in the hot path for anything.

Currently, Caddy will automatically fail over to ZeroSSL if it can't get a cert from Let's Encrypt. (You can even configure other sources in addition to, or instead of, LE and ZeroSSL. For example, you could have it fall back to a self-signed cert as a last resort.)

There's really no good reason for a site to go down due to PKI. Caddy has proven this.

Does it handle updating the domain's CAA record? Or do you have to not use CAA records when using this feature?

You can have multiple CAA records for a domain.

Given how slow DNS propagation is, rapidfire rewriting of CAA records as you failover to a different CA seems like a recipe for causing errors... which is the last thing you want as you're failing over.

I did a quick google search, and nothing indicated that Caddy will manage CAA records for you, which sounds very sensible to me.

Having multiple CAA records seems to me to kind of defeat the purpose of having a CAA record in the first place. But I guess if you limit it to two or three it'd still be fine.

The purpose of CAA records is mostly to exclude CAs you don't trust. It's less of a "I trust Let's Encrypt" and more of a "I don't trust Honest Achmed".

(Context for "Honest Achmed": https://bugzilla.mozilla.org/show_bug.cgi?id=647959)

This can be automated with DNS provider modules, if someone would like to implement it: https://github.com/caddy-dns

ZeroSSL is limited to three certs per account if you don't pay for ZeroSSL.

The pricing page is admittedly confusing, but look carefully at the bottom of the free tier... it says "∞ [infinite] 90-day ACME Certs". It appears to only be limited to 3 if you issue them through the web UI.

Based on documentation elsewhere on the site, it seems to clearly include all kinds of ACME certs, even wildcard, but I've never actually used ZeroSSL myself, as far as I can remember.

There's a particular class of bug that affects all datacenters in a globally distributed service. Almost invariably it comes down to knock-on effects due to multiple things happening "the wrong way" in tandem. Those individual failures typically happen because somebody said "well yeah we should have a better way to do this, but the other components have redundancy, so shrug". If a single domino isn't tested well and has all potential failure modes well documented and tested for, it can still take down the entire chain of dominos.

Or just a bad global configuration change. We used to take down Netflix globally with config changes until we changed the system to have config changes scoped to single regions.

And even then sometimes the failure wouldn't show up until multiple regions got the config change.

Always bet on a config change. Or DNS.

Neither of these apply to this situation though. We're working on it.

I lose many of my bets.

It sounds like it was a hardware issue [1]. First rule of electronics applies: thou shalt check voltages.

> [Update] We're conducting follow-up maintenance to address power supply issues from our earlier service disruption. All production API services may be down for up to 30 minutes.

[1] https://letsencrypt.status.io/pages/maintenance/55957a99e800...

It sounds like you're quoting from the classic "How complex systems fail":

https://how.complexsystems.fail/

In theory this should have no effect, right? Since the certificates should be renewing a few days before they expire, in theory it could be down for a couple of days.

Good way to make sure all the software is resilient to renewal failures though!

This affects us! But only because we do certs-as-a-service. Existing certs won't break but it has wrecked the UX for a lot of customers.

Well this saves me a half hour of troubleshooting!

Oh good. Something was going wrong issuing new customer certs for Darklang. Glad it's not my bug!

I’m not sure if this is affecting websites already having a certificate provided by Let’s Encrypt. I believe not.

Just the issuing of new certificates

The OCSP responder being down could affect websites. https://en.wikipedia.org/wiki/Online_Certificate_Status_Prot...

FWIW people can mitigate that issue by using OCSP stapling [1] assuming their load balancer / web server supports caching the response.

[1] - https://en.wikipedia.org/wiki/OCSP_stapling

Traditionally OCSP stapling implementations in the major webservers (nginx+apache) have been pretty bad and are not capable of mitigating OCSP server downtimes.

Apache has a new stapling implementation since a few versions that can be enabled with "MDStapling On", I don't think it gets enabled by default yet.

Caddy has had a robust stapling implementation for years.

Does the caching reliably work in common webservers by now? I remember having read a couple years ago that Apache would simply clear its cache if the connection to the ocsp provider breaks (or did something similar unhelpful, resulting in an error to the enduser).

Very likely you read my blogpost. See the update at the top: https://blog.hboeck.de/archives/886-The-Problem-with-OCSP-St...

Caddy does OCSP stapling + caching reliably (and has for years).

It will even auto-renew your certificate if it gets an OCSP response of "Revoked".

It's hit-and-miss per LB/server and I have not seen this become a priority since it's not a super popular feature. Here [1] [2] are a couple articles on the topic. My experience has been with HAproxy and F5 load balancers. HAProxy uses an out-of-band process to lay down a .ocsp file and load it via the API. This in effect acts like a cache assuming the script creating the .ocsp file has error handing to avoid clobbering the file if the upstream OCSP endpoint can not be reached.

F5 load balancers will cache the response in memory. I have not tested Apache with OCSP stapling/caching recently so I can only assume based on feedback from others here that they have not improved it. I would expect nginx to improve now that they are owned by F5, maybe, eventually.

I am a fan of OCSP stapling/caching for the privacy aspect. No need for browsers to leak to the OCSP end-point what domain you are visiting. There are enough nosy people sniffing our traffic already.

[1] - https://www.keycdn.com/support/ocsp-stapling

[2] - https://blog.cloudflare.com/high-reliability-ocsp-stapling/

I imagine that, for still-valid certificates, it will almost always fail open.

We have been serving OCSP responses out of another datacenter since the initial status.io went up.

This really shows why we need more than one competitor in this space.

https://zerossl.com/

https://www.xf.is/2020/06/30/list-of-free-acme-ssl-providers...

Heh. We shutdown a couple of legacy mystery servers today for a second scream test. I hope this wasn't us!

(and no, we are not affiliated with Lets Encrypt. It's just a funny coincidence. but who knows what those mystery boxes all in all do?)

Reply on news.ycombinator.com