Let’s Encrypt moving to new CA root | Hacker News Reader

Let’s Encrypt moving to new CA root

infoq.com

152 pointsalblue36 comments

Assuming you aren't able to update via a custom ROM on unsupported hardware, it should still be possible to import the new Root Cert on Android.

> System-installed certificates can be managed on the Android device in the Settings -> Security -> Certificates -> 'System'-section, whereas the user trusted certificates are manged in the 'User'-section there. When using user trusted certificates, Android will force the user of the Android device to implement additional safety measures: the use of a PIN-code, a pattern-lock or a password to unlock the device are mandatory when user-supplied certificates are used.

Source: http://wiki.cacert.org/FAQ/ImportRootCert#Android_Phones_.26...

You need root, the first command you enter is `su`, then you immediately make the /system partition writable (you can't do this without sudo). If you can't install a rom like lineageOS you probably can't do this either

There is no way to install a system root cert (user certs will not work in pretty much any app) without root, sadly

An easier way for an Android end-user or an administrator of a small number of Android devices to deal with this is to switch from the built-in browser to Firefox Mobile, because it comes with its own root store which includes the ISRG root, even on an OS that doesn't know about that.

(This suggestion is provided in the ISRG post about this change -- https://letsencrypt.org/2020/11/06/own-two-feet.html#if-you-... -- and elsewhere on the Let's Encrypt community forum. If the IdenTrust root expiration ends up leading to much consternation among users and site operators, this might be good information to spread around more widely!)

It's possible to have a locked bootloader (so no unsigned OS), but still have vulnerabilities in the vendor OS image that allow rooting (on devices that are 4 years out of date this is pretty likely).

And I believe Google Chrome is moving to its own root certificate store, kind of like Mozilla Firefox.

https://www.ghacks.net/2020/11/02/google-chrome-will-use-its...

I wonder if this will end up pushing the burden of CA root management to app developers on Android for apps other than browsers too.

This really seems like something worth decoupling from Android system updates – possibly even more so than emoji [0]?

Realistically, though, it's too late for Google to roll out something like this and we'll have to live with static copies of Mozilla's root CA set that will quickly go stale...

On the upside, it might convince apps that only talk to vendor-controlled backends (i.e. with a predictable set of CAs) to finally enable certificate pinning: If they need to think about their trusted root certs themselves, might as well only include the relevant ones.

[0] https://www.xda-developers.com/google-prepares-decouple-new-...

It would be nice if you could send two end-entity certs, with the same public key, and different intermediates, and the other side would pick which ever one it liked.

Otherwise, you have to figure out which one root CA all your clients like, because it's probably hard to figure out which clients need which certs.

You can do that I think. Your webserver sends your certificate(s) along with a chain of CA certificates up to a known root certificate. It is acceptable to send CA certificates that the browser already knows, that form signing loops, that lead nowhere, etc. It is the browser's job to figure out a signature path from a known root CA through the sent-along CA certificates to the leaf certificate you are using.

The only thing I'm not quite sure of is if sending along more than one leaf certificate will work.

However, if you want to try it out, create your two leaf certificates from the same RSA private key, and configure your webserver to send along the relevant CA certificates as well as your two leafs in a CertificateChainFile (i believe thats what the config option is called in apache). The ChainFile is just the concatenation of the text-form, PEM-encoded leaf and CA certificates you want.

iirc that's already possible, it's how letsencrypt started out without its own root widely distributed by having its intermediate certificates cross-signed with IdenTrust root.

No, your parent is describing sending two leaf certificates, which won't work.

You can send multiple intermediates, and some clients are intelligent enough to ignore the ones they don't trust. But unfortunately not all clients get this right, and the ones least likely to do so are also the older ones that likely will cease working when Identrust's DST Root CA X3 expires.

A smart modern client (e.g. most current web browsers) will just use anything else you send as a hint that can help it find a way to trust the leaf certificate. You can send irrelevant garbage and it won't count against you, but because other dumber clients exist, you are recommended not to send anything that doesn't form a logical chain of certificates back to a trusted root.

Old versions of OpenSSL for example provide a trust decision framework that assumes any chain you show it is the only valid chain, if that chain is nonsense or untrusted, it gives up and fails immediately, even if it would be obvious to a human shown the same certificates that there's a simple and correct trust path.

This is harder than it might seem because it's a graph problem and the Web PKI graph contains multiple cycles, a naive algorithm may spin uselessly forever or miss "obvious" answers.

Very interesting!

Out of curiosity, how could there be cycles in the graph?

This article has some errors which are unfortunate.

> From the start, Let's Encrypt used a top-level certificate (known as a CA Root) that was signed by ISRG Root X1 and cross-signed by IdenTrust

Their "top-level" root is in fact ISRG Root X1. This was not cross-signed by IdenTrust (that wouldn't be impossible, several popular CAs took this approach, but ISRG, the people behind Let's Encrypt, did not).

The CAs which were cross-signed were the Let's Encrypt intermediates (sometimes called subCAs), so far these have been named Let's Encrypt Authority X1, X2, X3 and X4 and now just R3 and R4 (the name has to be sent over the wire every time you connect anywhere, and just "R3" is fewer bytes than "Let's Encrypt Authority R3")

[ Why are there intermediates? Well, an issuing CA has to be connected, albeit not directly, to the public Internet. That's obviously not the most secure situation conceivable. To protect the roots they can't be kept online, they usually live in dedicated hardware modules used only for relatively rare ceremonies like creating a new intermediate. Then the intermediates are kept online. If anything goes wrong it's relatively painless to distrust a mere intermediate, whereas distrusting a root causes massive disruption ]

Let's Encrypt provide a handy diagram:

https://letsencrypt.org/certificates/

Anyway, what changes in January is not anything about the certificates you actually get from Let's Encrypt. Not one byte. What changes is essentially metadata, starting in January the Let's Encrypt backend system will by default stop giving subscribers a chain of certificates that proves this can be traced back to Identrust's DST Root CA X3, and instead give you a chain that traces back to ISRG Root X1.

Your certificate is the same either way, for most people it will say it was issued by R3 (the shorter name I mentioned above, began being used at the start of December so many Let's Encrypt users haven't got a certificate from R3 yet but will do so automatically when next renewing). What changes is the accompanying certificate for R3 itself, there are two versions of that certificate.

If you need to service clients which don't yet trust ISRG Root X1 you can (tell your software to) continue to request the other version of R3 that's signed by Identrust's "DST Root CA X1" until later in 2021 when that expires.

But there is no change to the actual end subscriber "leaf" certificate and that's a mistake in the linked article.

I thought we might see a HN link for Let's Encrypt new intermediate R3 going live in the middle of the week, but we didn't. So I guess that means it went so well that nobody even noticed.

>Anyway, what changes in January is not anything about the certificates you actually get from Let's Encrypt. Not one byte.

>[...]

>But there is no change to the actual end subscriber "leaf" certificate and that's a mistake in the linked article.

It's not really a mistake. You're correct in the sense that no change in behavior is required from the ACME protocol client, but the cert PEM it downloads at the end is different. The PEM contains the whole chain, and depending on which link the client downloads (the usual one or the alternate one in the link header) it'll get one chain or the other.

Thanks for the feedback. Will update the article accordingly.

> but the key problem is dealing with older operating systems that either don't have the correct root certificate, or don't have support for either the SHA-1 or ECDSA protocols that are used in the process of verifying a certificate.

Say what now? Pretty sure sha-1 support for certs is much older than android 7.

The article has quite a few mistakes. Nobody issues with SHA-1 any more, haven't for several years. It's almost all SHA-256 and at Let's Encrypt I believe it always has been because SHA-1 deprecation was already well underway before they span up Let's Encrypt so it would have made no sense to implement that only to immediately rip it back out.

> at Let's Encrypt I believe it always has been

True!

https://crt.sh/?id=9314793

This is the (first) helloworld.letsencrypt.org test certificate.

Now, for most (perhaps even all other?) Certificate Authorities that doesn't prove much, because after all you could manually cause issuance of anything. But the fun thing about Let's Encrypt is they've insisted almost to a fault upon only using their ACME API. After all, if you really believe in everybody else using automation it's hypocritical to have manual systems for your own stuff.

[ I say almost to a fault because there can be situations where for compliance you need to demonstrate something and it seemed (I may be wrong and Seth will correct me) as though Let's Encrypt always chose to just let nature take its course instead. Want to see an expired certificate? We'll put one of our auto-generated certificates on a server, and then just wait ninety days for it to expire. ]

So as a result helloworld.letsencrypt.org isn't just a proof that they could issue such certificates, it means logically that's what all the certificates looked like at that time.

I wasn't close enough to the process to be certain of this, but I believe you're totally correct here.

At least I know that there is no UI for manual issuance (or for manually resetting a rate limit condition), so there's no person you can call up and try to harangue/social engineer into doing these things for you.

I guess they meant SHA-256?

Yeah, my mistake - have fixed. Thanks for the feedback!

Yeah, this is 1 month old: https://letsencrypt.org/2020/11/06/own-two-feet.html

You are of course correct that the CA was announced then, but the news article was also about the fact that Let’s Encrypt had just turned five years old on the day that the article was published, and was a look back at what they have achieved over the last five years as well as looking to the future.

It was also noteworthy that they are moving to ECDSA and have reduced the length of their certificate names to save bytes (packets) in the initial TLS connection.

While true it is a good reminder that a ton of legacy devices are going to break after 2021-01-11.

It's not all-at-once but only beginning on 2021-01-11. Certs are valid for 90 days and the usual renewal period is 60 day.

So if you renew latest on 2021-01-10 your cert will be good until begin of April.

The server admin also has the choice to defer that to at least 2020-07-01, if they configure their ACME client to prefer the legacy chain.

2021 you mean?

Might not be so bad. Firefox uses its internal certificate store, and Chrome will have its own certificate store soon as well.

Does this mean we're going to see a bunch of services fail due to poor certificate pinning implementations?

This probably doesn't, but in principle other related charges could. In the past few days the new intermediate R3 went live, so if you have systems which pin Let's Encrypt Authority X3 (a bad idea but possible) those break when you next renew.

The tricky part of pinning is coming up with a good policy for what to pin. As so often the technology is easy, the policy is hard.

I've seen code that pins entire cert chains, up to and including a specific root CA. Sometimes not even the public key but memcmp with the root's entire public certificate

Not certificate pinning, but a lot of devices (particularly old Android devices) no longer work since their CA store doesn't have the Let's Encrypt Root CA that LE themselves run.

I wish there was a way to have multiple certificates (1st preference, 2nd for failover) so you can support two or more at at time - with feedback on how many using the failover. Its stressful to make a switch and then see what is broken.

You could split traffic and look for differences, if you just want to catch regressions with a new cert.

Painful

Reply on news.ycombinator.com