Let's Encrypt DST Root CA X3 has expired | Hacker News Reader

Let's Encrypt DST Root CA X3 has expired

twitter.com

74 pointssynack25 comments

Looking at the number of threads split out from https://community.letsencrypt.org/t/help-thread-for-dst-root... I think this has been bumpier than they'd hoped.

In particular these two problems don't seem to have been spotted ahead of time:

Android Dns-over-TLS trouble https://community.letsencrypt.org/t/android-devices-with-dot...

Trouble with Electron applications https://community.letsencrypt.org/t/issues-with-electron-and...

Hopefully this will be helpful to some.

The primary problem in most cases is the full chain cert not being updated or used in your configuration.

1. Your Let’s Encrypt script might be outdated and isn’t pulling the down the new chain file. This could be your first problem.

2. Your configuration for Apache or whatever app your using is referencing a chain file that isn’t the current one. This could be your second problem. It’s possible you copied the chain file somewhere a long time ago and have been referencing that one instead of the new one that gets pulled down.

The problem will be confusing because your cert will be current, but it still won’t be trusted.

Something like Dovecot can actually use the full chain cert file as the cert file. This will solve that problem instead of only referencing the .crt file, which won’t help, because it won’t be trusted. The full chain needs to be used.

This was not a fun morning for me. All of my sites which are running on Debian 9 suddenly weren't able to contact our authentication and other internal systems. I had to disable SSL verification until ops can move them to newer servers.

Debian 9 can't be updated to use the new root CA?

Yes, Debian 9 security updates contains the new ca-certificates package.

Right, thanks!

Indeed, unfortunately I already had updated all of my sites before I learned this.

This caused a major problem for me this morning on a Ubuntu 16.04 server. The fix involved removing the DST_Root_CA_X3 certificate from the server and then running "update-ca-certificates".

This is hurting me pretty bad right now, renewing my LE cert is not fixing it via Terraform.

As far as I understand, renewing with the certbot flag --preferred-chain "ISRG Root X1" [^1] will drop the hack for Android < v7 support and allow older OpenSSL versions to work again.

[^1] https://community.letsencrypt.org/t/openssl-client-compatibi...

google cloud monitoring's uptime checks started failing this morning too

https://status.cloud.google.com/ >>>Global: We have identified an issue affecting "Uptime Checks" within Google Cloud Monitoring, impacting customers using "Let's Encrypt" 3rd party certificates.

it generated a scary amount of alerts, almost finished fueling the single-engine before I realized what the problem was.

Oof, felt this pain. We are having to blacklist the expired SSL cert because our openssl version prefer it over the valid cert. Time to redeploy all things.

This also bit me. I thought I was in the clear not using anything with outdated CA keystores. Turns out that some TLS implementations don't trust the connection if the server provides an expired CA in the certificate chain.

This includes the Nextcloud client for Windows and the DNS over TLS implementation in Android 11.

Adding the argument --preferred-chain "ISRG Root X1" to certbot fixes this by not chaining the expired CA X3...

I'm fairly sure this is the issue I see with my TinyTinyRSS instance I'm running via Docker on my Synology (`60 SSL certificate problem: certificate has expired` error on a bunch of feeds using Let's Encrypt certs today). I haven't updated the image in many a moon (since it worked just fine), but now I might have to bite the bullet.

We had hundreds of old lambda functions that have been running for years and so we haven't thought about them for a long time.

Since they were on node v10 they stopped being able to talk to letsencrypt SSL sites today. Since AWS has stopped supporting v10 we couldn't upgrade them to the minor version of v10 that supports the CA

We got hit with gRPC considering the LE certs as expired: https://github.com/grpc/grpc/issues/27532

Luckily this was quick to fix by just renewing the server cert without the DST root.

Twitter is abuzz and not in a good way: https://twitter.com/search?q=letsencrypt&src=typed_query&f=l...

Slack and Shopify seem to be affected.

Slack had an unrelated DNS issue today

Today has been hectic. Anybody have tips on how to resolve the issue for apps on Heroku? I don't have access to remove the DST_Root_CA_X3 cert since my certs are managed by Heroku ACM.

From my understanding, the only way to remedy is to move away from Lets Encrypt...

I ended up moving away from Lets Encrypt to Comodo. As an API provider, I need to support older OpenSSL versions, and LE and Heroku offer no way for me to do that.

You should contact Heroku, to ask them to stop sending that 3rd DST_Root_CA_X3 certificate in the chain. And if you have only a few Heroku apps, you can fix temporarily by obtaining a Let's Encrypt certificate another way, and upload it on Heroku (their web dashboard allows you to do that). I myself changed the DNS entry temporarily, got myself a LE cert on another server, and then changed back the DNS to point to Heroku. I then uploaded me cert chain (two certificates; mine and the R3); and the private key, using the Heroku dashboard. Heroku now has 90 days to fix their side, and then I will be able to switch back to using ACM.

I am confused by this conversation you folks are having. My understanding is that Heroku has nothing to do with this. They are correctly using ISRG Root X1 letsencrypt certificates. It is the client's computer that is incorrectly using the old DST Root CA X3 certificate because they have an outdated certificate store.

Please correct me if I am wrong.

Anybody has an example of website that is running one of the new certificates so we can see if our devices can connect?

There's an official one here: https://valid-isrgrootx1.letsencrypt.org/

My PoC Xamarin app stopped working today... "The SSL connection could not be established."

Reply on news.ycombinator.com