Hacker News Reader

Comment by viraptor | Hacker News Reader

They haven't really published a list of good/bad clients. I'm interested in what's the practical cutoff point with mobile phones? I expect desktop browsers will be less of an issue.

View on HN

On Android, the root was first added in Nougat (~half of devices according to Android Distribution Dashboard). But I think that browsers (like Firefox and Chrome) on Android tend to bring their own cacerts rather than using the device's, so it's probably not as bad as it looks.

To that end, it was added to NSS 3.26/Firefox 50 (November 2016) and to Chrome 57 (March 2017).

On iOS, it was first added in iOS 10 (2016).

The root itself was created in June 2015.

Edit: they also have a list on their website - https://letsencrypt.org/docs/certificate-compatibility/#know... . It lists Android < 2.3.6 as being incompatible .. I wonder if that's updated for the non-cross-signed intermediates.

I was eating breakfast with a multitude of Android phones around me and four "older ones" could not access that site. The oldest that could connect was ~5 months old, all using new Mobile Chrome versions.

The reality is, for a bunch of usecases, you're gonna need to support 15 plus year old devices.

So Windows XP...

There are a lot of old systems out there running API's, automation, industrial systems, etc. They never get updates, and are expected to last decades. Most of them aren't on the public internet, but HTTPS would still be a good idea. This change is going to mean a bunch of them just get changed over to having no encryption.

This is because the IdenTrust root is expiring though, it's not something LetsEncrypt can do anything about.

They could get a new cross-signed intermediate from a root with broader compatibility.

Though I imagine that's extremely expensive. I expect that has something to do with this decision - they are a non-profit after all.

> it's not something LetsEncrypt can do anything about

This is going to cause a lot of stuff to break, and it's 100% LE's responsibility.

HN's root cert is valid through 2038.

LE could have gotten cross-signed by a cert that didn't expire so soon, but they didn't.

> LE could have gotten cross-signed by a cert that didn't expire so soon, but they didn't.

And they still could.

If you want to use an old, unsupported OS then don't expect support.

Expecting XP or say Windows 98 to still work on modern day standards is just laughable.

Upgrade, or get left behind. The concept of 'never updating' isn't one that is practical and you'll pay the penalty for it later on.

Consider that for mobile users, updated software might not be available for their hardware and they might not be able to afford new hardware.

Work for a university. We have a number of students in Africa, southeast Asia, and eastern Europe taking online courses. We do not support Windows XP or Vista, except for in the case of these students. Some of these guys don't have the resources for anything else.

I also know that we get a number of students connecting with Android v3 or earlier from these same locales.

Something like Ubuntu sounds like a great fit then.

Modern ubuntu is getting pretty heavyweight. Gone are the days it'll run on any old laptop.

Absolutely. I just refurbished old Core 2 Duo desktops. Ubuntu creaked along (it was Disco Beta).

I found Linux Lite, based on Ubuntu 18.04 LTS:

https://www.linuxliteos.com

Anyone running XP or whatever for a good reason will be able to install the new root certificate

Most industrial devices and even consumer kiosks probably access sites that are under the control of the device's owner or manufacturer. So the sites can just use self-signed certificates in the first place.

Or, for industrial use a hundred bucks on a certificate from an older CA is nothing.

Or, as 'vbezhenar hinted at elsewhere, the operators of the systems can distribute and install the ISRG root certificate to the devices that need them.

I built an app that's used on thousands of Android devices in an industrial setting. Most of the devices were acquired last year.

Just tested with the new LE root cert and it doesn't work.

LE says "it's CA problem, not a Let's Encrypt problem", but that's disingenuous.

Let's Encrypt chose to get cross-signed by a root that expires in a couple years.

For example, HN's root doesn't expire until 2038.

This is definitely a Let's Encrypt fuckup that will cause many sites and apps to break.

> I built an app that's used on thousands of Android devices in an industrial setting

If its that important to you or if its a commercial offering in an 'industrial setting', you should have no problems acquiring a cheap SSL certificate from another source. You can literally get them as low as $6 a year right now.

LE provides a great service and continues to do so. If you want to nitpick, then jump to a 'competitor'.

> If you want to nitpick, then jump to a 'competitor'

Sure, I can solve the problem by switching to a different CA, or by adding the ISRG root cert to each device.

But this is a problem that didn't need to happen. I blame myself for not anticipating it when I selected LE.

And I blame LE for cross-signing with a root cert that expires so soon. Not a good choice for a new CA that will take many years to be trusted on most devices.

Good to know, which also is the most relevant test I imagine with all android phones coming with chrome. Thanks.

I have a first-generation Pixel, so a couple years old now, and it accessed the test site fine.

That's still getting updates, isn't it?

Ah I didn't realize that's what he meant by five months old

That website list is for SHA-2 compatibility, not the new root.

That page seems to be out of date. I've filed an issue asking them to update it: https://community.letsencrypt.org/t/please-update-the-certif...

So iPhone 4S is out of luck. That's terrible move. Suddenly half of the Internet becomes inaccessible to a perfectly working device. Though I think that user can install new CA certificate, so it's possible to manuall work around that problem.

The iPhone 4S has not received a security update in >2 years, and has >700 known security vulnerabilities, so it is not a "perfectly working device," it's a "probably compromised device."

https://nvd.nist.gov/vuln/search/results?adv_search=true&for...

”Though I think that user can install new CA certificate, so it's possible to manuall work around that problem.”

Good point. Generally it’s a bad idea for users to install any CA certificates on their devices, but in this case I think it makes sense.

They provide a test site. It works on my Android One: https://valid-isrgrootx1.letsencrypt.org/

People with other versions of Android and iOS can test and report here?

Tested on a few tablets my company sell / used to sell :

- FAIL Galaxy Tab 4 7" (SM-T230) Android 4.4.2

- FAIL Galaxy Tab A 7" 2016 (SM-T280) Android 5.1.1

- SUCCESS Galaxy Tab A 9.7" (SM-T550) Android 7.1.1

- SUCCESS Galaxy Tab A 10.1" (SM-T580) Android 8.1.0

I don't have any Android 6 device at hand, but this is consistent with @regecks statement "On Android, the root was first added in Nougat" (which is Android 7).

This is going to be problematic, as there are still devices currently for sale on Android 5/6 (such as the aformentionned Galaxy Tab A 7", which doesn't have a replacement on some markets).

Tried the test site on a Nexus 7 running Android 6.0.1, Firefox was ok (seems it ships with its own list of roots), but latest Chrome rejected it.

My wife runs a blog which generates substantial income and uses certs from Let's Encrypt. It's a non-tech blog with primarily US readership. Checking stats for this month, 7% of all visitors were using Android 4/5/6 (20% of all Android users). The percentage of users on old Android running Firefox was basically nil. Losing all these users would be very costly.

Hopefully certbot will be modified so it is possible to pick the current intermediate during automatic renewal. If I have to do a manual operation to switch intermediates each time the cert renews (currently done by cronjob) then it is probably safer (operationally speaking) to just buy a cert.

I don't really understand why Let's Encrypt is making this change now. Sure, the current root is expiring "soon", but not until September 2021. Switching roots could be safely pushed off to early 2021 at which point hopefully most of these older Androids would be cycled out.

> a blog which generates substantial income

An SSL cert can be purchased for as low as $6 a year; if this is important to you, try buying one of those.

Isn't that what I said?

Edit: fortunately it looks like certbot plans to support using the old intermediate https://github.com/certbot/certbot/issues/6971 so this should not prove necessary.

As a tip to you/your company: unlike Chrome and most Chrome-based browsers, Firefox for Android has a separate root certificate store and your old devices will still be able to access Letsencrypt websites if you switch browsers.

Thank you for the tip.

The issue is not so much about the browser, as I'm an app editor (which happens to also sell tablets with our apps preinstalled to reduce friction). The issue is that apps that rely on the device certificate store aren't going to be able to use https with a server using a Let's Encrypt certificate issued with the new root CA.

Shipping a root certificate store would be (for my scale) a bad practice. I made the mistake of pinning a SSL key in the past, never again (you run into issues when your clients never even connect the device to the internet in 3+ years, and then your updater doesn't work anymore).

Fortunately for me, I don't currently use Let's Encrypt for my API servers, and that news was the last straw to make the boss decide we will stop selling Android 5 devices.

Unfortunately, this means our users who recently bought those devices will have some third-party apps might be broken starting 9th of July, and some sites will give a scary warning.

- FAIL on a Kindle Fire 7 (which is still for sale new) in Chrome running the latest update from November 2018 [based on Android 5.1.1].

> This is going to be problematic, as there are still devices currently for sale on Android 5/6

If I recall correctly, Android >=5 has pretty decent support for modern crypto (such as TLS V1.2, ECC). So, Android 5/6 could still just work, if the vendor is bothered to update the CA root store.

Vendor updates for cheap Android phones? Dream on!

This will indeed be a problem. Many old devices still run fine; outside of a password submission page, is https really worth the hundreds of dollars it would cost to replace my devices? I really don't know why people feel the need to get latest Android; outside of security, it's just gimmicks.

That site should have some text on it like "If you did not get any errors or warnings while opening this web page, your computer or device knows about the ISRG root certificate and Let's Encrypt will continue to work for you." Currently, this isn't immediately clear to (relative) laypeople like me.

That test site, though they do link it in their own announcement was actually created as part of their compliance with root trust programme conditions. Specifically Mozilla's conditions require them to prove their setup actually works (modulo the trust they're requesting) by setting up a web site with certificates that would work once that trust is granted. They were also required to provide example sites with e.g. expired certs so that a Firefox developer could check that does what you'd expect.

The good news is that unlike some of the required test sites, which took a bunch of advanced planning (you can't ask Let's Encrypt's service to mint you an expired cert so the expired cert was produced by requesting a valid cert and then waiting for it to expire and making sure not to lose it...) this test is easy for anybody knowledgeable to manually reproduce in a few minutes, as it's just the "wrong" certificate chain with the good trusted leaf certificate you got from Let's Encrypt. So even if they don't heed your call I'm sure someone else can.

* https://www.ssllabs.com/ssltest/analyze.html?d=valid%2disrgr...

Works on Nokia N900.

Reply on news.ycombinator.com