Why I still have an old-school cert on my HTTPS site | Hacker News Reader

Why I still have an old-school cert on my HTTPS site

rachelbythebay.com

16 pointspicture63 comments

That rant sounds like me some years and years ago.

I don't mind if the protocol if FUBAR. It works for me and it works for an awful lot of people. And it encrypts the web.

Before that, it was manually re-creating a cert, faffing with chains (converting formats and dealing with bad documentation) and update the production servers manually. It was a royal PITFA.

Don't forget to set a calendar reminder for the next year's renewal.

Or better, an alert to check the cert is still valid

You don't want an alert to check the cert is still valid, that's how you get outages. It's an alert to check cert has at least 2 weeks or more life left in it! (So it either fires ideally before you leave for that long vacation, or so that you can still take care of it when you get back.)

But also check it’s still valid, because revocations

This is all built in very nicely to https://github.com/prometheus/blackbox_exporter - Though they could use a blazing red link to an example rules file.

There's also issuance via DNS which took most of the complexity out of it for me. (Especially as the first time I went to deploy LetsEncrypt I had something like four web servers behind a LoadBalancer without the LB terminating TLS.)

I think the tooling has improved somewhat since too, although I didn't look too closely.

issuance via DNS is awesome when it comes to LE, lets you get certs without your service being reachable from the outside

> I don't mind if the protocol if FUBAR. It works for me and it works for an awful lot of people. And it encrypts the web.

To me, it feels like most stuff becomes much more troublesome once you peel back the curtain and drop down to the internals. For most folks (myself included), using a web server that handles most of this for you will be enough. There's Traefik, there's Caddy and plenty of other options.

Even Apache has support for ACME with mod_md, about which I wrote on my blog last year: https://blog.kronis.dev/tutorials/how-and-why-to-use-apache-...

It's not perfect (you still need a restart/reload for the new cert to be actually used), but the configuration side of things is simple enough and most of the issues are known: https://httpd.apache.org/docs/2.4/mod/mod_md.html

> Before that, it was manually re-creating a cert, faffing with chains (converting formats and dealing with bad documentation) and update the production servers manually. It was a royal PITFA.

This has always been a pain to me, for example, the OpenSSL CLI is a bit like tar in how unintuitive it is, vs something like the Docker CLI which walks you through a tree of commands with all of their parameters that you might be interested in. Curiously, I've had most success with using graphical software for manually working with certificates, like Keystore Explorer: https://blog.kronis.dev/tutorials/lets-run-our-own-ca

The linked article is a bit barebones (I didn't feel like Googling for the OpenSSL equivalents for the actions to demonstrate the difference in UX), but I want to express the importance and usefulness of your software (CLI, TUI, GUI, API, whatever) telling you what options are available to you, which makes a world of difference.

In my eyes, there's no reason why a junior dev or someone who hasn't worked with OpenSSL (or an equivalent piece of software) in the past should have their first time doing so be miserable. Then again, needing to think about bunches of different formats and what technology accepts what is probably always going to be at least moderately painful or annoying: https://serverfault.com/a/9717

> Don't forget to set a calendar reminder for the next year's renewal.

Thankfully, this is pretty easy to deal with! Even if you use ACME, you'll still want this in place, in case automatic cert renewal fails, or if the cert just isn't swapped over to by the web server.

For example, for the self-hosting crowd, something like Uptime Kuma allows you to have all sorts of reminders, including certificate expiry: https://github.com/louislam/uptime-kuma (note: Uptime Kuma does not batch those notifications, I got about 30 messages in Slack at the same time because a bunch of sites had the same wildcard certificate that was going to expire)

Basically, it's nice to see that as a end user of software you can have a decent experience, for the most part. Definitely a bit better in some regards than what you might have been dealing with 10 years ago or so. Not that all of the problems or nitpicks are solved, or will be for the foreseeable future.

Personal worries: what if Let's Encrypt becomes a non-viable option for whatever reason? There aren't that many ACME compatible providers out there.

So, the post's title says they won't use LE, but the actual post's content has nothing negative to say about it and instead bashes some other free SSL cert service? Why not use LE instead then?

I am equally perplexed.

> This is about me finding a supposed alternative and going through the same process of due diligence to understand just what I'd be getting into.

I don't think the author has a good understanding of Let's Encrypt. Whatever service the author found claiming to be better seems to be lying, based on the author's experience.

For the benefit of the unfamiliar who stumble upon this, getting a Let's Encrypt certificate is a fairly painful process, unless you make it painful. Many hosting providers have the process built into their offering. For those self-hosting, the certbot client is very easy to use, after setting up your site and domain, which you would need to do regardless of what service you use to obtain a certificate.

It only gets complicated if you have a complicated set-up, such as a non-public site or are perhaps using some specialized web services software, in which case I would assume you are sufficiently technically inclined to overcome the obstacles you yourself built.

In any case, I find free and automated is preferable to expensive and manual.

When I first got started with LetsEncrypt, I wanted to do the whole thing manually so I could understand it better. This was probably 4 years ago.

I expected a web form to fill out (e.g. what domain to do you want, paste your CSR here, etc.), but such a thing didn't exist. The LetsEncrypt site didn't have much useful in the way of instructions except to say "Use Certbot".

So I set up certbot, ran it in manual mode, and got my certificate. Cool. The only problem was that there was a little too much magic. What is this "account" that I'm using to renew my certificate if I never made an account (i.e. username password). What happens if I accidentally delete all the files certbot created for me? And a bunch of other questions.

The documentation I found didn't give a whole lot of straightforward answers. Instead I pieced together information from blogs and forum posts.

I'm hoping the documentation situation has changed, but I can understand how someone can be confused about LetsEncrypt. Many people are happy to use something without any concerns about how it works. Others want to understand and when they find that they can't, they get frustrated.

When I decided to use LE for my website's HTTPS cert, I wrote my own ACME client from scratch based on the RFC. Never tried certbot, since from the beginning I wanted to to have the cert issuance process run in an Azure Function. The RFC (and the related RFCs for JWK and JWS) are very readable and easy to implement as long as you know how to delegate the crypto to libraries.

(I actually ended up writing three different versions, one in PowerShell, then one in F#, and finally one in Rust that I still maintain.)

Once I had it working, I wrote a blog post with a higher-level description of the whole cert flow, which you may find useful: https://www.arnavion.dev/blog/2019-06-01-how-does-acme-v2-wo... It links to the relevant sections of the RFCs, so you can follow along while reading the post.

Because she doesn't like the protocol LetsEncrypt uses? What a strange reason to avoid it.

> nothing negative to say about it [LE]

That's a very liberal interpretation of the article:

> It looks like it was created by people who had been drinking FAR too much of the web kool-aid, since it's chock full of terrible things. It should be a small amount of drama to start a process, receive a magic string, sock it away somewhere at a magic path, then poke the validator and say "go for it". Then you just check back and see whether it worked or not.

That seems to be about ACME, or at worst LE (the organization) not LE (the service)

You said:

> Why not use LE instead then?

If she doesn't like ACME, how would she use LE (the service)?

You can use LE without really interacting with the ACME internals. But to each their own I guess.

Sure it does - it says:

* ACME is a badly designed standard, a mess of binary-in-base64-in-json-in-canonical-form-hashed-sent-over-http-sent-over-ssl. The author doesn't like it.

* LE invented ACME, and can only be used via ACME

Personally I don't know enough about ACME to say whether the author is right or not - but I've worked with SAML so I know if extensible human-readable text formats are a foot then digital signatures are a gun.

JWS (the thing that the author is complaining about) involves a canonicalization step to resolve that footgun.

… well, I kinda like ACME. It's one of the few actually RESTful protocols out there, so it is quite straight-forward to navigate. The media types are well specified compared to some HTTP APIs.¹

certbot (presuming that's the thing she's talking about the code quality of) can be a bit hard to interface with if you stray outside the default set of use cases, IMO. I don't know why it would be shelling out to openssl, though, when Python has cryptography, which is IMO one of the most pleasant libraries to work with if you need to build X.509 certs or CSRs etc. as it represents those things directly and plainly, and doesn't get in the way.

… if the issuance limits on some random third party CA are a problem, … Let's Encrypt has generous limits.

¹my last experience was Github's API … which will happily have an attribute that's an enum, but not tell you what the variants are. Like … seriously? Azure's docs/data structs permit representing many illegal values easily, and don't document, e.g., when two things are mutually exclusive. The lack of REST in Azure's API means what could have easily been just a single attribute holding a URL gets broken out into 5 different attributes strewn across 3 different structs, a VM's initial image, I am looking at you.

Rachel's Site shows an Apache 2.4 banner.

Apache has a little known module as part of the build, at least in Ubuntu: mod_md.

Use this, it is trivial to set up, and completely automated the Let's Encrypt workflow and renewal, no external stuff necessary. It is as simple as adding a few lines to the global config, remove almost all SSL config and restarting the server (twice).

Yes, but the author doesn't seem to care about how easy it is to use existing software; all the criticisms here seem to be attacking the underlying protocol and nitpicking implementation details. I'm sure if she looked at mod_md she'd download the source code and find a bunch of (in her view, at least) problems.

She really complains how ACME is bad because a client implementation has code written in lines sometimes exceeding 200 characters..?

How about we stop trying to micromanage essential services on our servers, and just get on with literally anything else that brings actual value? Let’s encrypt works, it works without attention, it’s free, and ubiquitous.

Whatever technical merit the rest of the rant may have had, screaming that long lines killed puppies certainly burned a lot of provisional credibility.

It doesn't work for me, on one of my older phones.

If so, that's by design in a way.

That's an indicator that your root level certs are not being maintained or updated. That risk is worse than just a few certificate authorities not being recognized.

>Let’s encrypt works, it works without attention, it’s free, and ubiquitous.

It's not so easy, since there is always a tradeoff with these things. The exact same argument could be applied to Java as a backend and many people happily do so... until it turns out that some lone guy in Switzerland, who has thanklessly maintained a highly popular logging library for the last 20 years, forgot to sanitise certain inputs. Aaaand now you have remote code execution on Apple's servers and just about every other Fortune 500 company. The world needs more people who scrutinise widely used security relevant software, not less.

The inevitable folly of an engineer is succumbing to that nagging desire to tear down the existing solutions and build your own.

Old school? Old school was when you could put up a harmless, noncommercial, mostly static HTML web site and run it over plain http without being considered some info security risk outlier.

I have one of these web sites. After some coaching from a modern web literate colleague, it now autopromotes to https using a LE certificate. It works. Or does it? Half my relatives who access a (private and password protected) photo server promptly reported losing access. This is the still using 32-bit Chrome on Windows XP, save a password in a browser once and then never think about it again, kind of crowd.

True dat, way to many devs nowadays don't really test things properly.

Accessibility is something I've always taken a liken to, sometimes maybe obsessively. But them again I interacted and went to workshops with people from places around the world that don't have high speed internet still nor the latest devices.

Look at gov.uk ... That site is an accessibility masterpiece.

As someone who had to automate OpenSSL workflows often enough (fortunately, way in the past), the described automated signing workflow sounds much like an automated signing workflow. While I hope that the mentioned 200 character line width is being used more as a literary device (elevating snarkiness) than for an actual argument, I fear the latter.

Having said that, don't all (public) SSL signing workflows use web technologies in some way? To me, using some web technologies (aka 'kool aid') specifically made for this purpose, such as jwk, feels better than having some intermediaries sending back and forth CSRs, keys, and whatsoever using home-made interfaces and data structures.

This was really difficult to parse, unusual for rachelbythebay.

The article seems like it's ranting about Let's Encrypt (well it is a little). But a lot of the criticism seems wrong. Ah, the critical quote:

    > But what about now? This isn't about Let's Encrypt. This is about me
    > finding a supposed alternative and going through the same process of due
    > diligence to understand just what I'd be getting into. Somehow, a couple
    > of weeks ago, *I found this other site which claimed to be better than LE*

It's ranting about some other scummy service that's never mentioned by name!

It's kind of a scam with accounts and logins and a free tier that only lets you do 3 90 day certificates. Rachel likes the web API better. Well, yeah you still want to steer clear of that! (is it https://zerossl.com/pricing/ perhaps?)

So then why not circle back to LE? The reason nobody seems to care about the ACME protocol is that it does get the job done. And if you don't like the "certbot" client, which can be kind-of a pain, there are like 50 other implementations.

https://letsencrypt.org/docs/client-options/

So you might not be able to do wildcard domains and the renewal period is a relatively short 90 days (both by design), but it's trouble free and actually free with no draconian limits.

https://letsencrypt.org/docs/rate-limits/

Well, goodness please use whatever service you want. And if you dislike the code quality of LE enough to pay for a traditional certificate every 2 years, great. But it seems odd to be ranting about LE by name and then throwing a bunch of critiques about another different unnamed service in at the same time.

If you gave up on LE in 2018, you might want to try again. It's honestly pretty decently simple to setup and feels pretty opinionated.

I would have loved to know more about the setup process and cost for the "old-school" cert.

Looking at its metadata, it appears to be from Gandi,[1] which charges US$16/year for a single-address cert that works on standard hosting. (The "free" tier is if you buy Gandi web hosting, which starts at US$6.20/month.)

1: https://www.gandi.net/en-US/security?taxes=no

Automation is king with Let's Encrypt. I just add a daily cron job to run certbot renew and don't worry about the rest.

Perhaps the process for obtaining a free 90-day LE certificate will be automated in TLS proxies so it is executed automatically every 90 days. Maybe the Kool-Aid eventually wears off and the reality sets in. No one likes "certificates as a business" except those who profit from it.

It has an infrastructure cost. Either it's donated or it's cost is recovered. The shifty part is not being very up front about this although I suspect bits of this are tech "do I look like I read the manual" things.

I use letsencrypt. It's less shifty than normal CA to me.

Guessing she is talking about acme.sh, the 8000 line shell script.

It has lines like:

  _dnsAltnames="$(${ACME_OPENSSL_BIN:-openssl} req -noout -text -in "$_csrfile" | grep "^ *DNS:.*" | tr -d ' \n')"

But in its defense, acme.sh works on all major OS

So you auto renew every 90 days. Is that really a problem vs doing it every year or so with a paid cert? What am I missing?

If you're talking about the "saner" service OP talks about at the end, I read that as 90 days and then you have to start paying

LetsEncrypt with auto renewals via certbot is free unless there's some misunderstanding here.

I missed this too, took three readings to de-conflate that Rachel's talking about some other unnamed scummy service, for the bulk of the writeup not certbot/LE.

Ah okay. I suppose you have more patience than me. :)

I hadn't realised the certbot implementation was quite that bonkers. A while ago I spent far too long trying to get certbot+nginx working sensibly in docker, before giving up and using caddy instead, which embeds an ACME implementation into the HTTP server. I can't speak to the quality of it though.

Park that shit behind Caddy. Case closed.

> 270 days for one domain or 3 domains for 90 days, and then you're screwed. Isn't that great?

Huh? I haven't looked at the fine print but I use quite a few more domains. Never ran into the rate limits.

With what provider? That part is talking about some unnamed alternative, not letsencrypt.

Instead of whinging you could just write your own script, if for some reason you're not okay with running the same code as tons of other people?

certbot also now insists on using snapd, even on Debian. There's still certbot packages in the distros' repos, but who knows how long they'll continue to work.

I sometimes use an older phone to browse the web. I've noticed that LE sites just don't load.

It just says "can't establish a secure connection".

Larger sites don't have this problem.

I'm not sure about the exact year, but I doubt LE will work for devices before 2018. I think they changed their root cert at some point relatively recent.

Edit - I stand corrected. Likely I've been confused by a few devices and Linux distros I had that simply broke on that day.

Also, I remember having to manually fix a few Ubuntu servers that day when this happened. Anyone else had that experience around Oct 2021?

The big transition happened in October 2021-- that's when Let's Encrypt's root stopped being cross-signed by IdenTrust (allowing older devices that had the IdenTrust root but not the ISRG/Let's Encrypt root to still validate Let's Encrypt certs): https://letsencrypt.org/docs/certificate-compatibility/

I corrected my message - but I am still pretty sure that some of our Ubuntu servers broke that day, despite having been updated. Unfortunately I can't remember the details. Good to know that this didn't affect most phones etc.

https://letsencrypt.org/docs/certificate-compatibility/

Support goes back pretty far.

Not far enough it seems, the link you posted has the same problem.

Edit: this particular device is iPhone 4

How old is your phone? Android v2.3.6 and iPhone 5 are supported, which are pretty old.

So according to the link, you would have to have android 2.3.5 or earlier for it not to work (circa 2010. The final release of that branch was 2.3.7 which should work but maybe your phone provider didnt provide updates for your phone)

It looks like iPhone 4 runs up to iOS 7.1.2, and according to the above link, LE now requires iOS 10 or later.

In theory you can use MDM to add the LE root cert to your phone's certificate store.

> this particular device is iPhone 4

Okay, so Wikipedia says the newest OS for that phone is iOS 7.1.2, which was released June 30, 2014. So you've got a phone with no OS updates, including security patches, in ~8 years. Considering that AFAICT Apple bakes webkit updates into the OS, your browser has not been patched in 8 years. Bluntly, HTTPS certs are the least of the reasons that your device has no business being on the internet.

> Bluntly, HTTPS certs are the least of the reasons that your device has no business being on the internet.

What a silly statement. Any device any user chooses to has business being on the internet. That's what the world wide web is.

I dont think that's true. People originally thought that was going to happen due to the root certificate expiring in old android devices, but turns out android does not enforce expiration on roots, so everything ended up ok.

https://letsencrypt.org/2020/12/21/extending-android-compati...

No, they didn't.

What happened relatively recently was the DST cross sign expired; but any device should, by that point, have received the ISRG root, or you haven't gotten security updates in the last 7 years.

Additionally, there were/are problems with some TLS libraries (e.g., GnuTLS) that have bugs in path building. (Fixes have been made, since LE not working is pretty obvious.) They'll build a path to the expired DST cert, and then conclude (incorrectly) that no other valid path can exist. (When in fact, there is, to LE's own root.)

(Additionally, I've also seen vendors corrupt the ca-certificates directory, and then OpenSSL fails to do path building. This is sort of a pathological case, and probably not your phone's issue.)

> Larger sites don't have this problem.

All larger sites? Wikipedia uses lets encrypt (although it may depend which data center you hit, i think, not sure). Stack overflow uses lets encrypt.

If its only small sites maybe lets encrypt isn't the problem.

Yes, Wikipedia sadly doesn't work for me on this device.

Reply on news.ycombinator.com