Hacker News Reader

Comment by raziel2p | Hacker News Reader

> Somewhere someone has decided that Letsencrypt should auto-update even if absolutely nothing had been changed on the system that it was running on

Letsencrypt is not a piece of software you install on your systems. You're probably thinking of certbot, which is just one of many available ACME (Letsencrypt-compatible) clients that can manage your TLS certificates. I've used dehydrated and acme-tiny in the past, both work well.

The article is also devoid of any details on what the weird technical issues you ran into that make you think you need a completely new machine. For all I know the machine was running a 10 year old version of OpenSSL.

I assume the change in the challenge protocol being referred to is ACME v1 to ACME v2? Why not just say this outright?

View on HN

> Letsencrypt is not a piece of software you install on your systems. You're probably thinking of certbot

This is such a shitty mischaracterisation of how LetEncrypt markets themselves. Typical "move fast and break things" kind of victim blaming excuses for a total lack of professionalism on their part. Want to be a part of a significant portion of the web's security stance? Take some fucking responsibilty for making sure you don't break crap and then claim "It's not LetEncrypt, it's ACME!!!" This is why so many managers keep signing off on huge Microsoft bills. They might ship crap, but they rarely break your productions systems on you like this...

A two year old install of mail-in-a-box. I am not in the habit of reinstalling my main mail and DNS server just for kicks, it is a lot of work to put it mildly, and very easy to mess up.

> I assume the change in the challenge protocol being referred to is ACME v1 to ACME v2? Why not just say this outright?

Because this has now happened twice, for entirely different reasons.

The first time letsencrypt-auto totally hosed a machine, the second time mail-in-a-box (which was otherwise humming along just fine) spat out an incomprehensible error about it not having enough authority to request a new certificate even though nothing whatsoever had changed on that machine.

Without any details, it's uncertain what the real issue was. It could very well have been attributed to a misconfiguration on your machine.

I manage over 20 client domains over a dozen servers. Have letsencrypt protecting them all. The only time I needed to manually intervene after the initial domain + SSL setup was to upgrade to ACME v2. Everything generally works out of the box with Letsencrypt + Certbot.

Good, so this hasn't bitten you. I'm sincerely happy for you. But my experience so far has been pretty rotten.

The real issue was a deprecated protocol, one that - as far as I know - could have been kept running at zero cost to LE because it already worked just fine.

Likewise I have had issues 3 times. In none of those cases did I really understand what went wrong, the fix was blow it away and redo it. Something just gets messed up the answer the community comes to is just start again. Its hard to answer the question of what went wrong when the error message is often garbage and gives no clue and just says the certificate can't be updated and there is no information on what just got broken.

The answer for me each time has been the age old turn it off and on again with redoing the install (or changing the way I utilised it in docker completely) and I have just found the whole thing more hassle than SSL certificates ever were.

The responses here are an interesting mix between those who have not had it happen to them (yet) and those who have (or have had it multiple times).

The former can't imagine the latter are anywhere near competent, it could obviously never happen to them so the person experiencing the problem must somehow be at fault.

It's the message board equivalent of 'can't reproduce', closed.

My comment was not accusation of incompetence, rather the lack of details provided. It's like receiving a ticket saying "This doesn't work. Fix it now"

> The first time letsencrypt-auto totally hosed a machine

Fair, I also don't like certbot as a software, hence naming alternative ACME clients. Though to get to a point where it completely hosed a machine sounds impressive.

> the second time mail-in-a-box (which was otherwise humming along just fine) spat out an incomprehensible error about it not having enough authority to request a new certificate

As far as I can tell, Mail-in-a-Box doesn't do any Letsencrypt certificate management so this confuses me.

I find it hard to support your conclusion that this is Letsencrypt's fault and not, for example, Mail-in-a-Box.

Mailinabox uses letsencrypt to issue certificates for locally hosted web properties (mail in a box is a bit of a misnomer, it does much more than just mail).

Letsencrypt shut down a protocol version that Mailinabox critically relied upon and which worked flawlessly as little as a few months ago. There never was - as far as I could see - a user visible warning or a warning to the email address supplied with my Letsencrypt certificates that something was about to change and I'm not in the habit of monitoring each and every such component. Especially not because the whole idea of Letsencrypt the way it was originally promoted was that a free CA would allow for automatic updates of certificates taking away a pain point: to remember to upgrade your certs once per year. Even the biggest orgs got bitten by that one. So we now have 90 day valid certs that get replaced automatically because 90 days is much too short to do it manually.

But then renewal and issuing of new certificates should be as stable as they could possibly be and changing protocols without backwards compatibility does not fit in that particular playbook (at least, not according to me).

To be clear on the facts, Let's Encrypt hasn't shut down ACMEv1 (yet).

Accounts with domains which were already renewing via ACMEv1 can continue to do so, until June 2021. Only new domains are shut out.

To me, seems fair enough. If you're booting up a new server, you may as well use a supported ACME client version.

I do think that Let's Encrypt/EFF did make some big mistakes all those years ago on the ACME client side, though, which has directly translated to ongoing user suffering.

Python was a not a good choice. Innumerable efforts have gone towards figuring out packaging issues purely caused by the choice of language. The CI alone is ridiculous. I don't understand why they didn't use something easier to package, like Go, which was sufficiently mature in 2014-2015 (well, duh, Boulder is written in Go). Certbot had to be available universally and needed to have a simple dependency and packaging story.

Relying on external, volunteer distro package maintainers to figure out the rat's nest of Python and native dependencies required by Certbot without blowing up other packages is a problem too. God bless those volunteers, but those roles needed to be funded and given priority, because being unable to get a remotely recent (i.e. not years out of date) version or any of the DNS plugins is honestly one of the worst parts of using Certbot. Today's solution (Certbot snaps) isn't a panacea either.

I somewhat suspect that Jaques' issues may have been avoided entirely if packaging had gone down a different route.

I've contributed a bunch of patches to both Boulder and Certbot (and will continue to) but it's honestly a bit heartbreaking to see that the CA side do such a good job and the client is stuck in a revolving door of issues relating to their choice of programming language.

That acme.sh is a fuckin' gigantic hacky shell script (this is a compliment Neil, I promise) and is, in many ways, superior to Certbot, is quite disappointing.

This post isn't meant to dump on the past and current maintainers either. I have an abundance of empathy for them. I can see how many of the decisions would have been reasonable at the time, and how the evolution of the protocol seriously complicated things. I see only happier days for Certbot ahead and hope to be a part of that.

> If you're booting up a new server, you may as well use a supported ACME client version.

This assumes that the new server is what will be running the new ACME client but in this case it was just an alias added to an older installation in order to redirect the naked domain to the actual web server, which indeed was installed from scratch. I'm sure that's an edge case that nobody at LE thought of or cared about but adding a domain does not necessarily mean you are going to be doing a lot of work on that particular machine.

I sympathize with your python pain. The fact that I can't run software I wrote three years ago in the current environment is what turned me off of python for the rest of my career. I consider it to be good enough for toy projects but not solid enough for any long term project. The investment in time and effort can all be negated by breaking changes and the fact that just about every application seems to want to ship a complete python runtime along, which will in some magical way interact with the system(s) already installed.

One machine I was working on had 3 different python installations on it and that caused no end of trouble. The only python distro that I think is good enough is Anaconda but it is super bloated.

> That acme.sh is a fuckin' gigantic hacky shell script (this is a compliment Neil, I promise) and is, in many ways, superior to Certbot, is quite disappointing.

See also dehydrated. It's what we use in many places, and I find it much easier to understand what's going on that certbot.

There's quite a list of dependencies that get pulled in when one does an apt-get install certbot, and I'm not quite comfortable with that. I can generally follow a shell script though.

Agree. There are a number of Go-based clients, e.g. acmetool and lego.

As Andrew "bunnie" Huang said recently: "Python is a great glue, but a terrible structural material. Do not build skyscrapers out of Python" (https://bunniestudios.com/blog/?p=5863).

I can vouch for dehydrated. It only needed updating once (probably due to the mentioned protocol change), and it's running ever since on all my servers without any issues, or updates, or bloat. It's just a shellscript. (several years already, around 4-5)

Reply on news.ycombinator.com