Hacker News Reader

Comment by ohazi | Hacker News Reader

I can sympathize with the frustration, but I disagree with the criticism. Certbot is capable of doing dry-runs, and is configured by default to try to renew with a month to spare. It logs failures. You should read your logs.

If you've built a system where silently failing is a problem, and you haven't configured any sort of "email me when anything isn't nominal" then you haven't finished configuring your server.

Software updates happen. Unattended updates are a thing. It's not the ninties anymore... You can't hook something up to the internet, chuck it in your closet, forget about it for five years, and expect zero issues. You need to pay attention to it.

View on HN

>You can't hook something up to the internet, chuck it in your closet, forget about it for five years, and expect zero issues. You need to pay attention to it.

This kind of contempt for users is why I try to avoid using any libraries/software written in Python. Somebody always thinks their minor backwards incompatible change is such a precious snowflake that it justifies wasting collectively thousands of hours of users' time debugging and updating their stuff to work around it.

I don't think this is "contempt for users," it's a position that configurations that appear to setup a web server to run without monitoring for years are deceptive and harmful to the admin taken in by them. It's an ethical (and technical) position that tools should push their users to adopt maintenance practices that are seen as responsible by the wider community. It's not contempt to inform people about the level of work you expect is required to do what they are planning. It's also, if letsencrypt happens to be wrong, not contemptuous to be wrong!

The one with contempt for users is reality itself. Computer security is hard, and whatever you have installed today will force you to update it later, directly or indirectly, due to security.

Don't blame the messenger.

I don't think that's what the author says. There's been some changes in the past to the acme protocol requiring to update whatever client you use.

It isn't contempt to realize that security landscapes change, and what was best practice in the past is insecure now.

That's not a Python specific problem, but otherwise I agree with your point.

No, but when it occurs, it's usually Python

> You need to pay attention to it.

My whole reason for switching to LE was that it was announced as an automatic way to upgrade certificates. So it required less of my attention than before. Or at least, that was the spiel.

> So it required less of my attention than before.

And that is why we run it. But we still monitor.

First we pipe stdout things to syslog via logger(1) to record each run in cron; however stderr is not redirected so we get that sent to us. Further, in our hook scripts, we have an "apache2ctl configtest" when a cert is renewed to verify that Apache (or equivalent) can read the files correctly, then we do the restart/reload. The configtest is sent to stderr as well so we can track when renewals happen and when there's an issue.

On top of all that, we don't actually trust our ACME client(s) to work properly all the time, so we check the expiration time on our certificates. We're monitoring the HTTP(S) service anyway, so adding another check doesn't add much cost.

And all this runs in the background silently and it only becomes top-of-mind really when it shows up in our service alert dashboard.

> And that is why we run it. But we still monitor.

The difference for me is that Let's Encrypt has required this kind of manual intervention more often than the previous status quo. Previously, I would put an item on my calendar to renew the free StartSSL cert, would do it on schedule, and things would generally go off without a hitch.

For me, the automatic renewer has fallen apart multiple times (for several reasons, including failing to reload nginx after renewing the cert), more than once a year on average, and worse still, they don't fail on any kind of regular schedule. I'm not a pro, just someone who wants a cert for my personal site, so I don't have any complicated monitoring system set up. This means I have to regularly remember to check up on certbot and see if it renewed the cert this time.

We've had one minor hiccup when LE changed the company that they used for front-end load balancing, but other than that it's set and forget. Generally we don't think about LE/ACME at all at $WORK. We still have some annually-renewed certs for 'appliances', including a wildcard, but practically we don't think about either 'system' more than the other.

If spending $x per year solves the problem for you, then go for it, but it does seem odd to me that you're having so many problems.

As for monitoring, a simple thing to do is to install some well-tested, pre-canned check scripts (e.g., from Nagios):

* https://packages.debian.org/search?keywords=monitoring-plugi...

and have them kick off via cron on a regular basis. A non-zero error code will cause an e-mail to be sent to you.

While we use Nagios (for now) to monitor from a central host, we also leverage them in things like keepalived for running HA clusters to trigger fail-overs. No sense re-inventing the wheel.

Interesting perspective. For me it was clear from the beginning that I'm trading convenience for free certs.

Criticism of the "official" client was also very loud from the start.

I do agree that LetsEncrypt has been a constant source of maintenance problems over the years. For example now, I have to switch to another alternative acme client, since the old one doesn't seem to be in active development anymore...

However, certificates used to be expensive, even unaffordable for some projects.

My time isn't free.

Try renewing certificates by the hundreds the old fashioned way and do a comparison of time used. Let's Encrypt is a godsend.

Yes, I can see how that would change matters. But I don't run hundreds of servers (fortunately).

Pay somebody to handle your servers, that's the easy answer for "I don't want to spend time on it" and "it should always work forever without me doing anything ever".

Good point.

You should demand a refund. Vote with your wallet.

Of course that was a bit trenchant, however you're complaining about a promise that was never made. No one ever promised that you could setup a old certbot instance and it was out of sight and mind for perpetuity. There are any number of issues that can occur, and honestly if one expected certbot to run without issue, having it automatically updating as well seems to be a base minimum.

Also worth noting that LE was early with ACMEv1, but a lot of alternatives started with ACMEv2. ACMEv2 became the common standard.

Yes, but that hole was closed almost immediately as far as I know.

No, it was not. It's a protocol bug. The hole was "closed" by deprecating and then removing tls-sni-01.

Since my college days, I've been running a Linux server, of some sorts, somewhere. Usually running Debian. To be honest, I also haven't configured any sort of notification when anything isn't nominal. Once I did set up Zabbix. But it turned out that Debian/Linux is incredibly reliable. When I migrated to a new server (or rather VPS), I simply haven't bothered installing any kind of notification. It just runs and runs, it's like an appliance.

I can see people falling into the trap. They run Linux their whole lives, and it never failed them. Then they introduce something new, like Certbot, and for the first time, there's a hiccup and they get frustrated :D

> You need to pay attention to it

If I have to pay attention to it four times a year and regularly spend a day updating software I didn't want to have let alone update, I should have just paid GoDaddy or whoever their $100 grift and only had to update the cert once a year and not hunt down stupid error messages and look for release notes on display in the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying 'Beware of the Leopard.'

Amazon haven't done something this stupid with their 'free" ssl certs for load balancers, and you know why? Because the provide them to "customers", not to "early adopting cheapskate beta testing schmucks"...

>and you know why?

because they control the software on both ends and can update certificate clients on the load balancers anytime they want?

I've been using let's encrypt nearly since it started issuing certificates, and I had to intervene only once so far.

I think one thing that's missing from the discussion is scalability.

Sure you can pay a CA for a cert that is valid for a year, but you still have to do your update dance once per year per site.

With let's encrypt, you write an Ansible playbook (or whatever it is you use for automation) once, run it everywhere. You select a TLS monitoring service (or run your own) once, enter your URLs, and you're done.

You only have to intervene when problems arise (and that can happen with CAs that you have paid as well).

The root CA certificate used to sign the AWS RDS certificates has recently expired and everyone needed to update the root store on their servers. Those who didn't pay attention lost connection to their database:

https://aws.amazon.com/blogs/database/amazon-rds-customers-u...

It is actually easier to deploy a repeat certificate you bought as well. Once you have it all setup it is usually just a drop in replacement and a config reload of the web server. Multiple times now I have spent hours fiddling with certbot trying to get it to do its job properly and stop invalidating my certificates. SSL has never been as much hassle as it has been with letsencrypt. Its free and I feel like I am getting what I paid for!

> It logs failures. You should read your logs. > If you've built a system where silently failing is a problem, and you haven't configured any sort of "email me when anything isn't nominal" then you haven't finished configuring your server.

I mean, I built a system years ago then got a full time job. Shit breaking every 3 months is kinda bad for a hobbyist website / blog. Even when the monitoring is in place, there's ample room to decide 'you know, family emergency is more important than rearchitecting my TLS certs again.'

Running your own server connected to the Internet is not a hobby and nobody should get the impression that it will be. Otherwise you're just making a donation to a DDOS swarm.

Do we have any real evidence that hobbyist servers are significant hitters when it comes to DDOSes? The idea that hobbyists shouldn't run a server is, IMO, misguided, and dangerous to the health of the internet.

A simple debian VM + nginx + $app_server with automatic updates turned on is not rocket science, and I struggle to see why one shouldn't run that as a hobbyist.

It hosts a static website with no meaningful network dependencies other than the control plane and letsencrypt. I'm relatively comfortable with the situation.

This is a great comment.

There are a lot of people who are going to get tripped up because they aren't really system admins. Or it might not be that important if it fails. If you only have to mess with it rarely, then it's probably not critical.

But for people who spend a substantial amount of time with their hands in servers, this comment is spot on.

>" It's not the ninties anymore... You can't hook something up to the internet, chuck it in your closet, forget about it for five years, and expect zero issues"

Ouch! This is just too bad. I suggest we go back.

You also get an email if your cert is about to expire.

> You can't hook something up to the internet, chuck it in your closet, forget about it for five years, and expect zero issues. You need to pay attention to it.

You should. 2 examples: - the average age of the planes for major airliners is 20 years. Maintenance - yes, retrofitting the cockpit every year just because a vendor decided to change the protocol for the sensor or control, no - about 18 years ago an intern in my company built (with external developers) an application for internal use; today the intern's career reached senior director level, left the company, the application is still used on a regular basis by every single employee; the only change was to move it from a physical server (there was no virtualization in 2002) to a VM. 5 years is not an unreasonable expectation from a damn thing to work without pampering

>the average age of the planes for major airliners is 20 years. Maintenance - yes, retrofitting the cockpit every year just because a vendor decided to change the protocol for the sensor or control, no

Civilian airliners don't face a HUMAN DRIVEN ADVERSARIAL ENVIRONMENT for general airspace. In the rare occasions they do, everyone dies. Internet connected devices all do all the time, and indeed that is the entire point of all these security measures and upgrade requirements. I don't know why this is such a frequent screw up in comparisons by people complaining about security. Why bother with TLS at all? Plain old http is a ton simpler and easier. Why not just run everything as root? Etc etc. Systems were much simpler and more hackable back in the day, but running that way these days is simply not doable.

If you wanted to compare to planes, the proper comparison would be military aircraft. And those do indeed have constant block updates over their lifetimes to ECM, comms, weapons, etc. Dealing with an adversarial environment with intelligent adapter attackers requires adaption in turn.

> I don't know why this is such a frequent screw up in comparisons by people complaining about security.

I have a suspicion that many, many engineers don't conceive of networked systems as participants in a human-driven adversarial environment. Many conceive of their software systems as little artifacts that function mostly independent of the outside world. When complete and functional, they should require no touching at all to continue doing so.

In other words, many people regard their computer systems as mechanistic and discrete things that can be declared "done" and treated as such. Many engineers work in shops where this is indeed how they work - ship a package and someone else does all the touching to keep it going.

I'm sympathetic. It took me a long time to make the jump to understanding software as operating services that always require maintenance, monitoring, and other forms of operation. Modern computers on modern networks simply cannot be treated as toasters if we want that level of reliability.

The paradigm shift isn't always easy or fun.

Yes, but you are taking the analogy a bit too far. This is more on the level of vehicles requiring constant updates from the factory in order to be able to still use the roads.

>This is more on the level of vehicles requiring constant updates from the factory in order to be able to still use the roads.

No, it really isn't, unless you drive roads where people are constantly shooting at you and trying to hijack your vehicle. And even then, nothing stops you from just driving your regular vehicle, but insurance won't cover it and manufacturers won't support you and so on. If you want their armor/security systems and indemnities, you must agree to constant updates to counter constant new threats. Don't like that? You've got other options at your own risk/expense.

Adversarial situations are simply more dynamic no matter what they're applied to. What we gain is worth the cost, though it definitely can be a real pain. And to the extent the pain points can be reduced without diminishing security it's always well worth pursuing (and indeed easiness is of course a big part of good security). But that's going to be in the form of "trying to make regular updates easier/more reliable" not "throw it in a closet for 5 years" short of some sort of far future formalized bug free system that doesn't exist.

Fair enough. Thank you for the insight.

I'm extremely sympathetic to the frustration that led to your post but, yeah, I think that was probably the best phrasing of the argument that you and I can stay grumpy if we want to but probably shouldn't get our way that I've encountered for a while.

If that system was mission critical, would you just leave it running there and not pay attention to it? No monitoring whether the VM is running, the disks are full, the processes are down?

Paying attention doesn't mean that you have to personally look at it every day. Set something up to let you know when something isn't right.

And as you mention: planes get maintained. Nobody lets a plane fly for 20 years without paying attention.

Planes are checked between each flight for basic issues, have a lengthy checklist to go through before using and are often under support contract from the manufacturer.

If you want that level of support you need to pay for it, and you can by using other CA:s. The rest of us we can use letsencrypt.

The first plane started in 1903, they have a bit more experience. Also they are crazy expensive and dangerous. Your mail server is not.

Mail and web servers should be a lot simpler to set up and maintain than aircraft too. But Letsencrypt, instead of making things simpler actually became a source of extra work, sometimes a lot of work.

Reply on news.ycombinator.com