Hacker News Reader

Comment by dsr_ | Hacker News Reader

If Jacques has been running the same system for five years, he probably missed five Nginx CVEs and 200+ kernel CVEs. Some of those required changes that ended up affecting the behavior of the system.

Certbot is not a replacement for a sysadmin.

DevOps is not a replacement for a sysadmin.

DevSecOps is not a replacement for a security engineer.

DevSecNetOps is not a replacement for a network engineer.

You need to have someone with the skills who applies those skills to your systems, or else you are hosed. This has always been true and is likely to always be true.

The good news is that cooperative projects make it easier. Let's Encrypt, the Debian project, the Google security hole finders, the whole CVE infrastructure -- the people working on these add an awful lot of value to the world.

View on HN

That system is about two years old.

As for the rest of your comment, you are somewhat underestimating what I can do but I like to automate away that sort of work because there are more important things in life.

And I expect things, once properly automated, to continue to function unless I mess with them. These cascading failures caused by some policy change elsewhere culminating in millions of installations having to go through sometimes costly upgrades are not the future I had in mind when I started my career in IT.

I don't underestimate what you can do -- I saw the LEGO sorter. That's nothing short of amazing.

But I am surprised at the attitude of "this automated system will continue to function". It's never been true in connected systems that care about security, and it's only true for limited timelines for independent systems. In the long run, everything decays.

But in this particular case there was no good reason why it came out the way it did. And that's what bothers me more than anything about it, that such a popular package would force users to expend a lot of effort for zero gain. It's just make-work.

Unattended-upgrades plus checkrestart plus periodic reboots for kernel upgrades can get you a surprisingly long way.

I think one problem was that LetsEncrypt's marketing failed to make it clear that it's a tradeoff: You get automation and free, but at the cost of extra sysadminning headaches. I was initially sold on both the cost and the turnkey nature of the system, but realized only after 3 months that it needs to be constantly babysat.

Are you suggesting the three people should be required to use a mailserver?

I can't speak for Jacques, but when I deliver software, I try to avoid any breaking changes for as long as possible. In the event that breaking changes are absolutely unavoidable--and there seems to be a question about whether that applies in this case--then I communicate clearly beforehand in simple terms about WHY the change is necessary, and WHAT the steps are to implement it, and I ensure buy-in on the part of those using my code. I try to make such changes as simple as possible, preferably "run this script and check the simple expected output."

That does not seem to be the focus here. The update might not have been strictly necessary in the first place, and if it was, it wasn't communicated consistently well to customers, and the communication doesn't seem to have been so simple and straightforward, either. Barring the communication, humane error messages would be nice, but those are also lacking.

I guess I would add to your list: when a company hires sysadmins, security engineers, and network engineers, they should also hire some UI people and good product managers to communicate with the outside world.

I run several static websites. Thanks to S3, they do not require a sysadmin. Thanks to Google, they do require regularly updated certs. We run certbot from an AWS instance that exists for about 10 minutes/day. Are you telling me that my little projects, which I have purposefully designed to require minimal maintenance and which cost me ~$10/year in static hosting, require an actual sysadmin?

I'm saying that someone needs to know what they are doing, and check up on them when something goes wrong. That person is the sysadmin.

If you don't know who the sysadmin is, it's you.

> If you don't know who the sysadmin is, it's you.

Brilliant. I'm printing this out and taping it to my wall.

Put Cloudfront in front of your S3 bucket and they Amazon will handle your certificates for free.

Egress traffic is cheaper too I believe.

Reply on news.ycombinator.com