Hacker News Reader

From a different perspective...

... perhaps if we were to actually get serious about doing small/incremental updates all the time, we'd get better at actually doing them without incident.

I think one of the major obstactles is that the software stack has become ridiculously (and hideously) complex over time. If we could converge towards a world more like what's described here[1], then I think we'd be in a better place overall. (Granted, this the video is talking about deploying services, but AFAICT there's little essential difference between that and a modern GUI system with all its DBUS interfaces and whatnot.)

[1] https://www.youtube.com/watch?v=GaHzdqFithc

> ... perhaps if were to actually get serious about doing small/incremental updates all the time, we'd get better at actually doing them without incident.

That would require more developers to actually give a fuck about the user experience not breaking from release to release.

I simply expect that doing an upgrade of the system (as opposed to grabbing the latest version of applications and libraries) is going to totally destroy my box and I'll spend the day recovering, because that's happened to me multiple times, from multiple OS vendors.

Hopefully (and I realize that this is somewhat wishful thinking), more users doing haphazard upgrades and complaining about them not working would be feedback into the whole cycle.

(I'm not very well placed in the software cycle to actually start this feedback loop, but one can wish, right?)

Given the general quality of response to user reports in free software ("The source is there, fix it yourself", "WONTFIX", "That's a feature", "you're doing it wrong."

Small, incremental upgrades are something I'm finding are essential to maintaining sanity as the only dev at my company doing what I do. Waiting to upgrade only increases the pain. Now instead of having one potential issue to troubleshoot every now and again, I have dozens when I finally do buck up and upgrade.

So I get in the habit of upgrading the dependencies of all the apps I work on, every time I work on them. Issues happen, but only to one dependency at a time. It's manageable.

What I would love is to eventually have a CI server do it for me. Every single day, it would run bundle update, run the tests, and deploy unless there's a problem. If there is a problem, it drops me an email with the trace and fixing it becomes part of my morning routine.

If subtler problems surface this way, then I've discovered an oversight in my test coverage, or an overly complicated architecture that I need to remove dependencies from.

I'll probably implement this sometime this year. I'm thinking I'll want to redo deployment instead of relying on Capistrano, then finally growing my own CI solution. I'm slowly moving away from big monolithic apps to smaller, homegrown solutions that do only what I want them to do. I've already reimplemented provisioning and configuration management. I believe in DevOps as code.

> What I would love is to eventually have a CI server do it for me. Every single day, it would run bundle update, run the tests, and deploy unless there's a problem. If there is a problem, it drops me an email with the trace and fixing it becomes part of my morning routine.

Agreed. A while back, there was a security fix in the default php5-fpm configuration (0666 -> 0660, I believe), and I skipped it when updating. My website went down until I learned to change the owner of the process. And this wasn't even a dist-upgrade.

Sometimes manual intervention is required.

In my case, I run updates almost every day. If I had to wade through 6 months of backlogged updates, I would have wasted a lot more time identifying where it broke. Smaller feedback loops are a win.

Gemnasium provides something similar to what you describe in the context of Ruby applications with Gemfiles. It can be configured to automatically generate pull requests updating outdated dependencies, which could trigger your test suite to automatically check for regressions.

I use Travis CI to maintain my Ruby gems, and one thing that I do is turn on a build that allows failure, where it uses the HEAD version of my upstream dependencies. That way I know within a day when upstream has done something broken, which is especially nice if it's by accident.

The problem is that OpenBSD has had some pretty significant user-facing changes in only the past year. The past few releases have seen apache replaced with nginx, and the next version is going to be getting rid of nginx for their own httpd. If you're just using the basic webserver, that's some pretty significant change right there. While they're advertising clean and simple, they can and do introduce potentials for heavy breakage in nearly every release.

Presumably that's just the default web server, and you can use the webserver of your choice as well?

The old code is one pkg_add away, though. Breaking binary compatibility is at least as likely to hurt you - and simply upgrading PostgreSQL without a dump-reload cycle can hurt plenty.

That still requires admin time to update the configuration and change management tools, QA time to run a regression test that will exercise all the potential failure paths, development time if you count on the ability to do configuration updates based on application state, etc. Updates that look small can still require considerable changes.

If you're used to it, you can streamline these processes and automate as much as possible. Even if it's a change you can't easily test or automate, by making just the one change at a time you know what to blame if it breaks. :)

Maybe that's an indication that the configuration of the web server should be partitioned into "basic configuration" (which they can all support) and a few other categories (which are supported by a subset), i.e. the configuration itself should be "standardized".

I realize that this is a nontrivial proposition, but if we're ever going to realize the dream of "interchangeable parts" we're going to have to learn to do this stuff.

I've held jobs at both ends of the spectrum, from bleeding-edge to death-by-legacy. There's a good deal of pain with both approaches, that's the reality. Would your org rather deal with security vulnerabilities and no upstream support, or deal with things blowing up once in a while?

What I like about this article is it's talking about what the global optimum is -- what's the best approach for the entire ecosystem? Not just for one individual or one org. What's Pareto optimal? (Or maybe it's a little less general -- what's the best approach for those who depend on FOSS, and don't pay someone like RedHat for long-term support and CYA insurance).

And basically I agree. The world's upgrade cycles could stand to be a whole lot shorter than they are. Every time I think about all the hardware in this world that'll spend its whole lifetime vulnerable to heartbleed, or shellshock, or a gazillion smaller vulns, my skin crawls.

> They take more than a year planning a big OS upgrade.

That sounds painful.

I think the author is really advocating for a faster feedback cycle. Imagine if instead of a big upgrade every X months, you had a tiny upgrade every day, which didn't require rebooting and happened in the background.

Breaking changes could surface the same day they were shipped and be rolled back or fixed.

Now imagine that conservative players also constantly upgrade, but always stay 6 months behind the bleeding edge, getting only releases marked as "tried and true". Eg, if something had to be rolled back or patched, they skip over the bad version.

Wouldn't that make upgrades much less painful for companies like you describe?

You do realize he's talking about a stock exchange? Even a millisecond downtime is unacceptable. Some things don't even need backup, others need testing on identical hardware in identical configuration before deployment. That's just the way it is.

To be fair, it's _unscheduled_ downtime that's not acceptable for a stock exchange. They have scheduled downtimes all the time. For example, as far as I can tell the NYSE is only "up" 9:30am-4pm local time, Monday to Friday, and not on certain holidays.

Now this does mean that you have to make very sure that the upgrade you start at 4pm on Friday will be totally done and bug free by 9:30am on Monday, of course.

And I agree with your larger point that in a situation like that testing done by someone else in a different configuration is of limited (though nonzero) utility.

I understand, and I admit I haven't done anything like that. But isn't it still easier to thoroughly test a day or week's worth of OS changes at a time than to test 6 month's worth or a year's?

> As a ubuntu user, I find that I cross my fingers every time I run sudo apt-get upgrade. About half the time I get broken builds and once a year my OS will just flat out crash from it, or fail to reboot in virtual box.

What I'm about to say has been said a thousand times, but I guess we need to keep saying it as long as this myth survives. Are you using PPAs? Are you using PPAs that provide newer versions of software that also ships with the system? Are you using third-party drivers (looking at you, Nvidia)? If so, there's your problem. It's not that difficult to blast away your PPAs with "ppa-purge" prior to running a distribution upgrade. As for regular package updates - "apt-get upgrade" won't break a system, that's not been already severely broken by the user. Period.

I use Ubuntu on the desktop and the server, and have done many upgrades over many versions. I don't recall a post-upgrade broken system in recent years that wasn't due to third-party stuff. Even then, it's trivial to fix - reboot to recovery mode if you can't get your X11 on, "apt-get purge" the offending stuff, reboot and carry on.

My impression is that Ubuntu screwed up on a major version upgrade several years ago, and their upgrade process is still being dinged by folks due to the wave of bad publicity that that incident created. Which is a shame, because Ubuntu post 12.04 is a rock-solid operating system that is a joy to use (upgrading included). Granted, third-party packages can sometimes reduce the overall stability, but then what do you expect? It helps to carefully vet which PPAs you use - PPAs that are done by people who know what they're doing don't tend to cause trouble.

If you want a rock-solid system, do a clean install of an LTS release, don't set up PPAs, and enjoy. I have a side Thinkpad with 12.04 on it which is exactly that, and it's the most stable system I've ever used. And it's being kept current with security patches and bugfixes with regular "apt-get upgrade".

I know group who does REBOOT once year, just not to forget how it is done.

I appreciate what you're saying, but your seem to not be addressing the point of the post: you could reduce risk by moving to rolling upgrades. A year of changes is a big change, smaller updates are not. There's also a huge risk in stasis.

Not that it's relevant, but to respond to your first point: you know people who roll their browser, and a few Linux distros roll their release too. I've heard of some startups who use a Fedora rather than CentOS for that reason.

You find the author's view laughable, but I find your view laughable. Reasonable deployments of Ubuntu or Windows get upgraded with security fixes all the time, browsers update themselves all the time. Waiting a year between security fixes is just foolish.

The problem isn't the security fixes. Everyone agrees that pushing security fixes promptly is generally a good idea. The problem is all the other changes that you may or may not want that get bundled in with those security fixes in a forced-upgrade model. Certain web browsers are among the best examples I know of this user-hostile behaviour.

The entire point of the article is that "all the other changes" include security fixes nobody knows about.

They also include new security flaws nobody knows about.

> What about every single car produced today. I doubt you'd want to upgrade them every 6 months.

Tesla cars get automated over-the-air updates far more frequently than every 6 months.

> I don't think I could find a single person, technical or otherwise who didn't use the same operating system for more than a year.

You can. Think AWS systems (as an example) where you don't log in and patch, you spin up a new instance instead. This can't apply to everything (stock exchanges, MRIs, etc), but for the software affected by glibc, absolutely.

In fact its the default for almost everyone who isn't technically inclined to upgrade their os only when they get a new computer.

Most new computers come with a warranty that is 90 days. Why should software support be 5x longer?

Most new computers come with a warranty that is 90 days.

Where? In the UK, consumer protection laws would probably force a minimum of two years in most cases. Even after that, if an expensive device mysteriously failed on day 2x365+1 in a way that wasn't reasonably expected, the purchaser would still have some relevant rights.

Why should software support be 5x longer?

It is all about reasonable expectations. No-one paying for basic system software would reasonably expect it to stop working after three months.

Of course if we're talking about Open Source software that you're getting for free then it's your own responsibility to make sure it does the job you need and no-one owes you anything. This is part of the reason that selling support for otherwise free software is a viable, and sometimes highly lucrative, business model.

US, Canada, Mexico. While no one expects hardware or software to stop working after three months supporting it forever isn't really worth it either.

Depends on the country. In most places in Europe that would be 2y minimum.

Because hardware warranties and software support have completely different purposes? Seriously, this is a non sequitur.