Comment by forgottenpass | Hacker News Reader

Comment by forgottenpass | Hacker News Reader

Out of all the people commenting on #727708, The Spotify post is a rep from an organization with systems to administrate. Just like almost every other not-ctte, not-init maintainer person on that list, so it makes sense for Spotify to comment, but not really worth giving undue weight to their post in the larger debate currently happening on debian-ctte.

If you want a better picture of debian's init decision, the bug got punted to ctte to decide on, so check out the ctte ml archives for December [0] and January [1]. Russ Allbery and Ian Jackson are the most vocal CTTE members on the list and support systemd and upstart respectively. The thread "Bug#727708: init system other points, and conclusion" is where Ian [2] Russ [3] start making the case for their preliminary conclusions.

Edit: If there are other things worth pointing out on the list put 'em here. I've really only been skimming the posts for the last few weeks. They're also considering openRC, but it appears not to be a real contender. Tollef Fog Heen is a debian systemd maintainer and has posts worth reading on both what systemd actually is and what the impact of different decisions.

[0] https://lists.debian.org/debian-ctte/2013/12/ [1] https://lists.debian.org/debian-ctte/2014/01/ [2] https://lists.debian.org/debian-ctte/2013/12/msg00182.html [3] https://lists.debian.org/debian-ctte/2013/12/msg00234.html

I should point out that while this is theoretically the official position of our infrastructure team here at Spotify, we've also got a lot of opinion and debate internally (like you have on any topic among a large team of engineers).

One other fun problem Spotify is going to tackle is how we're going to handle systemd vs. upstart if/when we transition from straight-up Debian to Ubuntu.

Why are you transitioning to Ubuntu?

We're currently on Debian Squeeze so we need to move to something as Squeeze is approaching end-of-life.

If we stick with Debian, we'll find ourselves back in this situation fairly soon, since Debian's security team only supports oldstable for one year after a new stable is released (approximately — there isn't a set timeline available).

Ubuntu LTS releases, on the other hand, give us predictability and five years of support. That makes me very happy from an infrastructure standpoint. And the more up-to-date packages in Ubuntu repos make our feature devs happier, versus having to maintain backports for modern software.

Will those packages always be up to date even on the LTS Ubuntus?

Will you be able to support Postgres 12 and Nginx 3 on those machines?

I find it useful to deploy the application on the same OS it was dev'd on, but in a decoupled manner. The thing I despise the most is needless upgrades of infrastructure pieces of working apps (redis,python runtime,etc). Each should get there own static version.

A couple jobs back, I would package my own JVM along with the application (in an rpm) as ops was being really slow to upgrade machines. The only thing I depend on for a box is libc.

And how did you, or how do they now, handle security updates for this bundled JVM?

Sure - Ops being slow to upgrade is annoying, Ops patching a security hole and you leaving it open could be devastating.

There is a really good reason Debian forbids embedded copies of other packages, and why I despise "solutions" like Chef's OmniBus. These things are live grenades moments away from taking the whole ship down.

We would rebuild the rpm with a new JVM and redeploy the app after testing the fix.

Ha! How many copies of Lua are living in applications that Debian doesn't have control over? There are probably package maintainers excising those as we speak.

I don't want to get into a packaging philosophy war, not enough fun over text, needs to be face to face.

This is a good read, http://vagabond.github.io/rants/2013/06/21/z_packagers-dont-... on how over zealous single tree packaging can make a mess of things.

I had to bundle in my JVM because ops wouldn't allow more than ONE on a machine. I wanted

    /opt/jvm/1.6.22
    /opt/jvm/1.7.10

And I could symlink my apps to the one I needed. New apps could get new JVMs, old apps would continue to run just fine. But they wouldn't do this because it _broke_ Red Hat file system guidelines, for whatever definition of broke.

Reuse can absolutely cause over coupling. I prefer to have tractable dependency graphs.

Take a look at

* http://nixos.org/nix/

* http://www.gobolinux.org (defunct)

CentOS + collections would be my answer to those problems (with, presumably, the unstated "we don't want to pay a lot of money for support" as a third). But I'm happy with yum/rpm.

> Ubuntu LTS releases ... software doesn't get updated (except security patches) for five years

> more up-to-date packages ... no need to backport modern software

I get how Ubuntu gives you both of these features, but I don't get how it gives you both of them at once...

Ubuntu LTS tend to backport a good number of packages. For example, if you wanted a fairly recent nginx, you will find 1.4.x built for 12.04 LTS https://launchpad.net/~nginx/+archive/stable

Is there a benefit to using the nginx PPA rather than nginx's official repos[1]?

I use Debian Stable with several upstream repos, and I've never understood why people use a PPA when upstream provides a repo (e.g., nginx).

[1] http://nginx.org/en/linux_packages.html

The packages at release are more up-to-date, and the official backports repos are filled with newer versions.

The particular packages you need might be obtainable from Launchpad PPAs, which Debian doesn't really have an equivalent of unless you're willing to maintain your own.

If you don't want to phase-in updates rather than depend directly on a third-party PPA, you can also setup your own PPA and copy packages to it as you wish.

I'm going in the opposite direction - we're an Ubuntu shop, but I'm considering transitioning to Debian. Ubuntu is too much of a moving target, it seems, and seems to want to go its own way a little too much. It's focus is less on the server and more on the desktop and mobile, though I don't know how much that would affect future work.

One of our coders had to figure out something to fix a problem in his ubuntu desktop, and reported that he'd found a guide that showed it was done a different way in each of the four previous releases. I think they're doing fine for their original target audience - naive users - but I'm cautious about their constant change and go-it-aloneness.

> One other fun problem Spotify is going to tackle is how we're going to handle systemd vs. upstart if/when we transition from straight-up Debian to Ubuntu.

After two years of upstart I am wholeheartedly regretting ever touching it. It's broken by design and currently in such a bad way that you actually might have to reboot the machine to fix it.

Upstart is the main reason I'm considering dropping ubuntu for something else.

Works for me. What is the problem?

For example, there is a long-standing bug [1] where an improperly configured job using "expect fork" can cause upstart to become completely confused and unfixable without a reboot (or a really hacky ruby script [2]).

As the comments discuss, it's not really always clear all the time when to use "expect fork" vs "expect daemon" and the rules change with a `script` block. So this hits people all the time, including me.

It's these kinds of bugs in upstart that bite people all the time. I could go on about its obtuse configuration but I can declare unequivocally that upstart configurations are my least favorite part of system administration, bar none. `Start-stop-daemon` reduces some of the pain but is also difficult to debug.

1. https://bugs.launchpad.net/upstart/+bug/406397 2. https://github.com/ion1/workaround-upstart-snafu/

The recent discussion on lwn.net is also interesting:

Positions forming in the Debian init system decision: http://lwn.net/Articles/578208/

related: https://wiki.debian.org/Debate/initsystem/upstart

Russ's points are an incredibly well thought out and written discussion of systemd vs upstart and worth reading for anyone who needs background on the two.

Reply on news.ycombinator.com