The sad state of sysadmin in the age of containers (2015)

I don't think either is necessarily wrong; there's not a huge distance between `apt-get install unattended-upgrades` and a daily (weekly, whatever) container rebuild except that you need to cron the latter. Unless you're using a different OS between the two cases, they can use the same package database, the same tooling, and so on.

"With containers, you are responsible for figuring out which ones need which packages and how to re-build them."

Yeah, you need to know how to rebuild your world in ANY CASE. It's not an argument against containers that they require an approach that is the best practice.

Are all the systems out there in production actually unmanageable and unmanaged?

Software is becoming more complex and moving faster than ever. 'Controlling your dependencies' was always an illusion and a trade-off at best. Often a trade-off against security, inheriting the folder of JAR files from your predecessor and such.

I think of containers and all these new tools as designs which are supposed to help us manage existing complexity. They don't create it, it's already there in the requirements and real-world deployments. The friction we feel, is that some of those tools are not very good (yet), but maybe better than the previous generation (sometimes). On the negative side, there is a lot of nostalgia ("ah, do you remember bare metal...") and unwillingness to change and learn.

If someone doesn't like apt & the main repository they shouldn't use Debian because that is pretty much the only big thing separating Debian from the tens to hundreds of other distros out there. Take apt and the main repo out of Debian and it is basically Linux From Scratch with a bigger team of bugtesters. Distributions aren't 'jealous' about packet management, they are the packet management. It is right there in the name "Debian distribution". The Debian brand, and all linux distro brands, are deeply linked to how they execute packet management to distribute software.

I'm sad that this is not happening since in the process we're also losing the good bits of system package managers as per this article.

> install packages for a single user

because of the security angle, Not allowing this:

> install multiple versions of the same package

is completely indefensible.

Who are you, package manager, to decide that I shouldn't be able to use different versions of the same package? Do you know more about my context than I do?

[0] We should have also been following 11 but just didn't have the resources to even think about doing it.

I use it for most of my dependencies, and software in OS X/Windows WSL/Linux

It is a lovely environment to work with. Has completely changed how I feel about package management and the like.

I know you were asking figuratively but something does happen to slot in that space.

Not criticisms that apply to maven. Binary dependencies on released versions are the norm, and all packages in the central repository are signed with GPG, just as with apt.

> The computing world would be a better place if the people forcing their language-specific idioms on the rest of the world had instead spent their time improving the platform package managers that already existed at the time.

If those platform package managers had been open to being improved, which I don't think they were (and conversely, deb/apt originated because Debian insisted on doing things their own way rather than improving RPM). Apt's closed-world assumptions seem like a policy decision. Apt not being cross-platform is definitely policy. And surely it's occurred to people in Debian that the ability to have per-user installs, and more importantly non-shared/non-overlapping installs of libraries that are depended on, would be useful; the insistence on single system-wide installs of libraries can only be a policy decision, and one that's turned out to result in a less usable system.

All true. Did we learn nothing from the left-pad incident?

The three cardinal virtues of a programmer are laziness, impatience and hubris. I've never seen a decent argument against using ever-smaller libraries (as long as our dependency management is good enough to track them); all of the usual arguments for reusing libraries still apply, even if we take them further than their originators imagined.

> System package management systems do things just fine. Ask yourself, “when did I last properly package a Debian library? Do I even know how to do it properly?” Most people won’t be able to answer that because they didn’t.

Well, as we're seeing, Debian is struggling to keep up with packaging of everything that users want. I have actually created some (unofficial) debian packages; it's fine, but it's not a particularly inspiring experience either. Creating a maven/pip/... package is generally a nicer experience.

> But truth be told, having each programming language a unique package manager sucks. I’d rather have a consistent way of managing packages on a given system so that I can use various programming languages.

In principle I agree. But I'd definitely need to be able to install independent copies of the same library, different packages for different users, and fundamentally just have a nice development experience when working on that package manager. And that's not something I see the sysadmin tradition being able to come up with.

Not really. Language-specific package managers are usually capable of integrating with "system" dependencies from outside the language (certainly maven has decent support for this) - they kind of have to be after all.

> 3) fundamentally, not very good, particularly with regard to multiple versions of the same package (technically, OSGi could do that...), repeatable builds, and the desire for do-what-I-mean behavior.

How so?

    install:
        [ $(id -u) -ne 0 ] && echo 'please run sudo make install' || curl https//my.thing | /bin/sh

I want to like Nix, but it is far too pedantic to be practical. And this isn’t even mentioning the usability issues.

There's a world out there of LAMP/LEMP stack web developers that wholeheartedly disagree with this perception.

Without a deterministic build I don't know what I check in actually works across environment, or if the deployment artifact works.

And just why do you "Have to follow the herd"? Pretty sure just about everything new and creative (good or bad) was a result of not following the herd. I recommend going where you need to go. If a herd starts following you, that's great. At least they'll be slipping and sliding on the shit you leave behind, not the other way round.

It's not very surprising since 'products' nowadays are more like 'services' instead, and offer some kind of encapsulation: people are not interested much in how something ticks behind the scenes, and so it can drive down the behind-the-scene quality. They also are conditioned to accept lousy excuses (including none) for outages and breakdowns, because they got sold 'magic', and boy that is magical..

You'll might have to work in obscurity to keep up operations quality, which is in turn a driver for your own demise. Game over.

I'd like to suggest to you the following reading material:

- Bullshit Jobs (David Graeber, 2018)

- The Dilbert Principle (Scott Adams, 2000)

- Future Shock (Alvin Toffler, 1970)

There's nothing "complacent" about this: previous generations also relied on infrastructure and didn't plan for prolonged power outages or had backup ham radio network links for when AOL was down.

People using nom install instead of custom makefiles aren't ignorant or stupid, they have found better ways to achieve their needs. And if 10 years on the job don't create the need to learn some skill, there is no reason to invest time into it. And I have complete confidence that people would be able to come with some workable solutions rather quickly if the githubcopalypse ever happens.

There is some cultural component at play among the "luddites" here as well, maybe comparable to preppers? It feels like planning for really exciting emergencies when one's skills that have been derided for so long are suddenly needed and safe the day. In this analogy, I guess Makefiles are the equivalent of very masculine hunting and zombie-defending skills.

Now I'm losing sleep worrying if there's any way he can accidentally add a github.com remote to our private repo and push to it. I would be blamed, and I'd have to explain to managers older in turn than me what both git and github are.

I’m young enough for this to not be something I have experienced, but I don’t have to have to understand that depending on things you don’t have control over can be a bad idea.

That's putting it lightly. To me "terrible" would mean they just didn't support it, "batshit insane" means they supported lock files but ignored them every time you ran npm install.

My favorite comment from this stackoverflow post: https://stackoverflow.com/questions/45022048/why-does-npm-in...

"Why would you expect something called package lock to lock the packages? Package lock is analogous to how, when you put any key into a door lock, the lock reshapes itself to match whatever key was put in, then opens the door. Now if you'll excuse me, I'm late for a tea party with a rabbit."

In certain industry branches this is even mandatory if you want to be seriously considered as a supplier. If a build tool does not support reproducible builds in such a way (both fixing dependency version and getting it from a cache somehow) or makes it difficult then it is considered as a hobby toy that has no place in the workplace.

Even for small businesses I would advocate to take this seriously from the beginning. It's not that hard and will save you headaches later on when suddenly reproducible builds become important.

Even if the downtime is fine, I don't find it goes over too well if framed as "we rely on a 3rd party service, that we have no contract with nor any guarantees of reliability or product longevity."

If yarn would fix a particular bug that blocks our workflow I'd be burning political capital at work to get us off of npm as fast as humanly possible.

As to proxies, we have something misconfigured with ours, such that occasionally it gets latest of half of the React or Babel ecosystem and latest-1 of the other half, resulting in dependencies that can't be resolved for a few hours when they increment a minor version somewhere.

That's exactly what you should be doing, at least if you're relatively small scale. Google probably takes a different approach but most of us aren't working at that scale.

Second: Yes, generally you can use a build system like Jenkins, and a registry like Artifactory to automate the process. The docker image is updated, and then you can push it out with your orchestration in whatever method you choose. It's not an obscure or difficult thing to manage..

Seems a good thing, to me at least.

On the other side of the coin, putting stuff in boxes is the essence of architecture. CPU opcodes? Boxes. Compiling readable code into bytecode? Boxes. Objects? Boxes. Functions? Boxes.

Sure, it can be abused. There might be spaghetti in those boxes. The spaghetti should be chopped into digestible chunks (smaller boxes!). But putting it in a box makes that an implementation detail. Putting stuff in boxes is still progress.

What containers does beautifully is dependency containment. I make something and put it inside a container, with the exact dependency it needs and it'll work alongside other containers with a program that might have a different incompatible dependency.

What we're seeing is not so much that developers suddenly decides to throw best practice into the wind. They just do not have to put the same amount of effort into fixing the broken process exactly because we have dependency containment. I believe we would see the same mess without containers, and the process change that would fix one would fix the other.

Already done: Solaris zones. Available in the SmartOS distribution near you. Combine with OS packaging, imgadm and vmadm commands for maximum impact.

Before containers we would script the install process using Ansible. With containers we script the install process using a Dockerfile. If you know how to do one, you know how to do the other. Just don't be lazy and use unofficial images created by people you don't know or trust. Just like you wouldn't pull any random role from ansible-galaxy. Or pull a random guy off the street and ask them to provision your infrastructure for you.

Then all you have to do is just rebuild your images every week and you're set. All we had to do for this was have our Jenkins build pipeline run weekly & have it pass `--pull --no-cache` to `docker build`.

Do they though?

Whenever I write Dockerfiles that depend on external downloads, I always check the hash matches one baked into the Dockerfile itself, and I've always seen others doing the same.

You can say exactly the same about vendors on a VM - applications running as root, bad passwords set, reboots impossible, patching impossible, backups... Or a virtual appliance. Or a third party application you installed and cannot upgrade due to a weird dependency on libc, or java, or a specific windows versions.

Vendors like that are always a mess, no matter the technology. Containers can turn into an equal mess, yes, but that's up to the build chains and ecosystem built around the containers.

I tried it recently and was shocked at how awful the syntax and usability was.

It's begging for a tool to auto-detect your dependencies from source. I was trying to port an existing project to it and I gave up when I realised that I'd have to manually create about 30 CMakeList.txt files or whatever they call them.

Meanwhile during the same weekend, autoconf allowed me to build gcc from source with three commands. I know I know, I'm not supposed to like autoconf, and the orthodoxy is that it is garbage ...

It seems every generation ignores the previous generation's tools and invents worse ones to replace them.

If this continues we'll soon go back to banging rocks and sticks together.

If you make developers pull double duty as sysadmins, then outsourcing surely makes things better for development. But it's still more work than handing the infrastructure and administration over to people that specialize in that. A well managed service is a ticket away, not getting someone who "knows docker pretty good" to have another background responsibility to neglect until it sets on fire.

Throughout the cloud boom I've worked at a company with a great IT department, not just a glorified helpdesk. The division makes sure that we provide a sane deliverable that doesn't need constant handholding and 10 hotfixes a day, and they give us the peace of mind that it's not going to go sideways. They sweat the details so much that the rare issues that have escalated to the head of IT and myself at the same computer investigating something were still relatively minor in the grand scheme of things.

I've worked at a bank. It's a docker file. Running Hadoop. On top of a Linux Container. Running in Rhel 5. On top of a vmware hypervisor.

- docker containers were built from source

- Dockerfile published with the code

- Built in a new CI environment

- Pushed, Pulled and deployed from the sha

- Collecting network traffic, undertaking protective monitoring, that looks for those backdoors.

Just because you can pull arbitrary bullshit doesn't mean you have to.

Though for the record, the same sysadmins that whine about newer tools are generally the same ones that implicitly trust their older toolsets. Just because you can compile it, doesn't make it secure, so you need to be running monitoring solutions and hedging your bets no matter the tech.

Of course once we come to enterprise environments the opaque build/deployment process is atrocious.

Security has a cost, as does the lack of it, hence the trade-off.

Think about your server consuming resources: It needs a [machine] password stored someplace in order to authenticate itself to those resources. If it stores this encrypted, then an operator must be present to decrypt it when (re)starting the server. If this is stored unencrypted, then it is not and the server can be started unattended (or autoscaled or whatever). That is a security tradeoff because there are both real benefits and risks gained and lost with each approach.

If docker was ever a tradeoff, it was between taking the time to have a thoughtful architecture versus trying to get acquired, and I think we know which way they went.

I made rather complex images that included a large database (e.g. 30 GB) and Java apps, and found the easy way to do it was write a program that analyzes the problem, writes a shell script that builds a bastion and builds the software, then put that in the cloud-init... The machine "phones home" via SQS to tell the controlling server to shut it down and image it.

I've tried building it for my day job, I would rather have a colonoscopy without sedation.

It's more pleasant and dignified.

And good luck running Spark without Hadoop ;)

No one is willing to invest the time to untangle the build process and fix compat issues in the software. The project is slowly dying out.