Packaging Kubernetes for Debian
lwn.net
lwn.net
This also heavily influenced our choice of OS away from ones that emphasized LTS package management and towards more recent kernels with better BPF and kTLS support.
(We also intentionally used short-lived TLS certificates to prevent anyone from running a cluster longer than several months without updates. Worked great. There are workarounds of course but anyone aware of them is the kind of person who updates anyway.)
Because the underlying infrastructure gets replaced, it means spinning up a new cluster, deploying the application on it to make sure that everything is working. Then switch the traffic to it. If the cluster hold any state this can be quite a bit of work to replicate all of the data.
What will you do if that happens?
It's a bit like upgrading Postgres. It's a good idea to at least make a backup before changing the underlying storage.
We’ve also found breaking bugs in Kubernetes that prevented in-place upgrades in some cases (which we reported upstream and were patched).
Most companies today are better off NOT running distributed anything, as everything will get harder to do, from logging to deployments. People start with clusters when they really should start with centralized services and only go distributed when they need to.
What's important is to put the tooling in place to make the deployment reproducible. Deploy using Terraform and Docker to the single VM. Avoid modifying things manually. That's what provides the agility to then scale the infrastructure into bigger shapes as needed.
That's why we developed https://github.com/numtide/terraform-provider-linuxbox , to make that use-case even easier.
Once you get more revenue, hiring an operations/devops person might be a good idea. A good operations/devops person might make you save his own salary in infrastructure optimization (depending on the scale of course).
another good pattern that i've seen in action is contracting the operations to an external company that's specialized in doing this. the approach would be to let the external company design and build the cluster for you, define responsibility boundaries for your company vs that company, and let such company take care of daily operations and cluster upgrades, possibly with on-call availability. A small support/consulting package (say, 15-20 hours/month) can get you very far.
We've been doing something similar with SigHup (sighup.io) and their Kubernetes Fury distribution, it's been working very well so far.
(disclaimer: i'm not employed by sighup, just pleased to work with them)
It's so important to be deploying early on. Get the kinks out of the system and the code aligned for production.
If you hire a full time devops, they will most likely build too much infrastructure to keep themselves busy. And if you don't do any devops, the transition can be quite painful.
I tried this approach, using Packer with Ansible to build the AMIs. It took 15-20 minutes to build an AMI. Would you suggest a different way of building the AMIs?
You might also consider using containers without kubernetes. Keep the same AMI, package your app as a docker container, make VMs download and run the latest release from a private docker registry on startup. Publish a new docker image tagged as latest.
Now you can rollout your autoscaling groups without running packer and ansible.
Could you define this term? I tried Googling, but it looks like there are just two references and neither is clear to me.
I think if one uses a managed Kubernetes keeping it up-to-date is pretty easy. I did an AKS update in 10 minutes.
200 library dependencies is not too many for Debian to handle, as long as they don't change very often and the work is useful for other packages. But Kubernetes' dependencies do change often, and there are not enough other Go systems around to spread the work around. This is not a stable system, therefore Debian can't ship it in a stable release.
That being said, OpenStack is technically still around, so it's not completely "dead". ;)
With static linking you have to track where the static library ended up (recursively) and then issue rebuilds in the correct order for all packages that contain the static library (directly or indirectly).
With embedded copies you have to search the source code archive for the copies and then patch and rebuild each copy.
In some ways static linking is more complicated to deal with, due to the case that the static library ends up in some other static library.
In case of shared libraries, only the affected library package has to be patched.
The benefit of a distro is shared, trustable work. Increasing the workload on the distro maintainers is not great, and Debian is an all-volunteer organization, especially sensitive to that workload.
Furthermore, rebuilding and distributing a large number of large binaries every time a vulnerability is fixed is harmful!
- It encourage users to delay security updates. Hundreds of millions in the world have slow or expensive or capped Internet connectivity.
- Makes the distribution unsuitable for many embedded/IoT/industrial/old devices with very limited storage
- It gives the impression that the distribution is bloated.
So, why can't there be a repo for the all the vendored things, and make a policy for the maintainers there?
By not having that escape valve, the pressure shoots out on the maintainers-- in this example the maintainer's work is made impossible as evidenced by the statements of the previous maintainer that it would take two fulltime devs to package this according to the current policies. So you encourage burnout on your volunteers.
This also encourages passive aggressive software design criticisms. Look at the very first comment here that shifts to talking about the "maturity" of the software under discussion. I'd be willing to bet I'd see similar flip judgments on the list discussion of this-- all of which completely ignore the monstrous build systems of the two browsers that Debian ships. So apparently there's an escape valve for exactly two packages, but no policy to generalize that solution for other packages that are the most complex and therefore most likely to burnout your volunteer packagers.
Keep in mind you are already on maintainer #2 for this package that still does not ship with Debian because shipping it per current policy is too burdensome. Also notice that you've got a previous maintainer on the list-- who already said this is a two-person job at least-- calling out the current maintainer for being lazy. It seems like a pretty lousy culture if the policy guidelines put policy adherence above respect for your volunteer maintainers.
This is a false dichotomy.
> By not having that escape valve
Please do your research before posting. Building packages with bundled dependencies is allowed, actually.
Having a handful of small files from 3rd parties bundled in few packages is relatively harmless (if they are not security critical) and allowed.
Having 200 dependencies with hundreds of thousand SLOC creates a significant burden for security updates.
Put security-critical code in some of dependency and the burden become big. Make the dependencies unstable and it gets worse.
Now create a similar issue for many other packages doing the same and the burden becomes huge, for the whole community.
> This also encourages passive aggressive software design criticisms.
I would call it outspoken criticism of bad software design.
Because this argument is often made, I think it's worth pointing out:
Just shipping a fixed dependency like `libssl.so` isn't enough to make the fix effective on the end users' machines. You also have to restart all the running executables that link in that dependency.
As far as I'm aware, Debian does not handle that for you. So even if it ships that small, nice dependency-only package update via manual or unattended upgrades, your long-running nginx will still be vulnerable until you manually restart it.
If the upgrade fails to restart nginx properly, your customers won't be seeing the pages they need to. If the upgrade fails to start sshd, you have just lost access to the system(s) you would need to fix. Plus my personal favourite: if the upgrade-and-restart breaks your message broker, EVERYTHING is on fire.
In most non-orchestrated, non-cloud-native environments the right way for security upgrades is to have them available, preinstalled and configured, but not yet active. What you do need is monitoring to tell you these things are waiting so you can apply them as soon as feasible.
Although to be fair, once you have orchestration, robust zero-downtime rollouts and a good CI to rebuild new versions as upgrades become available... that's a different story.
I don't buy this argument: It makes an arbitrary distinction between software in which the fix is reflected, and software where it isn't.
For example, if said upgrade breaks an on-demand job which isn't a permanently running process but invoked by your customers through nginx, your customers also won't be seeing the results they need to.
If you wanted unattended-upgrades to not have its changes reflect in the system, you might as well configure it into the mode where it only notifies you, instead of installing the new version automatically.
No, that's pretty much the Debian definition of "stable": software that doesn't need changing for multiple years. The Debian term "stable" has nothing to do with whether or not a program crashes regularly, but how often it requires maintenance. In that definition, "evolving" software indeed isn't "stable" (yet).
>Beyond Kubernetes, web browsers clearly fall into this category. Distributors have generally given up on trying to backport patches to older browser releases; they just move their users forward to new releases when they happen. The resources to do things any other way just do not exist.
Exceptions are already made for browsers, but they're browsers. They're practically essential to 99% of graphical debian installs and don't expose the really nasty unstable bits (like V8's api surface) to the world. I doubt the debian TC will make that exception for devops software with much less mindshare and a public API surface that is the software, at least not on the stable channel.
You can tell that is what the current maintainer was hoping for here. Instead, the previous maintainer-- who literally wrote that it would probably take two full-time devs to properly package this and maintain the package-- goes full Vogon and summons the great Debian bureaucracy to solve this with their poetry.
If that were true, one would expect the packagers at Arch to be leveling similar complaints about the "maturity" of the software or the "awful" build system used by Kubernetes.
I'm going to rankly speculate that:
1. That is not the case.
2. Kubernetes was packaged in Arch in less time than it's taken for Debian to discuss the policy disagreement between the previous and current package maintainers, neither of which have been able to deliver a maintainable Kubernetes package in Debian yet.
3. This says more about the efficacy of Debian's packaging policies and package dev UX than it does about Kubernetes.
I'm the package maintainer for go and maintain a number of go packages in Arch Linux, along with writing the current guidelines and looking into packaging strategies for golang.
I'll offer a more cynic view then the one presented: golang is not a mature packaging ecosystem.
So 1) is partially correct; go just sucks from a packaging perspective. Not really the issue at kubernetes.
We can probably package up kubernetes within the next hour and drop it into `[community]` as Arch has less structure and QA around the packaging process. However our largest hurdle is that we package the latest version. Is kubernetes going to work properly on go version `1.15.3`. By experience container software brings out the worst of the go runtime and any changes to the syscall, goroutine or memory management is a cause of concern.
The other hurdle is the cadence of when kubernetes decides to support which container runtime. Docker is wonkey at best, but I haven't seen a lot of details on cri-o nor containerd.
So frankly our problems are not really related to packaging, but the challenges of providing recent versions of packages and making sure they work.
Vendoring is shifting focus back to individual projects though. K8s could almost replace an OS in many regards. It is made to be modular, but only within its own framework. So it is changing the way software is run, similarly to initd and systemd. Such systems as we've seen, tend to "take over" and not be made to be part of and integrate well from inside an OS. Vendoring is really old practice, but also makes testing and support simpler for upstream. We all know the dynamic approach tend to be more unchanging, or associated with several circles of dependency hells.
It's just different end goals for software development and deployment, that affect how software is distributed. Kind of similar to touted goals of JEE of java fame, which also is too monolithic to integrate well within complex systems.
AUR does have a process to rate, flag and comment on packages. It is also much easier to review the installation script using PKGBUILD. There are a lot of high quality packages in the AUR but you should review everything you install unofficially, just like installing packages from a PPA.
Leave an arch system without an upgrade for a month and you're playing Russian roulette.
Of course, Debian doesn’t require manual intervention for most updates, but it also isn’t rolling release. If you want packages that are actually recent you have to sit on Sid, which is a lot less stable than Arch. And if you have a Debian machine that’s a couple years old and you want to upgrade it to the latest Debian version? Good luck. Some of that isn’t Debian’s fault, but whereas on Arch major, breaking package changes are rare but irregular, on Debian they tend to hit you simultaneously in one major upgrade. For home machines, I definitely prefer an occasional manual intervention and continuous fixes over major breakages every couple years where I often just give up and reinstall from scratch.
That said I use Arch a lot less nowadays as I’ve moved onto NixOS. Nix is clearly headed somewhere new, though it remains to be seen if the complexity of the approach is actually maintainable.
If I want to make an idiomatic Arch package, it’s usually easy: PKGBUILDs are intuitive and simple. Nix is much the same for most stuff, especially since most of the boilerplate for various build systems has been automated; though for a program as complex as Kubernetes its a nightmare, to be sure. For Dpkg though, it seems like so much complexity, and I don’t think I ever really had a good experience.
And honestly, I have no idea how to make an RPM package anymore.
To counter that with my anecdata, my Debian installations usually outlast the hardware they've been built on. I've found that it's much easier to start new builds with a backup (or the original disk) of the machine they're replacing, that's how easy it is to maintain a Debian system for me.
Heck, my first Linux desktop started out as Ubuntu Breezy, and ended its life as Debian Squeeze. Its replacement started out as Debian Wheezy, and is now on Bullseye.
There are other options. For example, you can sit on stable and get specific packages from unstable, so long as they don't have dependency conflicts with the rest of your system. It requires a little more thought than just running with the defaults, but the docs are pretty clear and detailed (man apt_preferences), and it's mostly a one-time operation. I do this with docker, the kernel, and a maybe a couple other things.
> And if you have a Debian machine that’s a couple years old and you want to upgrade it to the latest Debian version? Good luck.
I guess I'm among the others here who have had good luck, then. The first counterexample that comes to mind is a NAS appliance that is now around ten years old. I replaced its stock OS with debian stable (Squeeze, maybe?) and ran it that way until a year or so past the next stable release, then upgraded. It went fine. I did the same thing repeatedly over the years, until finally decommissioning it last month. It now runs Buster, so that's at least four major version upgrades done past their expiration dates. None of them gave me any trouble. The only thing wrong with it is that my needs have outgrown the hardware.
I find this comment odd and highly dubious. There is absolutely nothing wrong with Debian's packaging system. At most, some individual packages could have been packaged differently, which is arguably a matter of personal taste. Normally, everything just works, and works very well.
Either you shed some additional light on your personal struggles with Debian's packaging, which you did not do at all, or I have to scratch off your baseless assertion as a kin of someone on the internet shouting at clouds.
I can at least tell you that people don't choose Arch over Debian for the installation experience. :)
Well, I do (sample of one). The Arch installation has been streamlined a lot. Now it’s all about
0) boot the image (cd, usb, PXE), which is actually an Arch install
a) creating your filesystem (pick you poison), and mounting it
b) telling pacman to install base, base-devel, and a bootloader on the target fs
c) installing and configuring the bootloader
d) rebooting
Done.
d-i barely takes care of that for you, it’s “just” wrapping it behind a UI (which is sort of useful, sure saves one from reading docs, but has been an annoying abstraction/obfuscation/magic layer for me more often than not).
The remainder (setting up X/Wayland/whatever is no different on Debian than on Arch, as d-i does not help much.
And yet the number of DDs keeps increasing (and Debian is one of most successful projects).
Indeed it takes time to do packaging, and this is by design. Packagers are expected to thoroughly review the code they are packaging and smooth out various sharp corners.
Many times I've found bugs in the upstream code while packaging. Sometimes around security and privacy, often around documentation, usability or non-x86 architectures.
Every time I check if other distributions opened bugs or applied patches for the same issues. It almost never happens.
This is why I might spend 2 hours on a package instead of 10 minutes.
Ignoring the weird scaffolding you need just to package a static "hello world" binary, there's also all the dh_ scripts which you should use for your package to be "well-made".
I remember looking up how to properly package a Python application, found like 5 different ways documented on the debian wiki, couldn't get any of them to work, gave up and just shipped a whole virtualenv.
The "standard" way to build even the simplest deb file seems super overcomplicated whenever I've tried to do it, with layers of complexity, shims and fudges.
For most simple stuff you just want to specify a bit of metadata, a couple of install scripts and specify "X goes in /usr/bin/X, Y goes in /usr/share/...".
I tend to find that fpm (https://fpm.readthedocs.io/en/latest/) does that for me, and I now use it for building my own deb files pretty much everywhere.
What do you mean plain files with simple dependencies? A deb file is just a zip that stores metadata and your artifacts exactly where you want then to be in the directory structure you see fit. You specify the packages you depend on in the meditada, zip them up, and that's it.
If you need something fancy you can add shell scripts that are triggered in parts of the installation lifecycle, but those are far from being mandatory.
Hell, if you prefer to follow the lazy way out you can even get build systems to generate your deb files for you. Cmake handles this out-of-the-box.
> Ignoring the weird scaffolding you need just to package a static "hello world" binary
What? That "weird scaffolding" you're referring to is literally the directory you zip with the artifacts in their destination and the metadata! Is it too weird for you to deploy libraries in /use/local/lib before you package them? What are you talking about? You build your static "hello world" binary, you place them in the destination folder, you fill in the metadata to specify dependencies and stuff like your name and email address, and you proceed to zip up the package. Done.
> (...) found like 5 different ways documented on the debian wiki
No you really didn't. At most you found references to 5 different tools that help you do the same exact thing.
Here's a link to a process you tried to described as undecryptable:
this small "fill in the metadata" part is already half-an-afternoon of reading and understanding all the concepts and mandatory files in there: https://www.debian.org/doc/manuals/maint-guide/dreq.en.html ... and that's only one small part of the manual.
Arch packages in comparison can be a single 15-20 lines PKGBUILD shell script.
The built-in Debian packaging tools are already wrappers on top of wrappers on top of wrappers, adding another layer on top of that (with Cmake or whatever) only adds to the confusion and goes to show how over-complicated it is.
For example, the article you linked uses dpkg-deb --build while I've seen alternatives use dpkg-buildpackage and debuild.
The 5 different documented ways was specifically referring to Python packages.
There are also helpers like dh_systemd which I could never get to work so just ran systemctl commands in postinst and such.
That is incidentally the most reasonable way to ship a python applications. :D
https://wiki.archlinux.org/index.php/Python_package_guidelin...
Like any organic solution it works, and somewhat impressively works for all the use cases created by debian's 60k odd packages. However, pretty or simple it isn't. Learning the interactions of the various dh_ helpers with the build systems they try to automate, the "guesses" (term they use in the manual) they make and when they are applicable is a huge job. The learning hill is consequently steep, and I would say unpleasant.
However, I don't want to undersell the end result. The QA checks done FTP masters, lintian, the discipline imposed by testing and the long transition from testing to stable mean the end result is very, very good. Better than I've seen in any other distro, including the .rpm ones I used to use. From a sysadmin point of view, that trumps everything. From what I can tell debian could be described as a bunch of programmers beavering away for free to produce something that none of them could produce on their own - a Linux distribution so good the base their day jobs on it.
Also, Arch usually puts all files into a single binary package while Debian splits arch-dependent and arch-independent files into separate packages. Also, Debian separates packages into runtime and development libraries, another thing Arch doesn’t do either.
Arch definitely has separate packages for runtime and development libraries. It doesn't have as many, but they exist and can be found by simply searching for `-dev`.
> Arch usually puts all files into a single binary package while Debian splits arch-dependent and arch-independent files into separate packages
Arch has `any` and then architecture specific packages:
https://wiki.archlinux.org/index.php/Arch_package_guidelines...
> Arch doesn’t have to support multiple architectures and releases
https://wiki.archlinux.org/index.php/32-bit_package_guidelin...
There are also distros of Arch for ARM and 32bit, however if you're looking for a more integrated multi-arch PKGBUILD-comparable distro then Alpine is it and probably what I'll be migrating to at some point.
They are an exception to the rule, where the benefits outweights the negatives. It's been done to ensure we have smaller container images, or if the maintaine thinks it makes sense. But as a rule, we do not care while debian does.
>Arch has `any` and then architecture specific packages:
How is the `any` arch related to package splitting?
>https://wiki.archlinux.org/index.php/32-bit_package_guidelin...
> There are also distros of Arch for ARM and 32bit, however if you're looking for a more integrated multi-arch PKGBUILD-comparable distro then Alpine is it and probably what I'll be migrating to at some point.
Those package guidelines exist, but are dated and not used by us, the packagers.
Arch ARM and 32bit are also seperate distirbutions that isn't affiliated with the 64bit Arch Linux distribution.
How many are vendored in Chromium/Firefox?
If you're planning to run those as a platform, it sounds like more pain then necessary.
I've run openstack for a while and... realistically if you run a platform, you need to have people/time to manage it. I understand from conversations about k8s that is not that much different.
You also know, that your issue with that setup might get noticed by someone else using the same and will fix it for you.
You get sane defaults and you get k8s optimized for the os.
I run microk8s at home. Its simple, fast, easy and works.
Not seeing any issue with it.
(I use NixOS for personal use.)
The customers were screaming for it. They want a simpla and reliable way to deploy k8s without a team of specialist, and the guarantee to receive security updates for years without having to upgrade to a new version.
The article uses so many convoluted phrasings and arguments to avoid saying that. A bureaucracy of *.deb.
It's implied in few paragraphs. It's not that hard to infer if you take off your hat. Thou who read Corbet often know a thing having pros and cons is implied in his reasoning.
His skill is the warts are laid bare, but not unkindly and often with humour.
Some Debian people don't like it. Users don't care, as long as they can apt install it.
The good the about the distro fragmentation is precisely that, If debian curbs their ambitions then it is no longer debian, and those that prefer the debian way over ubuntu or fedora or arch would get screwed in the process.
It’s fine to not like it and not use it, but I do like projects that keep their ways.