20 years as a Debian maintainer
suihkulokki.blogspot.com
suihkulokki.blogspot.com
Debian is an awesome and somewhat under-rated distribution. Being a maintainer always seems like a thankless and slightly forgotten role. Thanks for having the persistence to keep going so long.
I installed Debian for the first time almost 13 years ago and have enjoyed the "Debian way" every second.
But as the saying goes, all good things must come to an end. Due to various decisions by the Debian community, Debian Wheezy will be the last version I'm going to install and for the last few years I have been in the proccess of migrating thousands of servers away from Debian.
In any case I want to take this opportunity to note that FreeBSD is quite nice as a daily driver on your workstation. The only missing thing is bug-free suspend/hibernate, which works for some and for some doesn't.
edit: added missing adverbs.
Of course, you don't mention macOS. I never had issues with suspend on any of my MBPs. If I did, it turned out that was my battery got empty, and the few times this happened I did think about suspend failing. Turned out I was wrong.
I said "I GUESS ACPI is a mess", because most implementations I've used (and I've only used it; I know nothing low-level about it at all) have had some problem or another.
How wrong of me not to have any experience with your preferred platform, and leave it out of the discussion.
> How wrong of me not to have any experience with your preferred platform, and leave it out of the discussion.
No need to feel so overly offended.
I've come out the other end looking for alternatives... In the meantime I'll stick to an OS that uses init, and hope systemd get's better given enough time.
I am looking at trying out GNU Shepherd as it is the init system of the Guix distribution, so you get both Nix-style package management and an init system that is not systemd, both written and configured in Guile Scheme.
https://www.gnu.org/software/shepherd/ https://www.gnu.org/software/guix/
I switched a couple of months ago from Arch, and was surprised at how easy it was, and how completely useless (at least for my use case) all the systemd steaming mess had proved in retrospect to be.
One thing I do remember before Systemd though was the fact that every service was essentially a glorified bash script, and I hated that. With Systemd, there seems to generally be a much more clear-cut definition of how things operate, without all the cruft.
My biggest issue with systemd is that when something goes wrong, it's pretty impossible to tell why. As an example, virtually every system I've used with systemd after a few months starts to have different services fail to stop on shutdown, causing a timeout of (by default) one-and-a-half minutes. From searches that I've done online, I don't seem to be the only person who runs into this, but I haven't found any good solution, leaving me with the options to reduce the timeout to something more manageable, force shutdown my laptop literally every time I'm done using it, or sit through 90 seconds of systemd trying and failing to stop whichever service is failing that time. Maybe I'm just lucky, but it's fairly rare that any of my non-systemd systems is unable to shut down properly once, let alone every single time consistently.
But journald lacks way more than simple text files. I tried to "mute" a process and redirect output to a file, because it was counting in the maximum journal size, and found it to be impossible "by design". Had to use an actually good log daemon, but it seems quite hard to just disable journald and not lose any log.
Personally, I found journalctl much better than text based logs.
For the timeouts, I think try the systemd-bootchart thing.
Systemd replaces init and uses a declarative approach for the system and its services, and the dependencies between the services (replacing init-scripts). Systemd is more complicated but can do more stuff, like initialize things concurrently. Functionality that used to be implemented in services themselves (e.g. restarting, recovery) is migrating into systemd, and systemd is acquiring more and more logic. Some people feel that this is contrary to the *nix philosophy and is not architecturally sound.
Operationally, instead of using shell scripts and symlinks designed to be sorted in a particular order (normally maintained using other tools), you use descriptions of how the service should start and what it depends on.
Security-wise, systemd is a bigger hairier ball, so it probably has bugs. But it also implements stuff once, whereas before implementations were distributed and of variable quality. So the variance in security level is probably lower with systemd, but depending on your mix of services, the mean may be higher or lower. And you don't get to control it.
(As an ex Debian developer myself, I spent many many hours working on this stuff while I was one of the sysvinit maintainers.)
> I did a fairly extensive evaluation of both upstart and systemd by converting one of my packages[] to a native configuration with both systems. []I tried to approach each init system on its own terms and investigate what full, native support of that init system would look like, both from a Debian packaging perspective and from an upstream perspective. I also tried to work through the upgrade path from an existing init script with an external /etc/default configuration file and how that would be handled with both systemd and upstart.
> I started this process with the expectation that systemd and upstart would be roughly evenly matched in capabilities. My expectation was that I would uncover some minor differences and variations, and some different philosophical approaches, but no deeply compelling differentiation.
> To my surprise, that's not what happened. Rather, I concluded that systemd has a substantial technical advantage over upstart, primarily in terms of available useful features, but also in terms of fundamental design.
The essay goes on to elaborate on the details. I personally found this and other writings a compelling argument in favor of the approach. Another useful article was "Why systemd?" [2]. There's also the blog post series "Systemd for Administrators" [3].
What systemd strives for makes a lot of sense to me. It allows you to describe the startup of services with declarative configuration in a simple and easy-to-understand format. Systemd is natively integrated with OS namespaces, cgroups, and the process hierarchy. Russ gives an example in his essay of how this allows systemd to track and display more information about daemons than the alternatives. It might be more complicated than the alternatives in one sense, but that buys you the power to do things like: activate services on-demand, when requested by a client; automatically launch services per user session, and clean them up on logout; concurrently start services for a fast boot; consign managed services to an OS namespace. By supporting these features in the init and process management system, it frees individual services from redundantly building this logic in their shell scripts and daemonization routines. That reduces complexity, and makes the entire system more feature-rich, more secure, and easier to manage.
As far as security, systemd makes it substantially easier to employ kernel namespace, cgroups, capabilities, and other isolation facilities with simple configuration switches. Let's say that we'd like to run some daemon with a private /tmp directory for isolation. This is as simple as adding "PrivateTmp=yes" to its configuration. What if we want to change the run-as user, or even launch the service in an isolated user namespace? Perhaps we want the daemon to have a private network or private /dev? It's as simple as setting User=, PrivateUsers=, PrivateNetwork=, PrivateDevices=, etc. respectively:
[Unit]
Description=Demo service
[Service]
Type=forking
ExecStart=/usr/sbin/my-daemon -d
User=foo
PrivateTmp=yes
PrivateUsers=yes
PrivateNetwork=yes
[1] https://lists.debian.org/debian-ctte/2013/12/msg00234.html
[2] http://blog.jorgenschaefer.de/2014/07/why-systemd.html
[3] https://gist.github.com/bcremer/8cdf6900c35dda65f387
[4] https://www.freedesktop.org/software/systemd/man/systemd.exe...
barrkel talked about systemd as a replacement for init, but that's not the goal of its authors. Nor was there a debate over the two.
There was a debate in Debian Land over at least four choices: systemd, upstart, OpenRC, and sticking with van Smoorenburg rc with Debian's various enhancements.
The stated goal of the systemd authors pretty much from the start was not to "replace init", or even to replace both init and rc. What barrkel wrote could be said of daemontools from 1997, after all. That, too, encouraged a move of common procedures and mechanisms out of bespoke daemon programs and scripts and into a common daemon management system.
systemd, rather, was to provide a common layer, beneath everything else and above the kernel, used on all Linux operating systems. Its authors saw the differences such as /etc/sysconfig/network versus /etc/HOSTNAME versus /etc/hostname versus /etc/conf.d/hostname and wanted to unify all that, so that all Linux distributions worked the same. They didn't just write process #1 program. They wrote a name-lookup server with a local protocol to replace the DNS protocol (and the protocol that GNU libc uses to talk to its lookup helper processes), a network interface setup/teardown utility, a whole bunch of service utility programs such as a program to save/restore randomness from /dev/urandom across system restart, a centralized log writer, a centralized login session manager, and a whole bunch of programs that provided RPC interfaces, over a centralized system-wide Desktop Bus instance to GUI tools running on user/administrator GUI desktops, for things like setting the default timezone and pretty-print hostname. To that they added rules about where to find different sorts of stuff, from administrator-written unit files to /etc/machine-id; guarantees about "API filesystems"; rules about /run, /run/user, and a whole bunch of related memory filesystems; deprecation of things like /var/run/lock; rules about what sockets old syslog programs had to change to using, in place of what they had been; per-user service manager instances and a whole extra set of PAM hooks that connected it with the new runtime directories and the login session manager; and requirements such as that /usr be already available at the point that /sbin/init is invoked from the initramfs. They got some additions made to Linux in support of this, such as subreapers; and failed to get others, such as kdbus.
"systemd replaces init" is both superficial and a blinkered Debian world-view. In the world outwith Debian, in Ubuntu Land and Fedora Land, systemd replaced upstart, which had been the Fedora and Ubuntu system and service manager for a number of years before systemd was invented. The world has never been van Smoorenburg rc scripts versus systemd, not even when the whole Debian debate was had.
* http://jdebp.eu./FGA/debian-systemd-packaging-hoo-hah.html
Systemd came out a few years ago, and wars were fought over it. It fixes a lot of bugs that perennially came up with run scripts, but it's also a huge monolithic program that is in charge of nearly everything on your system.
That's about all I'll say on it, because it's almost as divisive as the Israel/Palestine conflict.
But who doesn't love a good bit of conflict, amirite?
I fully migrated my personal laptop to FreeBSD (as TrueOS) a few weeks ago, after using it as my second OS at home.
And because I need Linux for work, I migrated to Devuan on my workstation, as I need to use Ansible and Docker in a stable way as part of my DevOps job.
At the same time, it is sad to see how much work you guys need to do for things that should have been automated or done by the original developers... Let's hope AppImage changes that
Edit: Changed flatpak to appimage
https://www.debian.org/doc/debian-policy/
Sections 6 thru 12 are what it means to be "a Debian" rather than say freebsd packaged into .deb files.
For some very end user applications that don't interact in any way with anything else, games perhaps, that works pretty well, until something is run into that does need to interact with other components also operating under the same anarchy.
The problem with not having a closed system or standard or method of operating a system aka an operating system, is you end up with the deployed machines having an Apocalypse Now quote conversation "Are my methods unsound?" "I don't see any method at all, sir." If you're a system Administrator what does it mean to Administrate mere anarchy?
PS: Thank you, Riku. I never understood the sheer amount of effort involved in being a packager / maintainer until I started doing the same task for NixOS. I regularly depend on Debian's excellent patches and CVE details to do my work. May you not be bogged down by nirvana fallacies :)
I think the Solus guys would have a word with you. Even with no packaging experience it is pretty simple to create a package for Solus Project.
Here a short intro:
You can't just upload it and forget, you have to sign up and become associated with that work, you have distribute your keys and get other people to trust you, you have to make a bug and file the package against that bug, etc. There is this whole social dynamic within Debian that I just don't understand at all.
If it was more like software development; make the package, git commit, push, post it to software like reddit and have people approve (upvote) it, then I'd be a maintainer already. Instead there's this whole process behind becoming a maintainer, finding a "mentor" etc, that for years I've just found to be a complete roadblock.
Leaving aside the specifics of Debian's tooling and social process (undeniably, these are arcane and complex), I suspect this has a lot to do with why Debian is such a reliable environment over time in a way that few software collections manage.
It would almost certainly be better if it were easier than it is to participate in Debian. I've thought about contributing for years, but I certainly haven't had the energy to clear those hurdles. On the other hand, there's something really important in the distance between typical modern software publication and packaging for an ecosystem like Debian.
I think we'd be a lot better off if more people were committed to the hard, tedious process stuff that renders software accessible to users, a good neighbor to other projects, and maintainable over the long term.
That said, the Debian processes are over 20 years old. I find contributing to the FreeBSD ports and MacOS X homebrew package collections simpler, and without the same level of jumping through hoops. Homebrew's git-based submission, review and CI testing is simple to use. Likewise submitting a patch for the FreeBSD ports. Debian could do something similar, but its workflows predate this significantly. Were Debian to adopt a similar process, I think it would make the process significantly more transparent. The existing practice is still oriented around single individuals maintaining and uploading single packages (though it can also be done by groups with their own private version control for the package/group). The newer methods are significantly more open with much lower barrier to entry.
Upload and forget leads to dead packages that get removed next time there is a library transition. Debian is about long-term maintenance.
The social stuff (as well as the social contract and DFSG) is also what creates trust within and towards the Debian community and holds it together, which is the main reason it has lasted so long.
Already often a big part of Debian packaging work is making the software compile outside the developers personal enviroment without manual steps - or in the other end decoupling from the developers CI loop.
The point was that a lot of packaging would be done by the devs, allowing minutes maintainers to so more important stuff.
For interested readers, here's some best practices for being a good upstream:
- Just use the GNU autotools. Users expect `./configure && make && make install` to work. Too often people roll their own configure scripts and Makefiles and they always miss something important. Distros expect there to be certain knobs to tweak, and configure scripts and Makefiles generated by the autotools have all of them.
- Don't bundle third-party dependencies. For security (and for better documentation of the true dependency graph) distros often must go through extra trouble to unbundle third-party libraries when present. Some project even add their own custom patches to their bundled source. Resist the urge to do this.
- Include accurate copyright information. Put a license header at the top of every source file. Any serious distro will need to do at least a cursory inspection of licensing info to make sure it meets requirements.
- Make proper source release tarballs. Do not depend on your version control tool being available at build-time. Do not depend on the autotools being available build time. Use 'make distcheck' to make a fully bootstrapped tarball to distribute.
- Do not make any use of the Internet during a build. That means no downloading third-party libraries, pre-built binaries, etc. It's imperative that a build can succeed without network access, and some distributions isolate builds from the network to ensure they don't misbehave.
- Do not hardcode absolute paths to binaries. No /usr/bin/bash or etc. Your assumption will surely fail on a non-trivial number of systems. Find the location of a binary at configure time by inspecting $PATH. GNU autoconf can do that and substitute the absolute file name where it's needed, such as in a script's shebang. The same advice can be applied for anything else you need an absolute file name for.
- Do not assume /usr exists. The Filesystem Hierarchy Standard is not as popular as it used to be. This is a more generalized form of the previous point. Again, if you use the Autotools you will be doing the right thing by default.
There's surely more, but that's what I can think of right now. Surely a Debian developer or someone else has compiled a more thorough list. Anyone know of one?
I think that today's software being so difficult to build is making practices that are frowned upon by distributions (for very good reason) seem like acceptable solutions, which leads us to the growing popularity of Docker, Snappy, and Flatpak. "This software is nearly impossible to build, so just use my {Docker,Snappy,Flatpak} bundle!"
tl;dr - Make your software easy to build, don't just package up a mess.
You've pretty much hit the nail on the head. You forgot one additional bit though, please for the love of god don't have a crazy web of dependencies.
I see a lot of Node and Ruby apps online that I think would be incredibly useful in the Fedora package collection and have considered contributing them on more than once occasion. What always stops me is the 50+ dependent NPM packages or Gems they require that aren't already packaged by someone else.
The incredibly annoying part is most of these packages provide minimal functionality that you could have just implemented yourself, or that shouldn't in turn need another 3-10 transitive dependencies of their own. I get that not re-inventing the wheel is generally a good idea, but please try to pick your dependencies wisely if you want to see a distribution include your package - because a volunteer maintainer likely doesn't want to be responsible for your package + a dozen or more dependencies if they can avoid it.
I see more and more programming languages trying to bundle their own dependency management tools 10 years ago I thought CPAN was great nowadays I'm not as sold it's basically reinventing distro style package management but in a way unique to each programming language.
The real problem comes when developers start using these package managers with reckless abandon and letting their dependency tree grow out of control. I don't mind packaging an extra library or two, but a dozen or more is pushing my patience.
[1]: https://www.digitalocean.com/community/tutorials/how-to-use-...
I packaged a Github clone called Gogs (written in Go) for Debian/Ubuntu, complete with Lintian support. But I had to compromise on the 'rules' file and add a "get-orig-source" target that uses Go's package manager to grab all of the dependencies. I used that rule to grab all of the source files required to create the source package (which can then be built in isolation).
But if I understand Debian's official packaging rules, this is verboten because it winds up including a bunch of interconnected third-party libraries. Since I didn't write Gogs or any of its dependencies, I can't exactly go through and eliminate all external dependencies. And even if I could, much of Go's standard library exists only in ecosystem form.
How should a prospective package maintainer handle these kinds of ecosystems? Trying to distro-package every library (Perl-style) would be a Herculean effort, and could conceivably be met with hostility by the upstreams.
There is so much software being written in Go/Rust/Ruby/Node/etc. How can we go about packaging it?
In the meantime, we can use the information available in these language package managers to help bring that software to the systems package managers. How easy it is all depends on the language. If the language/package manager is sane and the build system isn't conflated with the package manager, we can make quick progress by writing importer scripts that automate most, but not all, of the work. Node, Go, and Java are utter nightmares for various reasons. Python is pretty good. Ruby is somewhat annoying but doable. It seems that Rust is decent but I haven't used it. All I know is that someone recently wrote a Crate package import for GNU Guix (the package manager I contribute to and recommend highly) that seems to work. [0]
[0] https://www.gnu.org/software/guix/manual/html_node/Invoking-...
Would you happen to know a good video or writeup on why language-specific package managers are a bad idea? I mean, the situation with C and C++ libraries seems significantly worse to me, and I personally really enjoy having the fully Crates.io index at my disposal on any box that runs Cargo.
The two styles of managers just have different goals. That is, a package manager such as apt has the goal of creating a single, harmonious system from stuff written in many languages. But a language-specific package manager like Cargo has the opposite goal: to provide a good way of writing software written in one language across multiple systems. This is where most of the tension comes from. The rest of it is from the same general structure, but with different specifics: the goals of these kinds of systems are very different, and conflicting.
Software is hard.
I think language-specific package managers are fine for easily sharing source code amongst developers using the same language, but they shouldn't be used in a production system.
Nix and Guix don't work on Windows, right? They're still not close to a solution until they do.
The third sentence is not a contradiction. I'm just saying that I can live with people using language-specific package managers, but really they would be better off with a general-purpose one.
https://en.wikipedia.org/wiki/List_of_software_package_manag...
I'm not saying packaging is easy, and I do try to make it friendly for distributions, but don't pretend that doesn't make it worse for general users in the process.
But please, if you decide to bundle third party libraries make sure you can build without them and use ones provided by the system instead. It's a political nightmare to include packages with bundled libraries because it makes security updates a huge headache since we can't simply rely on Anitya (https://release-monitoring.org/distro/Fedora/) to send us notifications that a new release of the library is available, not to mention the extra work of actually updating the bundled library once we do find out an update has been published.
And I'm not saying that you should never provide some prebuilt binary to your users if their distro is lagging behind. And if you really feel the need to bundle third-party libs then just make sure there are configure switches that can be flipped so that system libs are used instead. The best thing for users is for them to be able to get all of their software from their distro, and that requires distros and upstreams to each do their part.
It is in many scenarios. If I have a little app I want to package then it becomes my responsibility. For commercial software I make it always is and this is part of the reason that linux sucks for commercial software.
And then there's issues like security patches. Developers need to know what branches are used downstream.
You are talking about proprietary software, where developers have unjust power over users. If you want to distribute such software then yes, you have to do the work of making binaries for each distro you want to support by yourself. I would argue that it's not GNU/Linux that sucks here. If instead you gave your users freedom by using a free software license on the source code, then others may package the software for use on the system of their choosing.
If any of the dependencies aren't currently packaged in Debian, how would one follow this guideline?
If I wanted to package my third-party dependency, the first thing I would do is "learn about personal interests of sponsors" and see if my third-party dependency and a sponsor's interests intersect. There's a link to a page describing the sponsoring process, where apparently I'd file a bug against a "sponsorship-requests" pseudo-package and then, I guess, wait.
Next (or perhaps concurrently) I'd file a separate "Intent to package" bug against the "Work-Needing and Prospective Packages" pseudo-package. There's a whole page about WNPP and format guidelines for submitting said bug using the "reportbug" tool. Those format guidelines are longer than the JSON spec.
Then I'd still need to make the package, after all. That link you gave lists five important reference materials, one of which is said to be "must read" and has 12 chapters and 7 appendices. There's also a "New Maintainer's Guide".
Then I need to publish my package. There's an account to sign up for. Plus I'll need to create, keep up with and sign stuff with a GPG key because uploads are http/ftp only.
Once that is finished I apparently get an email response. Finally... I am done!
Now it's time to find a sponsor.
There's a whole section on what to do if you can't find a sponsor. The first is to follow up on the WNPP request I was supposed to make six paragraphs ago. The other is apparently to look up sponsors in a sponsor search-engine on the Debian website and bother them.
Then there's another section on actually getting the package into debian through an ftpmaster. (Both the sponsor and the non-Debian Debian-package maintainer are ominously reminded here that the ftpmaster's _opinion_ on inclusion is binding.)
And then maintaining it.
I would be, for the life-time of my application, maintaining the Debian package of one of my third-party dependencies. This, in response to my query about how to be a good upstream citizen in the hopes that downstream maintainers can more easily package my application! :)
In the cases (which happens more than you might think) where an upstream developer has actually got really good packaging, I've usually taken the approach of tidying up the last few details and committing back and then mentoring them for a while with them doing the majority of the work and I just double check and upload it.
The level to which people are willing to work on it varies from accepting patches that affect the distribution package version (e.g. hardcoded paths or porting issues) to actually doing the packaging.
If you're interested in helping out with Debian: https://www.debian.org/intro/help
If you're an upstream developer and you'd like your package to be in Debian: https://wiki.debian.org/UpstreamGuide
I am upstream for some of my packages and I don't provide the same debian/ directory upstream as I use for Debian.
I've heard time and time again that there are issues with Debian's version being out of date, but this doesn't have to be this way, at least not by policy. If there is a backport maintainer (doesn't even have to be the same person working on unstable) then the latest version can be installable within a stable system giving you the latest and greatest of the applications that need it with a stable base underneath.
Thats very nice. Although I wonder what Debian's 'official' container format will be.. almost everyone else seems to be standardizing around Flatpak so it'd be nice if Debian did as well, especially since that will mean Ubuntu (hopefully) will switch to it in the future as well instead of Snappy, same as the whole Upstart -> SystemD story.
I used fte as my primary editor on OS/2, and when I started using Debian I was happy to find it available there as well.
I considered become a DD about 10 years ago, but a friend was going through the process and it took them over a year with at least one restart-from-scratch because the bureaucracy had been lost or something.
> Regis NM did start somewhere in 2003. There was a period of him being on hold, in 2006 we did continue the process, which used some time, but most of the delay up to now is, again, my fault. Seems like all the few NMs left in my AM queue do have some huge level of patience available somewhere...
https://lists.debian.org/debian-newmaint/2007/08/msg00046.ht...
I wonder if Debian has improved their process or replaced the ineffective people since. Do they still have the second-class-citizen system (whose name I can't remember offhand)?
Debian muscle is behind many other Linux distributions, and Debian maintainer work is fundamental.
https://suihkulokki.blogspot.com/2017/01/20-years-of-being-d...