The Delusions of Debian
unixsheikh.com
unixsheikh.com
> Due to the sheer size of the current Debian release it is infeasible for a small team to be able to audit all the packages, so there is a system of prioritizing packages which are more security sensitive.
This was, at best, poor communication. Of course nobody would ever audit all of 90,000 packages, easily billions of LOC. Especially not when the vast majority of these packages have a very small user base.
> The fact of the matter is that Debian has long been experiencing a decline in the amount of people willing to participate in the project.
The author is claiming an assertion of the fact. From a very recent thread "Is Debian sending people away" [1], there is no decline.
> Before that, Michael Larabel, the former leader had these ridicules points in focus for the future of Debian.
I don't know who they are, but they are not a former Debian Project Leader [2].
And so on.
[1] https://lists.debian.org/debian-project/2022/03/msg00037.htm...
[2] https://en.wikipedia.org/wiki/List_of_Debian_project_leaders
>I don't know who they are, but they are not a former Debian Project Leader [2].
That one at least appears to be a brainfart by the author; they meant Sam Hartman. Michael Larabel is the author of the Phoronix article about Sam Hartman.
> This was, at best, poor communication. Of course nobody would ever audit all of 90,000 packages, easily billions of LOC. Especially not when the vast majority of these packages have a very small user base.
How is that any different? It rather sounds like you've restated the same thing and then claimed the author's wrong.
That statement is entirely reasonable, yet the author frames this as the the point where "things begin to spin out of control!".
Any time I see the domain, I know what I'm in for. But aside from "everything was fine the way it was", I've not seen the author ever present a better way forward when discussing how much they dislike systemd/Wayland/Gnome > some version, and now we can add Debian to the list.
I get it, having a rant is fun, but for your rant to be useful, especially when it's about FOSS projects, presenting a potential path forward is both an antidote to the incessant "bah humbug", and a way to maybe inspire the change you desire.
Edit-The easiest way to make some points on the internet is just simple criticism. Doesn’t even have to be informed to come across as legitimate on the internet. Which is why internet blogs like this can do huge amounts of harm.
That's a meaningless number because every distribution has different way of packaging things.
What should be counted is the number of source packages, because curl would still be counted as one package regardless of whether your distro split it into 8 (https://packages.debian.org/source/buster/curl) or 3 (https://archlinux.org/packages/core/x86_64/curl/)
Even when not doing a comparison between distributions (like when author complains about # of packages per # of LTS maintainers ) this is still the more meaningful count because package maintainers don't deal with every binary package separately, they deal with source packages.
As of today, Debian has 31306 source packages in stable and 34179 in unstable:
udd=> select count(source) from sources_uniq where release='sid';
count
-------
34179
udd=> select count(source) from sources_uniq where release='bullseye';
count
-------
31306It’s a case of bad timing. Packages for PHP 8 were uploaded shortly before the bullseye freeze, which didn’t sit well with the release managers. See bug #976811 [1]. The maintainer also mentions in that bug thread that Microsoft will provide security fixes for PHP 7.4 after the EOL date.
[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=976811
But it doesn't work like that. Packages have reverse dependencies. Just updating PHP to 8.0 would have broken all reverse dependencies that didn't already support PHP 8.0. So it isn't just a case of shipping the newer version of PHP; all upstreams of reverse dependencies need to have shipped updated versions, too, and all of those need to have had their packaging updated.
In practice distributions patch support in to laggard reverse dependencies or even remove them to speed things up. But still, the actual work involved is much more than the naivety in the statement "PHP 8.0 was 10 months old yet Debian's 11 was released with PHP 7.4."
A second common misconception is that just because an upstream declares some kind of support period, distributions have to follow them or it's somehow a problem if they do not. Upstreams even having a declared support period is the exception, not the norm. Distributions have been declaring a support period for their release as a whole long before some upstreams started doing this.
I tried to get into Debian package ownership while a student, but came to realize I would need a substantial financial incentive to interact with Debian’s machinery. I spent a while running FreeBSD after that.
In other words, I'm questioning whether a revamp is needed at all (as opposed to incremental improvements which are happening all the time). This article might be making some claims in that direction, but has had most of its points debunked in this HN thread already.
i am not involved in the debian project, but there seem to be quite a lot of people who know "what's best". Funnily most of them are not involved in building the stuff. and he shits on reddit comments but also does quite the same. i guess it's because they did not share his opinion.
https://tracker.debian.org/pkg/samba
current stable at 4.13.13, accepted 2022-02-13
current testing/unstable at 4.13.14, accepted 2021-11-12
https://www.samba.org/samba/history/
current stable 4.16.0 released 21 March 2022
older stable 4.13.17 security release dated 31 January 2022
I don't want to point fingers at anyone but look at those version numbers and dates.
I'm pretty much an arch(and sometimes alpine) guy now, except some proxmox servers, which unfortunately is debian based.
For a daily driver always buy a ThinkPad, a year old model when new stable comes out, and use it as long as possible.
For servers, run the latest stable with minimal packages from main and occasionally few from backports.
Build from source/ download from upstream where Debian packages are available (specially browsers).
Reliable kernel, coreutils, vim and gcc from Debian stable, git + ooenssh from backports, tmux built from source, golang and "go get xyz" is mostly what I need. (I understand this can be too minimalist for many, but works for me).
I have a hobby website using fossil, running on 8year old SBC running bullseye.
At work, almost all bullseye docker images on minimal Bullseye running Dockerd. PostgreSQL, Redis, and few more central piece of software work fine with Debian.
Low maintenance, reliable systems that I seek, and Debian has never failed me.
Ahh and yes, when I need bleeding age, Arch is what I go to, but won't be running a daily driver machine or server on it. It's too fast for me to keep pace with.
I can safely say I have more peace of mind with free Debian than any paid OS I used (taken together with application ecosystem). All systems have their own pros and cons. I just picked what worked best for me, I am thankful for that and wish Debian project never runs out of great people.
On a tangent, i've been using Void for a couple of years now and i'm very happy with it. All the packages i want are up to date ( and i do not require LTS or backported security fixes ).
Things like runit and xbps-src are relatively simple. I can htop and know why every process was started.
Maybe I'd just reached the right level of skill to 'get' it once i switched, but i never had the feeling i was in control with other systems such as systemd and apt.
It's the same deal with Debian and systemd. It's not the init system. It's the thing it represents. Having to either adopt systemd or run GNOME in an unsupported configuration seems like a clear-cut choice (which is why systemd is now the Debian default), but having a major upstream force you to significantly rearchitect the distribution seems like a pretty significant loss of control. This is at the same time as other upstreams have gradually ripped control out of the hands of Debian developers in other ways: Firefox, librsvg, and python-cryptography adopting Rust suddenly made it a lot harder to support a bunch of niche CPU architectures. Speaking of Rust, and also Go, they use static linking and have their own library package managers, which both makes it harder for Debian to package and simultaneously easier for users to install a binary directly provided by upstream. And languages that don't have static linking support can always use Docker.
Honesty, I wouldn't want to be a Debian developer right now. Red Hat has tried to reinvent themselves in a world of containers, but since Debian is a volunteer organization and not a corporation, it's a lot harder to pivot (attempting to pivot would probably cause all their existing volunteers to quit, while failing to attract new ones). What sort of future does Debian even have, if they have no decision-making power over the core OS, and all the applications route around them?
1. "GNOME in unsupported configuration" is, with due respect, FUD.
2. There was no dichotomous choice to make. Debian could have let you decide whether or not to install systemd. Instead, the project decided to essentially force the use of systemd, necessitating a fork.
3. "suddenly made it a lot harder to support a bunch of niche CPU architectures" <- why is it difficult to offer a different selection of potential default browsers, on those niche platforms or in general?
4. Language-related package manager do make things difficult for an OS distribution, at least somewhat - but that's not particular to Debian.
However - I actually agree with your main point. I'm sure Debian maintainers/developers have it hard.
I really wish Debian would:
1. Admit the Devuan people were right and re-merge the distributions.
2. Recruit, offering a self-training track for aspiring package maintainers.
3. Modernize some of their tooling (as people seem to be complaining about that)
4. Fundraise effectively, to finance the above.
If GNOME doesn't work with Debian's systemd-shim, are they going to accept patches to fix it? Or will it be up to Debian to maintain those patches?
> 2. There was no dichotomous choice to make.
They didn't make a dichotomous choice. They made systemd the default, but maintained the ability to switch to another init system [1].
[1]: https://packages.debian.org/sid/init-system-helpers
> Debian could have let you decide whether or not to install systemd. Instead, the project decided to essentially force the use of systemd, necessitating a fork.
If this is "essentially forcing" the use of systemd, then what possible choice would have counted as not forcing it other than making sysv the default?
> 4. Language-related package manager do make things difficult for an OS distribution, at least somewhat - but that's not particular to Debian.
You're not wrong, but most Linux distributions either don't have "LTS" releases [2], or they have commercial backing.
[2]: Not having to backport security fixes saves them a lot of hassle.
> 1. Admit the Devuan people were right and re-merge the distributions.
Right about what? Devuan/Debian is hardly a comparable situation to LibreOffice/OpenOffice. It's more like the relationship between Librewolf and Firefox, where a bunch of loud fans are praising the fork to the sky, but most of the developers are still working on the original project and the "fork" is busy rebasing their patches onto each new upstream release.
> 2. Recruit, offering a self-training track for aspiring package maintainers.
That would be great, but are there volunteers lined out the door who want to join the program, and just can't? Or is it boring, frustrating work that few people are interested in?
> 3. Modernize some of their tooling (as people seem to be complaining about that)
I actually agree with this point, but I'm not sure if this is the primary problem, or if it's just a minor issue that distracts from the main problem.
> 4. Fundraise effectively, to finance the above.
There is no Debian Foundation. They would have to have somewhere to send the funds to, before they'd be able to collect it. Obviously, this is almost as contentious as systemd was [3].
No, they didn't. You can't install debian without systemd. They _said_ they let you switch, but they don't. People did not fork an entire distro just because they didn't like to press "option B" instead of "option A".
> If this is "essentially forcing" the use of systemd, then what possible choice would have counted as not forcing it other than making sysv the default?
1. Not having packages depend on systemd.
2. Bringing up a prompt/dialog during installation to make a choice of whether or not to use it.
> most of the developers are still working on the original project
Because Devuan is just Debian with some tiny changes and a different choice of packages. And of course, the infrastructure of a project - website, forum, IRC, download servers, etc. So of course most developers aren't concerned with that, they just provide/maintain upstream packages.
> Obviously, this is almost as contentious as systemd was
I didn't know about this aspect, thank you.
This implies either (a) refusing packages that depend on systemd, or (b) patching packages. Which brings us right back to the original problem: who's on the hook for making sure it works?
> 2. Bringing up a prompt/dialog during installation to make a choice of whether or not to use it.
https://www.joelonsoftware.com/2000/04/12/choices/
... okay, I admit I also need to offer a reason why it's okay for Debian to offer the install options that they do offer, while they also refuse to offer this option, and can't really come up with one. The way that Debian allows you to pick a desktop environment at install time, while not allowing you to pick an init system, is a bit of an arbitrary decision.
But the decision being arbitrary isn't a reason to offer every single theoretically possible option at install time. Debian never allowed you to pick an init system at install time before! Why start now?
Well, Devuan people made it work, while also maintaining an entire distro fork. This could easily have been made to work on Debian. The alternative would have been to lean on GNOME, I think.
> every single theoretically possible option at install time
systemd-or-not is not "every single theoretically possible option". It is a highly contentious and political issue. If the systemd proponents would have accepted defaulting to no-systemd, then great, no need for the choice; but realistically, both sides are somewhat adamant, so making the user choose is a reasonable compromise IMHO.
Of course not. It's an arbitrary line, because the line has to be drawn somewhere.
> It is a highly contentious and political issue.
Most people don't care one way or the other.
> If the systemd proponents would have accepted defaulting to no-systemd, then great, no need for the choice; but realistically, both sides are somewhat adamant, so making the user choose is a reasonable compromise IMHO.
How is leaking internally-facing political issues into the out-of-the-box experience in any way acceptable?!
It's an almost perfect reflection of exactly the reason Joel On Software hates prompting people up-front with configuration options; by refusing to pick a reasonable default, the small number of people who actually care one way or the other are forcing everyone else to deal with their baggage. First-time Debian users completely lack the context to make such a choice, and anyone who does care knows where to look for information on how to switch over a system after it's installed.
It's not a "line". Some choices are presented to the user, and some aren't.
> Most people don't care one way or the other.
If that were the case, then great - no-systemd by default.
No, the point is that the _people involved with the distribution_ care.
> How is leaking internally-facing political issues into the out-of-the-box experience in any way acceptable?!
Either people come to an agreement about a default; or they have a fight and split up; or they let the user decide.
... actually, there could be another option, which is a "mixed strategy", i.e. the installer flips a coin instead of asking the user. That could also be a compromise, which doesn't require bother the user. It has its own detriments though.
Years of experience have shown that programmers can't write secure code from the start. Maybe some day we'll have languages & tools that allow for network-connected programs that never need security updates, but today is not that day.
Attackers often exploit vulnerabilities reasonably quickly, so updates have to happen soon after vulnerabilities are discovered.
So we need timely updates, and we keep needing updates. Unless we define "updating software" as not contributing to maintenance, I'd say my point stands.
The maintenance is made even harder for distributions like Debian that want to backport security fixes without backporting feature changes or other refactorings. That produces a lot of extra work for the maintainers, and those maintainers aren't usually as familiar with the code as the authors further increasing the maintenance burden.
Most of the big names who have left seem like it's a win all around, they seem to be anti-systemd hacker/tinkerer types who will be much happier over at Arch, and never really agreed with what Debian was trying to do.
But at a certain point, you just need more people for that many packages.
Perhaps they should try to transition away from the volunteer model. I wouldn't want to work at a distro if nobody was paying me, that seems like some of the very top unpleasant parts of tech work all it one job. And from what I hear, the Debian people are basically working a second full time job and some are not happy about the thankless drugery.
Either that, or the world should transition to Fedora. I'm sure they could get to Ubuntu's level of everything just works, if they had the whole Debian team helping!
When you're anti-systemd, you won't be happier over at Arch, considering systemd is the only supported init system in Arch - in contrast to Debian, which supports alternatives.
Seems like a lot more people can kind of grudgingly tolerate systemd, but they really seem to hate outdated packages, and anything "unnecessary"..
Some of them are pretty neutral on systemd itself, or have given up on fighting it.
They seem to mostly just want maximum ability to build their own system
Debian has hurd and freebsd ports for which systemd is not available.
So, instead of volunteering for Debian, volunteer for IBM?
But that might be selection bias, maybe there's lots of happy distro maintainers that we just don't hear from.
Does anyone know how this compares to other distros? In particular NixOS/nixpkgs is of interest.
$ nix-env -qa | wc -l
39168
For example they have a lot of "-dbg", "-devel" and "-doc" packages that other distros might just choose to leave in the main package instead.
Edit: it seems to be the largest software repo according to this (2020) [1].
[0] https://github.com/NixOS/nixpkgs [1] https://discourse.nixos.org/t/nixpkgs-has-been-the-largest-r...
By that table, debian 12 has 32k packages, and nixpkgs stable has 67k
That said, repology doesn't track everything nixpkgs has nor everything debian has.
It's also difficult to compare debian and nixpkgs.
Debian has several packages for one thing where nixpkgs has only one, i.e. debian has "sqlite3, sqlite3-doc, libsqlite3-dev", while nixpkgs has the single "sqlite" package with the attributes "sqlite.bin, sqlite.dev" (and no docs split out I guess? They're probably in .out)
Because debian splits out dev packages from binary packages, that artificially inflates the number.
Another choice which makes the number very hard to compare is packaging style choices for certain dependencies.
For example, in debian, a go package's dependencies must all be packaged as their own packages (https://go-team.pages.debian.net/packaging.html#_dependencie...).
This leads to stuff like "golang-github-hashicorp-terraform-svchost-dev", a package that is only used as a library to build terraform. I'd argue it's not a user-facing package, just an implementation detail of the terraform package, but the count you mention above certainly includes it.
nixpkgs, on the other hand, is fine with not splitting out each dependency as a full package on its own right, so go and rust libraries are much more sparse in nixpkgs.
All of this is a lot of words to say "comparing numbers is hard, repology is the best we have right now"
Discussion on LWN: https://lwn.net/Articles/889026/
> The fact of the matter is that Debian has long been experiencing a decline in the amount of people willing to participate in the project.
> Oh, I almost forgot. This is a list of the people assigned to the Debian LTS support team. 32 people who somehow must provide long term support for 90.000+ packages.
To me, these two feel like an unfortunate reality in the *nix world - there are more packages than anyone reasonably needs in the first place and hoping to get someone to maintain each of those packages is unrealistic. Many people might have written something that they needed to solve their particular problem at a certain point in time, used it for a while and eventually moved on, that is inevitable.
In my understanding, it's a bit like walking through a graveyard or at the very least some old library, where most of the books are largely irrelevant and won't be read by anyone in the following decade, short of very particular cases. For example, have a look at the Debian popularity context: https://popcon.debian.org/
Let's look at the install results for the main repository, open it up and scroll down a little: https://popcon.debian.org/main/by_inst
You'll see that only around 4'600 of the top packages have over 10'000 installs (by people who participate in the contest). Around 13'000 packages in total have over 1'000 installs. About 30'000 of the packages have over 100 installs.
That means that the majority of those packages aren't actually used by many people in the first place, thus don't present a juicy attack vector. That's not to say that they won't be exploited should any vulnerabilities remain, but surely the impact will be far lower and thus it's pretty reasonable to focus on the things that actually are popular.
Though one can and should definitely call this out and maybe consider what could even be done about this, since some people do need these packages for their niche use cases and perhaps are concerned with things working rather than being secure.
> Yet Void is also suffering from a lack of maintainers, just like Debian, and as a result, many third party packages in Void Linux is hopelessly outdated.
> Despite the fact that Red Hat is an enterprise Linux distribution, the problems goes even further there where you e.g. still can find a so-called LTS version of PHP 5 that long since should have been permanently terminated.
In my eyes, that just demonstrates the above - the people who need to use PHP 5 to support some legacy solution are probably aware that they should be using later versions, but it's just not in the cards for them. I once helped someone write new software in PHP 5 while PHP 7 was out at the time because even though i advised them against taking that approach, they didn't have other options, or at least viable ones. So, i wrote the software that they needed, left ample warnings and left to work on other things.
In practice, sadly LTS isn't always as much about remaining secure (even though it should be) as it is about keeping things vaguely working in slowly moving environments, which was one of the main reasons why people picked something like CentOS back in the day.
> On FreeBSD, the package manager informs you if you're installing a package that has been abandoned. It also informs you about important security issues. On the FreeBSD website the procedure is described in detail.
This is an excellent idea, as would be automated reminders about CVEs. Why should we only use external tools for scanning our servers, when they could do that themselves, at a package manager level?
> Keep a list of all the software you install.
Better yet: install less software. Nowadays my personal servers have a pretty minimal setup (sometimes with Ansible) with just the packages that i need to get containers up and running. Sure, some are against the idea of them at least in their current implementation, but they help me have a very clear distinction between what's a part of the system/infrastructure and what's the business software that i want to run - hence i should never install MySQL/MariaDB/PostgreSQL/PHP/Java/Ruby/Python/... on the system directly (okay, maybe Python for some scripts/CLI tools), but instead manage the attack surfaces with containers, which lets the old insecure stuff keep running, while updating the underlying system itself without worrying about breaking the software.
Of course, it's not perfect and virtualization still is useful for added isolation/security (multiple separate VMs/VPSes for different sets of containers) until rootless container runtimes will truly get there, but in my eyes it's streets ahead of pretending that we can somehow have our cake and eat it too - have software that is both up to date and works with limited resources.
I don't know about the circumstances that others are in, but in my homelab and even some professional projects i lean towards acknowledging out of date packages as an inevitable eventuality and thus thinking about how to limit the fallout until updates would eventually (hopefully) be done.
Back to the topic of Debian in general: it has always felt like one of the larger and more dependable distros out there, alongside Ubuntu and CentOS. With CentOS out of the picture and no popular replacements (both Rocky Linux and Alma Linux might take a few years until they're available in every regional VPS host), that choice now falls between Debian and Ubuntu: the former has a shorter life cycle (the LTS variety isn't entirely official) whereas the latter is pretty okay but has some weirdness going on (e.g. snaps and other curious decisions).
What is everyone else even using?
Debian has the package apt-listbugs
How badass is Lennart Poettering? Dude just says alright I'm going to improve upon the craptacular init system. He follows through and the whole world starts using his software. That dude is a legend and he's only 41.
Devuan forked in 2015 and is basically a non-existent distro after 7 years. Systemd was never controversial as portrayed in this blog.
So what exactly is the delusion of Debian?
>But that's not all. The fact of the matter is that Debian has long been experiencing a decline in the amount of people willing to participate in the project.
There was another recent debian post on HN. No idea why Debian suddenly is so popular of a subject. People responded to me like as if Debian was doing fantastic.I just don't know how anyone can think Debian isn't in death spiral.
>Carter believes they need two to three times their current volunteer levels to accomplish all of their goals. He also feels Debian is too lacking in the area of diversity and not catching up fast enough. He blames Debian's diversity on large regions of the world not being represented enough, failing to put together a timely message regarding Black Lives Matter, and other concerns/ideas.
https://www.phoronix.net/image.php?id=2020&image=debian_prob...
Debian leadership thinks they don't have a strong position on black lives matter? That's why people are leaving the project?
So what have they done? They have built a diversity and inclusion plan.
They even enforce a anti-harassment team. https://wiki.debian.org/Teams/Community?action=show&redirect...
I encourage everyone to read that antiharassment page in depth.
As Debian leadership clearly indicated. They need significantly more people to contribute to the project, but they have an exodus/decline of developers. Clearly they are not doing enough with their antiharassment team.
People are harassing others at debian to such a huge degree the remaining volunteers are taking on ~3x the workload they should. Eventually they will quit as well.
I think what debian needs to urgently do... is figure out exactly who the harassers are. I very clearly can see exactly who is doing the harassment. If Debian makes a mistake and does not directly address this correctly. Debian is dead. Which is so sad to me.
The one that has the biggest impact on my life is Alpine, because I run postmarketOS on my phone which is an Alpine derivative. It has OpenRC instead, which doesn't have the ability to auto-restart crashed services (it doesn't do any action when a service crashes, by design), doesn't do socket activation, and doesn't support running as a user instance for a user session.
Ok, that's a fair. My python projects refused to work on alpine so Ihavent tried it.