I guess next time people ask why I like a distro like Arch that doesn't insert its own junk into upstream code over Debian/Ubuntu/et al, I'll have a new example handy to cite.
It's been a very long time since I worked on a distro (former Canonical employee here) but every distro carries patches of some sort.
Current linux release has one patch changing one default: https://github.com/archlinux/linux/commits/v5.9.8-arch1
https://wiki.archlinux.org/index.php/DeveloperWiki:Patching
As you say, the page notes that "[the] policy is intended to suggest, not to enforce", but having a policy is orthogonal to enforcing it.
But why is kernel is patched by distros at all? I run kernel from kernel.org always and don't see any issues.
I haven't been a distro packager in many years, but my recollection is that in other distros (debian, fedora, arch, etc) patches that add new functionality would generally not be considered okay unless accepted by upstream. I'd be interested to learn the rationale for not upstreaming this patch before including it.
https://sources.debian.org/src/accountsservice/0.6.55-3/debi...
https://sources.debian.org/src/accountsservice/0.6.45-2/debi...
The changes that fix both vulnerabilities can be viewed here: https://git.launchpad.net/ubuntu/+source/accountsservice/com...
Personally I question the logic of having accounts-daemon grovel through pam_env's config files in the first place. This isn't an API, it's an implementation detail! It just seems like a bad idea, even ignoring the vulnerability which anyone who isn't an absolute expert on the semantics of real/effective/saved UIDs & how to safely change them could have made.
https://www.usenix.org/conference/11th-usenix-security-sympo... (the famous Sendmail vulnerability therein is actually the _opposite_ of the accountsservice problem... sendmail _should_ have set the real uid as well as the effective uid, so that its effective uid could never be changed back to 0).
LWN has an excellent series of articles about thedesign of UNIX, one of which explains the problems with setuid: https://lwn.net/Articles/416494/
Do you think equally-bad bugs haven’t made it into upstream projects directly? Hell, many downstream patches exist to fix security bugs.
It was a predictable result of Debian policy, which Debian did not see fit to change. Debian still patches upstream sources including security-critical software, still does not have any dedicated security review of those patches, still leaves it up to individual maintainers to decide whether and how to clear these things with upstream, and still thinks all of this is fine.
> Do you think equally-bad bugs haven’t made it into upstream projects directly?
Honestly, I can't think of a single equally-bad bug in "normal" code, only in medical devices / industrial controllers / etc.. Cloudbleed wasn't this bad. Bumblebee deleting /usr wasn't this bad. It really was a uniquely awful bug.
It's not fine, but when the upstream authors do not regard or consider requirements of downstream projects, what can you do? like with the example of phonehome features?
arch and other distros are only possible, because for a long time, debian and other distros kept nagging the upstream authors for missing features or "nonfeatures". if debian wouldn't exist, i bet, arch had to patch much more itself.
Well, as one of those upstream authors whose code was patched: I was never contacted about it, so I never knew there was a requirement to be met, and so they carried around a bad patch for years, about which I knew nothing. Once a user pointed this out to me, the next release fixed the underlying issue in a better way. After years in which the Debian folks didn't file an issue or report the problem in any way that I could tell.
Mind you, Debian is not alone in this, it happens with other distros, too.
And to be fair, I think this rather depends a lot on the downstream package maintainer; I've witnessed this with other projects where they were quite good in interacting with upstream to get something sorted out. I am not really sure if any policy Debian/Fedora/... could enact would really help with it; people can (accidentally or intentionally) ignore them.
In my experience it's rather uncommon for a DD not to contact upstream. Would you mind sharing the package name and vulnerability so I and others can learn what happened?
Upstream was contacted about it: https://marc.info/?t=114651088900003&r=1&w=2
the maintainer made an error there. did you open a bugreport that you fixed it so the patch is not necessary?
> And to be fair, I think this rather depends a lot on the downstream package maintainer; I've witnessed this with other projects where they were quite good in interacting with upstream to get something sorted out. I am not really sure if any policy Debian/Fedora/... could enact would really help with it; people can (accidentally or intentionally) ignore them.
yeah and that's the point. people in this thread (not you as far as i see) say they do not trust debian because of this, but other distributors and packagers? do they have technical or organisatorial fences for avoiding such mishaps? if not, then other distributions are as problematic as debian, even arch.
debian did a whole lotta good for Free software and i really start to dislike how people shit on the project (again not you).
Many distributions have a dedicated security team that has to sign off any patches to security-critical software. Debian's position is that they do not have the resources for such a team, which is fair enough, but IMO the conclusion should be that they don't have the resources to be applying their own patches to security-critical software.
More subjectively I get the sense that Debian packagers patch more aggressively and generally think the Debian way of things is better. This isn't completely groundless: there's a lot of very high quality engineering in Debian, and for a long time their package management was head and shoulders above others, especially if we're talking about C programs/libraries where upstream dependency management is very weak. But it's also made for a culture where packagers think they know better than upstream maintainers, and an approach that ends up conflicting quite a bit with newer languages where there is high-quality dependency management in the upstream builds.
Which are some of those distros? (I'd consider using myself in the future)
This is false. Debian has a security team and it's way more active than most distributions.
> But it's also made for a culture where packagers think they know better than upstream maintainers
Yes and for good reasons.
On the flip side of the coin, there are plenty of open source authors who release their work into the wild and refuse to support or even engage with downstream packagers because they see any other distribution or use of their work as Not Their Problem. And they're not really wrong, building a supportive community around a useful project is totally optional after all.
Between the constant stream of version bumps, security updates, patching, bug triaging, and user support, distribution maintenance seems to be the most thankless job in the open source world.
It was interesting because the cause was a patch crafted to satisfy a static analysis tool they insisted on applying to every package, which demonstrated their want to go above and beyond with respect to quality. Kudos to Debian in that regard. But a blindly applied policy, and bad judgement in disabling some cryptographic initialization code, caused a terrible bug.
I wonder if this is less of an issue today because of well defined kernel interfaces for getting "good" random numbers (i.e. cryptographically suitable). I'm sure the devil is in the details, and OpenSSH's support for unpopular systems means intentional use of uninitialized variables is still in the code base. It's all very impressive to an outsider who knows enough not to tell cryptographers how to do their job.
More than a decade ago a programmer friend got Ubuntu running on a second hand desktop for a rather computer illiterate arts & letters student. He was able to navigate the GUI like any other system, and it had Firefox, OpenOffice, and mplayer. I knew then that Linux is a perfectly fine desktop OS. And now we're back to using Debian of a sort. ;-)
I stopped using Debian when its ridiculous "free" purely ideological approach to software actually caused me issues.
I needed to install Debian on a relatively old laptop a few years ago. It had all "mainstream" hardware. It's WiFi adapter was an Intel one (and a very common/popular one at the time), but it was one that wasn't open source.
How did Debian approach this? It's installer gave me a not very subtle passive aggressive message telling me that although they have the drivers for the device, it was not going to install them... because the ISO did not include them. With no working WiFi, it of course could not connect to the internet to download them! Worse still, turns out that even if it had connected to the internet it wouldn't have downloaded and installed them anyway. I found this out because I managed to use an Ethernet connection.
Completely stupid and frustrating and it felt like I was being blamed as if it was my fault.
Installed Ubuntu, included the drivers, connected to WiFi during the installation.
I have not used Debian on anything since.
I like it that that try to focus on a completely free version, as that is what they stand for and why a lot of people respect them, while at the same time being realistic about the real world problems that people face.
And Debian is great. We might take it for granted now, but 2 decades ago while everyone else was fiddling with their own package dependencies with RPM (pre-Yum) Debian had amazing package management with builds for all sorts of amazing architectures. They were leaders both in thought and execution.
I tried (but failed) to influence an employer to invest more in Debian; instead they went with SuSe which eventually caused a few small problems. Now Debian & Ubuntu are booming.
They do provide "nonfree" install media which include firmware blobs that many people need to get their wifi up - https://cdimage.debian.org/cdimage/unofficial/non-free/cd-in.... But it's not easy to get there (I google "Debian nonfree") and it has "unofficial" in the URL which I imagine doesn't help things either.
I understand that Ubuntu is meant to be the user-friendly Debian-based distro and that Debian is meant to be Free Software first and foremost. I just wish that this nonfree install image was a bit more official and easier for users to discover.
edit: oops, I should refresh before hitting "reply" I didn't see the other answer
So the "firmware" I required for my Thinkpad's wifi to work (and which was included in that Debian nonfree image) is this: https://wiki.debian.org/iwlwifi
update: actually on that Debian wiki there's a page called "Firmware" which explains things well https://wiki.debian.org/Firmware
It validates their hobby-like approach to their OS.
Most run Ubuntu or Debian, and a few run Arch. And every few weeks someone mentions again how something broke, and they need to spend some time fixing it. I think the last one was VMWare not working on a kernel yet.
Arch requires more time to maintain, and more manual maintenance. So in order to justify that extra spent time a lot of times they will bring up minor stuff that happens in Debian/Ubuntu, like this bug from 12 years ago, in order to justify running Arch, which is more of a "learn Linux" hobby.
That being said, running Arch on a computer that you need to be working at all time, might not be a good idea for professional reasons. However, I think you are essensially making a similiar argument for Arch, as they are making for the Debian bug.
I learned my lesson. From now on it is boring LTS distros only, and guix for things in userland where I want up to date things.
This is the feeling I get, and it's why I never bothered with Arch, or Gentoo before that. I started on Slackware, and while it was a great learning experience, I feel that's not a lesson I need to repeat.
Same reason for my move from RedHat (after an RPM dependency breakage) to Debian: Debian just works. It GTFO of my way and let's me focus on my code and my projects. Yes, there have been issues, no, Debian isn't "perfect", but like my preferred MUA "it sucks less than all the alternatives."
I've used Arch for the last six+ months and I don't want this to happen.
I switched as I couldn't get VFIO to work with Debian (presumably due to outdated kernel/qemu/libs). In my time with Arch I have had no problems. It has just worked and stayed out of my way. My experience with Debian is that it mostly works, but often uses much older software than one wants. Arch, in my limited experience, is an excellent distribution.
This is only personal observation, not direct remark to your case. I've just seen similar scenarios too many times.
On the other hand - I've played with Arch since early beginnings, but last time more than 10 years ago. Although minimal, simple and straightforward, I have never been able to grope that rolling-release core philosophy.
TL;DR: To silence a valgrind warning, Debian maintainers added a local patch to OpenSSL that commented out the entropy pool, making all SSL keys predictable.
[1]: https://www.schneier.com/blog/archives/2008/05/random_number...
Also, don't just blindly "fix things" because a tool complains. Understanding of warnings and error messages are important.
> Using uninitialized data as entropy source is an AMAZING idea in first place. “Fixing” or “unfixing” doesn’t make it any better.
I've always thought that Fedora and Arch were both much more "upstream first" than Debian is.
Yes, they do. And then they complain to you that this broke something: they split our package, from the main package A they split off parts into separate packages B, C, ... that depend on A. And then complained to us that B and C had a circular dependency and that this means we had to fix something. The audacity.
Our users still sometimes report issues where I immediately ask "are you using Debian or Ubuntu?" and the answer is usually yes: because guess what, they only install A (which matches our software's name) and then they miss the functionality which Debian moved into B, C, ... We actually print a warning about this when starting the software (along the lines of "Warning, B is missing, some things may not work as expected"), which then Debian patched out, calling it "misleading".
I've never had similar issues with any other downstream Linux distro.
So, no, I really can't agree with GP's statement.
all people are always about "slim systems" and alpine. but when debian does real engineering, then everybody complains it's complicated. that is a little bit annoying.
yes, debian is complex, but it's for a reason, since there are many requirements.
What I mean is taking one piece of software and splitting it apart into many libraries or binaries in a way that the upstream developer did not originally intend.
See some of the other responses.
IMHO, this is the correct thing to do.
I've spent a considerable amount of time as both developer and systems adminstrator. Not quite DevOps, but I've been doing both since before that was a term.
Programmers don't often have that systems level view that is necessary for systems administration. It leads to myopia like this Ubuntu patch, and packaging is usually an afterthought to many programmers. It's that difference between a "project" and a "product."
So honestly, complaining that Debian splits things would be like complaining they don't deliver tarballs, because back before GitHub, that was how upstream "delivered" "packages" (how many people remember "tar -xzf package.tar.gz && cd package && ./configure && make"?). And before you bring up "missing files", you can do an "apt-file" to search for things and then install the associated package, assuming that it wasn't installed already as a Recommended package or in many, many cases, you are using a metapackage which contains the "full" package anyway.
In any case, good or bad, it's invasive.
Regarding exim... exim is one of a nightmare no matter how one looks at it. I hate it with a passion, no matter the distro.
Gentoo has a much better track record of keeping packages vanilla, in part because patches are kept by default in the single Portage monorepo, so there is incentive not to dump huge patchsets in there. Only a few packages have significant Gentooisms maintained out of there (like the gentoo-sources kernel patchset, which as far as kernel patchsets go is still very light, and you can opt not to use it).
In 15+ years of using Gentoo I only recall two times their patches broke something for me. One, some obsolete crud in gentoo-sources broke booting without an initramfs on HP CCISS RAID controllers (because that's the only(!) block device driver that puts its kernel device names in a subdirectory). Two, a KDE Plasma patch to pull the right system Python version from the Gentoo infrastructure that manages that forked off a subprocess, but got fork() backwards, causing the "main" process PID to change and subtly breaking D-Bus policykit stuff due to security checks on the PID (aka: you can't mount devices from the system tray when the PulseAudio volume widget was loaded, how's that for a fun one to debug).
I don't recall Gentoo ever introducing a security issue, other than perhaps typical packaging mistakes (nothing comes to mind but I assume bad permissions have happened at some point, there's always some of those).
Not a bad track record for being my main workstation OS for a decade and a half.
sed -i '/dbus_conf_dir/s/sysconfdir/datadir/g' meson.build
then that would be not needed.
Does Debian actually change upstream significantly? I thought they kept things fairly vanilla.
I know Fedora patched in VAAPI support on Firefox before it was merged into the Mozilla codebase. I can’t think of a worse piece of software to patch, the last thing I want to be running untested code on is my web browser.
Thats not a trivial matter, its a potential source of many issues
There are really no perfect option here. Upstream and distro developers have different goals, and it's up to you to decide what you should adopt.
Forced updates every 6 months is annoying to most people. Forced updates every day or so would be annoying to basically everyone.
In particularly it was a significant improvement over the annoyances of Ubuntu, and I am still angry when thinking about the brief period I was maintaining a bunch of servers running Debian. Really unnecessarily difficult - particularly updating/upgrading and trying to clear the mess their systemd approach caused, in particular missing unit files for popular services (I think it was the isc-dhcp-server).
This is of course even worse if you are not the end-user but are trying to ship an appliance to your customers. Why would you want to update the entire appliance OS in the field fo all of your customers just to ship a security patch, instead of sticking with the same base OS on already released appliances?
Upgrade schedule for Fedora is twice a year, yes. Goes by without any issues, stuff still works.
There is change and then there is change. Yes, I understand that some things don't need to change, and why do they always have to. Do I need my Desktop to still resemble 3.11 because "Why did it ever change"? - No.
Some things in life simply change and they always have. I try to keep up where change is necessary.
And if I would be shipping appliances to customers, why should I have "forced updates" of any kind outside of my control? I would never sell anything without my own repos, quality control, etc.
Do you seriously sell Debian devices to anyone and support them, if anything outside of your control breaks? I seriously don't understand the argument.
And if I ship software of any kind that relies on some libraries, I have to maintain it, of course. This includes going with the major releases.
If this is embedded hardware, where there simply are only two or three updates in its lifetime, everything is different of course, but how on earth would I then have forced updates from RedHat??
Why are you afraid of updates then?
You aren't forced to update on Fedora. Fedora always supports current release n and n-1. There's even a window when n-2 is still supported after a new n is released.
Even then you aren't forced to update. I work with a guy still running Fedora 29. I wouldn't recommend that personally but his machine still works fine and is rock solid stable. Probably not the most secure though :-P
If I wanted an endless stream of new vulnerabilities I can just use upstream software.
I’m not saying those guys don’t know what they’re doing, obviously they are experts. But I would prefer to trust but verify (not just trust).
IME, they do keep things fairly vanilla, and I speak as someone who has run Debian since . . . I wanna say 2002? Plus I've done kernel dev work, and Debian's kernel patches ain't nothing compared to what RedHat used to do.
The various RPM flavors keep their kernels vanilla enough that I can mostly just rebase on top of linux-stable, and it's been that way since I started 7 years ago (according to github).
Arch is the winner. I've never had to make a custom patch for Arch.
Disclaimer: I work for Red Hat but opinions are my own
No, they radically change things to match their standards. E.g. look at their tomcat package which explodes it all over the filesystem.
Without the latter, all you have is a burlap sack full of software. Hardly a distribution. Fine if that's what you want, but it's not what Debian is.
> I also think Debian's approach has shown itself to produce something less reliable on the whole
I think the opposite. Is there something particular you'd like to point to? I accept the SSH disaster. But that's 12 years ago now, and it is a single example.
Debian's patches to cdrecord introduced so many bugs that they eventually ended up driving the original author away from open source.
I think I remember something about the python 2 -> 3 transition being harder on Debian because of changes they've made?
The packaged Tomcat on Debian is so different that every time I've seen someone running Tomcat they've installed it manually instead.
Debian has essentially given up on packaging Hadoop. Some of their criticisms of its build process are certainly valid, but others seem to be an unreasonable expectation that everything will build exactly the way Debian expects. E.g. if I'm understanding correctly, they essentially take the position that they won't package anything built with Maven, on frankly spurious grounds.
Debian tends to end up with very outdated versions of anything from an ecosystem that uses large numbers of small libraries, or, basically, anything that isn't written in C or Perl. E.g. Python libraries are typically a long way out of date (even compared to other distributions), and Debian's package management tends to be less compatible with Python's built-in tools like pip than other distributions (e.g. Debian is more likely to rename a Python library, in my experience, which then breaks reverse-dependencies on that library or leads to having two incompatible copies of it installed). So people running Python programs on Debian tend to end up with either a difficult-to-manage mix of system and non-system dependencies, or multiple parallel installs of Python.
You could say that Debian offers a reliable platform and it's the users' fault for installing these unreliable things on top of it. But an OS exists to run applications, not the other way around, and I find that in practice Debian's approach means users are forced to step outside the managed parts of it much more than with other distributions, and it handles it less well when they do, making for setups that are less reliable in practice. Put it this way: the "add-on repository" culture is a lot stronger in Debian/Ubuntu than in other distributions, and I think that's actually a weakness rather than a strength.
That's not how I remember it. I remember distros (Debian and others) patching cdrecord, which resulted in the upstream author getting furious and re-licensing cdrecord under a non-free license to stop them. Of course the software was then forked, and the author indeed ragequit free software development. People can read more of the story here: https://en.wikipedia.org/wiki/Cdrtools#License_compatibility...
There's a reason stuff like this is named after Schilling: https://mako.cc/copyrighteous/award
> I think I remember something about the python 2 -> 3 transition being harder on Debian because of changes they've made?
No? There was no transition. 2 and 3 coexisted, 2 was recently removed.
> The packaged Tomcat on Debian is so different that every time I've seen someone running Tomcat they've installed it manually instead.
Different as in file layout?
> Debian has essentially given up on packaging Hadoop. Some of their criticisms of its build process are certainly valid, but others seem to be an unreasonable expectation that everything will build exactly the way Debian expects. E.g. if I'm understanding correctly, they essentially take the position that they won't package anything built with Maven, on frankly spurious grounds.
Hadoop is not buildable with non-bundled dependencies, as far as I understand. This violates Policy. We were discussing reliability – am I missing something here?
> Debian tends to end up with very outdated versions of anything from an ecosystem that uses large numbers of small libraries, or, basically, anything that isn't written in C or Perl. E.g. Python libraries are typically a long way out of date (even compared to other distributions),
Can you show some examples?
> and Debian's package management tends to be less compatible with Python's built-in tools like pip
This is just not true.
> than other distributions (e.g. Debian is more likely to rename a Python library, in my experience, which then breaks reverse-dependencies on that library or leads to having two incompatible copies of it installed)
The package names are often renamed, but not the Python modules.
> You could say that Debian offers a reliable platform and it's the users' fault for installing these unreliable things on top of it. But an OS exists to run applications, not the other way around
Python libraries are hardly applications.
> Put it this way: the "add-on repository" culture is a lot stronger in Debian/Ubuntu than in other distributions, and I think that's actually a weakness rather than a strength.
Add-on repos are frowned upon in Debian.
Right, because he was getting blamed for bugs introduced by those patches.
> Hadoop is not buildable with non-bundled dependencies, as far as I understand. This violates Policy. We were discussing reliability – am I missing something here?
Well, if you can't install programs you want to use, that's not reliable. My point is that in practice you end up with an awkward mix of system-managed and non-managed programs installed, because there's a lot that Debian is unwilling to package, and such a system becomes unreliable, particularly when those system-managed packages diverge from their upstreams.
> Python libraries are hardly applications.
No, but in the Python ecosystem it's normal for new versions of an application to require relatively up-to-date versions of a large number of small libraries. So the applications end up out very out of date. I think there was a post here not so long ago from the Debian side about how it's increasingly unsustainable to try to include programs from ecosystems with a small-library mentality in Debian.
> Add-on repos are frowned upon in Debian.
And yet I've seen far more of them being made for Debian than for other distributions.
why would that definition make any sense ?
I have never heard that definition of an operating system, but it seems that every university & place has its own so we're all speaking different languages anyways when using that word :-)