Reading poettering's response (https://github.com/systemd/systemd/issues/2402) I'm left feeling like he needs a thorough "Mauro, SHUT THE F* UP" response from Linus himself (https://lkml.org/lkml/2012/12/23/75)
The problem is not systemd, its disastrous proprietary UEFI implementations that are shipping the most insecure and awful code in the world.
The problem is we cannot fix this for 9233. MSI will absolutely refuse to disclose the firmware to his laptop so that he can make it so his replacement does not also brick itself. People have been treating coreboot / libreboot like a joke for a decade, but this is exactly why those projects matter and why the continued erosion and trend towards firmware blobs and proprietary bootloaders cripples individuals control of the hardware they supposedly own.
Its the John Deere tractor problem, but until enough people care - I mean, enthusiasts and techies already don't care, and we would need a popular general consumer movement to care to inspire real change - it will only get worse.
All the 802.11 AC wireless NICs in the Linux kernel use firmware blobs. As of Skylake, there is not a single GPU supported on x86 systems in Linux that does not use firmware blobs. Almost every Chromebook is shipping Coreboot with cancerous unauditable firmware blobs. Samsung SSDs have bricked themselves because of their proprietary firmware blobs. Its a constant endemic problem yet nobody cares to put their money where their mouth is.
Hardware has bugs. A lot of hardware has had bugs for a long time. Linux has had tables of "quirks" for hardware pci ids / usb ids / etc. for a long time, for thousands of buggy hardware devices it needs work-arounds for. Some of those bugs are really in hardware, some are in the firmware loaded on the hardware, it doesn't really matter. This is a pervasive reality, and it can't just be demanded that the user get hardware which is not "shitty" by this metric ... it's all a trade-off.
And finally, I've used linux on bios systems and efi systems, and I've never needed efivars mounted, I've always set up the bootloader some other way (which was simpler for me to control and manage as I prefer). My personal biggest complaint about systemd is how it automatically mounts and starts and uses all kinds of stuff that I don't need. I prefer to set up what I need and want, and not have anything else cluttering up my system, just waiting to cause serious reliability or security problems, and getting in my way when I'm debugging something else.
So I'll be up-voting all stories about "systemd did something automatically and on some systems it was unfortunate" because yeah, UNFORTUNATE STUFF HAPPENS WHEN STUFF HAPPENS AUTOMATICALLY. This is why I left windows and OS X in the first place! So I had easy and convenient control over my computer! And now it takes extra effort to override and disable all the crap that systemd is doing automatically, and I resent it. (I actually already have a script on my systems that unmounts efivars and pstore and some other unneeded filesystems after boot.)
I think the blame is put on the wrong party here. Systemd mounts efifs so that only root can rw to it, root rm the fs, hardware is affected. Places where there is a bug, hardware+firmware + unix idea that everything must be a file. Systemd follows the specs and behaviors that are expected of it. If the EFI fs should not be a FS complain to Linus. If the EFI should not brick itself after a rm / complain to EFI developers.
Systemd already took the reasonable security precautions. Root can rm anything it wants on Unix systems. rm efi fs is dangerous, so only root can do it. If root does it then all bets are off. Root needs to be able to write to the fs, per api and other tools needs.
People are quick to blame Poetering, but it is Linus who is leading the project which has the design decisions that are causing problems.
In the end it sounds like 3 projects needs to change their code to avoid an issue with a user mistake/bad firmware combo that only avoids that issue an a blue moon Monday.
All in all the usual storm in a tea cup against systemd. In that case it is funny because the issue is that systemd is to unixy :) everything is a file (system).
Another reasonable option would be for a distribution to include SELinux policies that allow only the blessed tools (grub-install, systemctl, etc) to write to that filesystem. It would be a big change, though, because most distributions leave root and normal user logins unconstrained.
In hindsight you can say "we should have mounted efi ro and made our efi tools much more complex so they can remount it rw write to it and then remount it ro again" but that is asking the systemd and efitool developers to have been prophets that motherboard firmware makers would be so disastrously inept and stupid they would put essential parts of their boot process in a shared variable table with the OS.
Its like saying you cannot ever introduce a memory clear into a GPU shader because you might wipe the firmware and the GPU could explode when its fan shuts off. You have to have the reasonable expectation the API you are working with is conforming, and when it is not you need to raise hell about it, and in the short term introduce the work around until the upstream problem is resolved.
The issue here though is that the upstream problem is proprietary farmed out awful firmware that restricts your freedom and bricks your hardware because nobody can audit or fix its terrible design.
OS X isn't too bad of an approximation though, if you don't sign into iCloud, turn off spotlight integration with internet services (see the recent story about people's macs crashing due to a malformed response from the server!) and don't link your iPhone to a mac (or don't have one). Finally, don't update to the next release until just before it's replaced by another, e.g. don't upgrade from 10.10 to 10.11 until just before 10.12 is released (but do install security updates of course).
But then again ... you'd have to be a bit of an expert to know whether you could trust my advice, and if not, who to trust, especially since you'll definitely get contradicting advice from similarly-knowledgable-sounding people ... so you're ultimately on your own.
I personally do not want to be forced to know all the hazardously settings that exist in the kernel and base system. I want sane defaults, defaults that are set automatically when I install my Debian packages. If I disagree with an automatic default, my options are to either utilize the control (freedoms) that FOSS software provides, or issue a bug report and try get the community to agree with my views.
If "rm -rf /" is a valid use case/concern, send up a bug report. Its better than trying to remove automatic defaults from distributions.
That's kind of an ironic argument to make, given that you are using it to argue in favour of a design where a single mistake can actually brick your machine to the point that fixing it needs a soldering iron.
There are more operations that root can do that can brick a system or destroy hardware. Why should systemd try even harder to make root not do that?
Since the OS doesn't provide permission levels to express this difference, it makes sense to create that isolation otherwise.
I've run rm -rf as root in the wrong directory before, and nuked stuff that required a backup to fix. I'd prefer if everything worse than that required some extra mental confirmation that, yes, I'm sure I want to do that.
A lot of distros also alias "rm" to "rm -i", something that many users explicitly disable. Its a complex problem of security vs usability where most discussions has been rehashed several times.
Using rm on its own will happily perform the command without further verification.
On the other hand, rm -i will request a yes/no on every last file involved.
Personally i have taken to using mc for any "complex" file system manipulations.
I think the two competing ideas are:
Either you make libraries that strictly conform to the spec and are "technically perfect", leaving any bugs that stem from non-compliance to the violating parties (i.e. the UEFI implementors)
Or, you make libraries that conform to the spec and have some "dirty" handling to work around holes left by crappy UEFI implementations.
The first options feels great to write as a developer, but the second option is what most users really need - it's the "it just works" factor that users care about. I know it hurts to code around shitty implementations, but there's no other alternative if reliability and idiot-proofing matters.
The best I can suggest is to make the workarounds "pluggable" so that developers don't have to deal with the harsh realities unless they specifically go looking for the plugged-in workarounds.
The computer industry really needs to overhaul how system firmware is developed, tested, and deployed. Linux devs and users aren't the only ones feeling the pain; they're just the ones who are both skilled enough to trace the problem back to firmware, and inclined to rant about it publicly.
We have seen this happen elsewhere, and seen how it turned out. Browser rendering engines used to be NIH proprietary behemoths and IE6 was the pinnacle of the failure of that model. Graphics APIs and hardware are another great example - game developers are writing broken shaders that driver developers need to work around because the API is broken and the hardware is undocumented proprietary bullshit. Then you end up with insanity like Nvidia having the de-facto standard OpenGL implementation in the industry such that a ton of OpenGL programs break on AMD drivers because they don't have nuances and breakages of the Nvidia one.
But gradually people realize that its not a profit center to be a closed source broken douchebag. Either the industry goes through the pain of correcting the damage (gecko / webkit / blink / etc) or just abandons the insanity (Metal / Vulkan / DX12 / Mantle).
The thing is we are not putting any financial pressure on motherboard makers to not be complete proprietary douchebags. Libreboot and Coreboot are at best infant hobbyist projects with extremely low penetration and they take half a decade to support products or just ship firmware blobs that were half the problem in the first place anyway (cough, Chromebooks). Network routers used to be that way, until the grassroots OpenWRT projects were so succesful manufacturers started making sure their routers were supported (TP-Link / Linksys) or even in the best scenario selling hardware running open source firmware (Buffalo). This needs to happen with x86 chipsets and motherboards desperately because UEFI is making the ACPI mess worse when the answer has been obvious this whole time - open documentation and specifications and working together rather than working in secret.
God forbid you should try and play a new AAA game without the latest patches for your driver!
This is wrong on so many levels. It makes it really hard to compete with nVidia/AMD when you need to hire hundreds of driver writers to patch every game under the sun if you want to have reasonable performance.
For example, they included an exception in their memory protection code for SimCity. This because said game held a "use after release" memory bug that had gone undetected. And rather than have the game crash on later MS product releases, MS put in said piece of code.
It's a long-standing tradition to have workarounds in kernel modules for broken hardware.
Hopefully there is a sufficently-well-funded testing lab out there to get a complete list of UEFI implementations that are broken in this way so that the next rev of the uefifs module can properly refuse to do whatever it is that's bricking these dangerously broken motherboards.
But that is my whole point - there is. We can need market pressure that it is unacceptable to sell computer hardware whose controlling software's source is undisclosed.
There is no solution for this user, but settling for broken implementations and broken libraries is the stuff of the Windows world where everyone behaves like a baby who refuses to share their toys.
This isn't Facebook or any consumer proprietary product where the software is the product. This is hardware controlling firmware that is never pretty or sexy and whose implementation is not a trade secret or the difference between selling units.
The IBM PC, XT, and AT came with a physical copy of the complete source code for the BIOS, several hundred pages in a 3-ring binder. (You can get a digital copy of it here: https://archive.org/download/bitsavers_ibmpcat150ferenceMar8... ) IBM did this, despite the fact that they strongly recommended applications use the BIOS interfaces and not access the hardware directly. They could've saved so much paper if they just documented the API, but they didn't --- they released the whole thing.
Yet 30 years later, in an era where it's easier than ever to distribute large amounts of information, companies regard such details as confidential and proprietary, hiding them even from their customers --- the real owners of their products. It costs next to nothing to distribute a DVD with each machine containing all the source code and documentation, or just put it up for download. Unfortunately the only recourse seems to be the occasional leak[1], and what's more disappointing is the overwhelming response that this is a "bad" and "insecure" thing, when it's really liberating what users should've received with their hardware.
[1] http://adamcaudill.com/2013/04/04/security-done-wrong-leaky-...
Create a virus that goes around messing with UEFI vars to brick the machine. Don't damage any user data, just the BIOS.
It'll get fixed.
Why a filesystem and not a library? Write access doesn't require a filesystem mount.
How often do you need to run efitools though? Should it always be mounted read/write for the one or two times in the lifetime of the system that you need to adjust the boot variables? Wouldn't it be more reasonable for efitools to suggest you might need to remount it read/write while you're changing the variables?
Systemd has brought lots of good things to Linux, but tons of bloat and insanity too. I'm quite scared because the next thing they seem to be tackling is one of the core things that makes Linux so nice, package management [1].
They intend to container-ize everything. While that might be in principle good (NixOS got it right), their solution is likely to end up being a mess: no control over container contents. Imagine a critical security bug on e.g. OpenSSL. Good luck patching all your dozens of unmanaged containers running different OpenSSL library instances.
[1] http://0pointer.net/blog/revisiting-how-we-put-together-linu...
I think what gets quite few riled up over systemd, besides the continued shoggoth-like scope creep, is the flip flopping on how things are handled.
For example, if you run mount -a, mount will present an error pr failed mount, but will continue trying to mount the rest of the entries in fstab. Systemd on the other hand will fail the whole mount unit on just a single such error, and this in turn will fail the mount because various other units depends on the mount unit completing.
This has bitten admins of remote systems that has gotten "upgraded" to systemd as part of a update of stable distro releases. All because they had a straggling entry in fstab that may have been sitting there quietly for a decade.
Then you have them overriding a kernel default on mount namespacing, because the containers people want it that way, while others have come to rely on the kernel default being, well, default.
I don't think they have yet to "solve" handing NFS mounts, instead giving the world their own take on ctrl-alt-del, while disabling the sysrq sequences.
Or how systemd would (will?) blank out a environment variable via systemd-pam when su -l was used, resulting in people getting their personal settings getting mauled by root. Apparently su is "broken", according to Poettering.
And now Poettering goes ahead and closes this report with what amounts to yet another "wontfix".
Theodore Ts'o seemed to hit the nail on the head nearly 2 years ago[1].
And iirc, containers are crap sandboxes.
If Nginx, for example, has a security flaw, it's properly Nginx's job to make a new container. That flaw might be because of Nginx's dependency on OpenSSL, but it might also be because of a vulnerability in Nginx itself. Nginx (and every other "app" developer) needs the process in place to get new releases out in the latter case; if they have that process, there's no reason to not apply it also to the former case.
Distro maintainers have been picking up the slack of creating "security releases" for app software for decades now, but they only do it because app developers were unwilling to take on the responsibility of maintaining packages in several different package ecosystems. App developers are not nearly as timid when it comes to container-image releases, since it's just the one QA process rather than five. So, properly, the responsibility—and the SLA—is reverting back to the app developer.
You can't expect, now, that paying Redhat or Canonical for a support contract will mean that Redhat/Canonical will be required to rush you patches for your app software. It never worked this way anywhere else; Microsoft and Apple aren't responsible for patching the vulnerabilities in the apps in their ecosystems. (Even the ones you buy through their stores!) The Linux world is just coming back to parity with that.
Now, on the other hand, system components—things the OS itself depends on—still need to be patched by the OS manufacturer. Microsoft still delivers IE patches; Apple still delivers Webkit patches. Because those are fundamentally system components, used by the OS to do things like drawing notification dialogs.
Those components happen to come with apps, but the system component and the app are decoupled; there's no reason the version of Webkit the OS uses, and the version of Webkit Safari uses, need to be the same. And they're not: you can download a fresh Webkit from Webkit.org and Safari will pick it up and use it. Apple, thus, only takes SLA-responsibility for the system-component Webkit—not the "app Webkit." The same is (soon to be) true for Linux distro-makers.
---
The near-term effect of all this, of course, isn't that app developers universally pick up the maintenance-contract stone and start carrying it. Most have spent too long in the idyllic world of "downstream support" to realize, right away, what this shift spells for them. Instead, in the near term, it'll be your responsibility, as a system administrator, to be a release-manager for these container-images. This was always true to some degree, because every stack tends to depend on some non-OS-provided components like Nginx-current or the non-system Ruby. But now it'll be pretty much everything.
An interesting incentive this creates is to move your release-management more toward monolithic synchronized releases. If you're managing the releases of all the components in your stack, it's much easier to think of yourself as releasing a whole "system" of co-verified components, rather than independently versioning each one. When you upgrade one component, you bump the version of the system as a whole. This sort of release-management is likely familiar to anyone who has managed an Erlang system. :)
I can have containers, but these containers are still manageable, not black boxes.
Furthermore, since packages come as declarative recipes, one can try to reproduce binaries (see guix challenge command). Otherwise you are at complete mercy of the packagers.
For very large deployments, Docker-like containers are fine. But for desktop applications I think it's not the way to go.
If you're writing software from scratch, though, by all means, target Nix as your package-dep ecosystem, and make your own software into a Nix package. Nix is a great source for creating "community verified-build" container images, for any "legacy" container ecosystem. Where a Nix package exists, it's exceedingly simple to write a container-image formula sourcing it. Thus, having a Nix package makes the software much more accessible to many platforms.
It would also, as an aside, make perfect sense to enhance systemd with the ability to run Nix "units" that are just references to Nix packages that get resolved and downloaded+built at launch-time. (systemd would still be better off turning those Nix units into container-images, though—systemd relies on containers to provide runtime sandboxing, ephemeral container filesystems, and cross-image deduplication of base-files.)
So, for small teams, I see a likely move more toward the model we see with "community-made verified-build Docker container images": a GitHub repo containing the container-image formula for the release, that many small teams depend on and submit PRs to when vulnerabilities occur.
While not ideal, this is far better than the Ubuntu PPA style of "some arbitrary person's build of the release." It doesn't give you anyone to blame or bill for downtime, but it does give you frequent releases and many-sets-of-eyes that hopefully make your downtime quite small.
It's a bit like the atomized "bazaar" equivalent to the "cathedral" of a distro package-management team, now that I think about it. Each verified-build formula repo is its own little distro with exactly one distributed package, where anyone with a stake in having good releases for the software can "join" that distro as one of its maintainers. :)
It doesn't even matter if you're pro or anti-systemd. That sort of response just shows a huge lack of understanding about the severity of the problem.
Note that he say writable only by root and well protected. As far as I can tell, that's agreement that there needs to be something done to make it safer. All his other statements seem to be noting that it's not as simple as making it always read-only, as there is legitimate need for write access in some instances, and tips on how to mitigate the current issue.
The personal attacks on Poettering should be directed to Red Hat instead, who actually have the power to do something about it.
I don't think you quite understand the relationship Red Hat has with its employees ("associates" in company lingo but I abhor such doublespeak). Allowing those employees to act and speak independently is kind of a core value. That freedom is only curtailed when it directly impacts paying customers to the extent that it would be considered a breach of that relationship. Upstream, in the open-source community, almost anything goes. Yes, that means systemd developers can be a bit priggish. It also means other developers, also employed by Red Hat, can call them out for it. It's the only way to ensure the diversity and meritocracy that are essential to innovation. Otherwise, you end up not being able to trust a word that employees of a company say because you know they'd never dare say anything even slightly inconsistent with the party line. I used to see that when I worked at EMC, just for example, and it's really quite stifling.
Personal attacks on Poettering should not be redirected anywhere. For one thing, personal attacks don't get anyone anywhere. Legitimate criticism of his views should be directed at him, just as legitimate criticism of my views should be directed at me and legitimate criticism of your views should be directed at you. There's no reason to bring any third party into it. No matter how much you hate them or why, that's simply irrelevant.
Meanwhile, on the GitHub issue:
>Locking this one. Note sure which peanut gallery site linked this...
This is clearly a guy who thinks his own opinion is above reproach and the unwashed masses have no right to question him.
The problem you seem to miss is that his arrogant antics _do_ reflect on Redhat, and reflect badly on Redhat. I for one will _never_ use anything from Redhat, nor pay Redhat any money for anything as long as Poettering and the rest of his crew (Seivers et. al.) are employeed by Redhat.
I am sure I am not alone in this viewpoint. As more realize that the problem is a problem at Redhat, eventually Redhat management will be forced to intervene.
PS as an employee, you should be pointing your upper management to these types of discussions, for the very reason that the actions of a loose cannon group is reflecting very badly on Redhat as a whole.
I think you're missing the point. It's not about who you tell. It's about who you're criticizing. Red Hat is not the one making these comments. If you condone a company summarily removing a project leader you don't like, then you also condone a company summarily removing a project leader you do like. That doesn't end well. In fact, I could name projects on which I feel Red Hat has forced their will on upstream entirely too much, to the detriment of both. It's not the way a meritocratic community is supposed to work. I think in general it's better to let technical communities deal with their own issues, and in general Red Hat is wise enough to recognize that.
Believe me, I know where you're coming from. I almost didn't join Red Hat myself because of people like Ulrich Drepper (who was still there) and Al Viro (who would even have been in my group). I understand the sentiment. Criticize Poettering if you want, make sure Red Hat knows the effect that his behavior has on your purchasing decisions, but don't blame them for trying to do the right thing by adhering to a policy with a solid philosophical and practical foundation behind it. Do you want corporate overlords to be meddling in every project's internals?
Poettering, being a high profile employee of Red Hat, is one of the faces of the company. Because of this, I would argue that Red Hat has a duty to step in from time to time to tell him that what he is doing is harmful in the long term. If they don't do it soon, I can almost guarantee there will be a lot fewer support contracts for Red Hat in the months and years to come; they just don't create dependable developers.
If all Poettering did was community outreach, you would have a much stronger case that he's bad in all aspects of his job and should be fired. Trying to pressure Red Hat by saying it 'reflects poorly on you' is basically saying 'you should get rid of this guy even if you think the good outweighs the bad, because I don't like him.'
If this is true, then this is exactly _why_ the criticism should be directed towards Redhat. Because in his arrogant world view, he can do no wrong, so he will not report to his bosses that he has ineffective community leadership/stewardship. The only way his bosses will know of his ineffective leadership/stewardship is if the criticism is directed towards Redhet, and therefore, his bosses.
I.e., the criticism has to go around the roadblock, Poettering being that roadblock.
Three employees among thousands, representing two projects out of hundreds. Why generalize from that sample? Why ignore all those contributions to the Linux kernel or Fedora, OpenStack or Kubernetes, gcc or coreutils? Some pretty good leaders in there. And if "creating leaders" is supposed to be how we judge companies, what should we make of much larger companies where few employees engage with the community at all? I don't even mean unpopular companies like Microsoft or Oracle. What about Google, for example, or Apple? When it comes to community leadership, they're net negatives; existing leaders go in, and are never heard from again. When every single developer at a billion-dollar-a-year software company is engaged with some open-source community or other (often several), there will be a few losers. That's a poor reason to insult thousands of others.
The whole thing reminds me of how priests and monks would debate the number of angles that could dance on the head of a pin.
So that's basically what uint32 proposes: to make an impact.
I can't seem to locate it now. But i seem to recall a video released within the last year so so that showed various people from within RH, regarding the history of Linux. And at one point one of them energetically declared "we won" in regards to some "unix war".
All in all the video gave the impression of a company culture that had the mentality that they could do no wrong.
Start informing Redhat that these responses on Poettering's part reflect badly on them as a whole, and especially _stop_ paying Redhat any money until the issue is resolved, and Redhat will resolve the issue very quickly.
Poettering and Sievers are just freedesktop.org adding another layer of feces on the pile. The fact that people switch distributions to avoid his applications (NetworkManager, PulseAudio, Avahi, systemd), and the fact that someone hacked XBill to include him because he's a worse villain than Bill Gates, should put him on a level that even Ullrich Drepper couldn't touch, and everyone hated that guy.
Redhat has been a useless organization since they chose to promote Havoc Pennington over Carsten Haitzler (longer than any of you have known they exist). Blaming them does nothing to stop their slow lurch over the linux landscape. Boycotting them might, but there are too many RHCE's out there. Best to focus on FDO and Poettering/Sievers.