Ubuntu 17.10 corrupting BIOS of many Lenovo laptop models
bugs.launchpad.net
bugs.launchpad.net
At the time systemd got a lot of serious heat and poettering got called names for something that was not systemd's fault to begin with.
> To make this very clear: we actually write to the EFI fs in systemd. Specifically, when you issue "systemctl reboot --firmware" we'll set the appropriate EFI variable, to ask for booting into the EFI firmware setup. And because we need it writable we'll mount it writable for that.
Why would you even write to efi ? Maybe at boot, i don't know, but certainly not while running ? Not all the time ? Oh i hope there isn't a single program that continuously writes to efi.
In my opinion systemd is at fault, as they aim to be the "system" and yet every once in a while prove that they don't know shit.
EDIT: A program that changes the boot order is all i can think of. (I'm not the one who knows best, on this topic)
Userspace utilities expect it to be mounted r/w, so changing it unilaterally to read-only will break things if it is intended as a "bug fix" -- which is annoying to ship to users if one fix causes another regression. For example, OpenRC ships it as r/o, but various tools like grub-install do not remount on their own, so they fail strangely. It could be fixed to have userspace mount it, but those things require coordination. Remounting is also a race-y condition for various tools to engage in on their own, for obvious reasons. It also does not change the fact those EFI variables are still exposed as writable once mounted r/w, so even closing the race condition, your system can still be bricked due to a bug.
The fix for this in the kernel was to block writes to non-standard UEFI options in efivarfs making them immutable, which prevents accidentally nuking them. This was the actual source of the 'bricking' in the original bug. This is a low-cost fix inside the kernel, which already has to deal with hostile hardware in a million other ways and sanitize various other things to stop all kinds of hardware from crapping itself, just like this. It is also relatively easy for distributions to backport, does not require changing mount semantics for systemd, does not break existing UEFI userspace tools (which would take time to coordinate, and will not manipulate non-standard variables anyway), and stops people from bricking themselves.
Logistically, the kernel is really the best and most low-cost place to fix such an issue for everyone, regardless of mount policies on efivarfs in the init system -- in a way that is actually safe for everyone. It would have taken the same time, or more time, to fix it badly (e.g. break userspace tools and require userspace remounts) as fixing it this way. You could level better complaints at UEFI or efivarfs or systemd in general than this one, all things considered...
> (I'm not the one who knows best, on this topic)
I like how people somehow breathlessly say "systemd developers prove they don't know shit, of course its their fault!!!" and then immediately follow it up with a bunch of fairly basic questions and just admit "oh by the way i don't know what i'm talking about, can anyone tell me otherwise???"
systemd annoys me sometimes and I don't like some of its design decisions, but by far its worst "feature" is it causes people who are very unsure of what they are talking about to suddenly become experts about it.
>I like how people somehow breathlessly say "systemd developers prove they don't know shit, of course its their fault!!!" and then immediately follow it up with a bunch of fairly basic questions and just admit "oh by the way i don't know what i'm talking about, can anyone tell me otherwise???"
I like how stating to a bunch of people who don't know anything about you that you are not all knowledgeable, and asking questions to better inform yourself and to put things into perspective (for everyone involved) is looked at as a weakness/sign of ignorance.
For the record i know a lot about how computers, and linux, work. From the basic transistors and gates, to high level programming constructs. It's just that this particular topic does not interest me, nor will knowing it help me in any way, shape, or form in my life.
As for "systemd developers prove they don't know shit, of course its their fault!!!". It is. Shit has happened and their response was "Not my problem"... numerous times. If the people that declared themselves "the grand arch-mages of.. everything", and are forcing their "arch-mag-iness" as "the standard of all and forever", then they are responsible for the problems of "everything". As they say "with great power comes great force times distance over time", or something like that.
At the very least, efibootmgr. The problem with userland mounting it itself is that it has no code to mount it itself because even before systemd everybody mounted the filesystem read/write.
> Why would you even write to efi ? Maybe at boot, i don't know, but certainly not while running ?
A whole bunch of reasons. Maybe you want something that's tied to the hardware without relying on the OS (eg, tpmtotp). Maybe you want to be able to differentiate between a clean and unclean system shutdown. Maybe you want to indicate that the system should install a firmware update next time it's in boot services.
> In my opinion systemd is at fault, as they aim to be the "system" and yet every once in a while prove that they don't know shit.
At the time this issue was raised, nobody knew shit. We had no idea that removing certain variables might prevent a machine from booting. If we had done, I'd have written the filesystem differently.
> I'm not the one who knows best, on this topic
I wrote the code in question. I am one of the people who does know best on this topic. Systemd did nothing wrong.
Leaving efi writable like this by default strikes me as a very poor and unsubstantiated design decision.
Yes, except that none of the tools in question have any support for doing that and so mounting the filesystem read-only rather than read/write would break them. And congratulations, now you've still got a window where something can still break everything while the filesystem is mounted read/write (what happens if the tool crashes during that phase?).
> Leaving efi writable like this by default strikes me as a very poor and unsubstantiated design decision.
Yes with hindsight that's entirely accurate and it was also my decision and nothing to do with systemd.
All 4 of the tools (that can even be wrapped in a shell script until they are, as systemd devs say, "fixed").
>And congratulations, now you've still got a window where something can still break everything while the filesystem is mounted read/write (what happens if the tool crashes during that phase?).
Better to open a window in the morning then to tear down the whole wall forever.
I agree, but I'm also sure you would agree that a small window were things can go horribly wrong is much more preferable than than a perpetually open one (sudo and friends stand testament to this).
Please bear in mind that my comments are in no way an attack on your person or one directed at systemd. What I want is to find a workable solution or, at the very least, a compromise to prevent this type of issues from occurring. I strongly believe that protecting hardware is a core responsibility of any operating system.
Would it be reasonable for me to ask you to coordinate with all the major tool developers to gradually phasing out this behavior? As in both the filesystem and tooling continue to support both current semantics and a newer version until the older semantics can be safely disabled. This is how support tends to get phased out from the kernel for obsolete hardware.
Issues of this type have reared their heads many times already and I expect they will continue to occur if the underlying reason is not addressed. As the person who has the ability to do something about it, I urge you to reconsider.
What's even more preferable is a situation where the kernel doesn't let you do the damage in the first place, which is what we have now.
To see the state you don't need write permission, obviously.
When things are set, they are set by a few very specialized tools.
Those are the facts everybody can agree on, right ? Can we also agree that mounting and unmounting the efi filesystem is really easy ? I assume it can be mounted on multiple points.
So is there a problem in re/mounting it read/write only when needed ?
>At the time this issue was raised, nobody knew shit. We had no idea that removing certain variables might prevent a machine from booting. If we had done, I'd have written the filesystem differently.
That's the difference between systemd devs and people capable of critical thinking. You'r a good programmer, if you ask me. (edit: just to be clear, i don't see you as a systemd dev; even if you are (idk), you are not the one making high level design decisions (of which this is, kind of, one))
In a race-free way? No. In a way that fails safe? No.
I personally would blame Lenovo for releasing something so brick-able.
I also learned that backup bios chips were a thing but not usually available on a notebook PC
I would look at the wonders of UEFI.
Made the news some years back I think.
This problem is about breaking the BIOS in some way, something that you should not be able to do.
To learn the real story, go and read https://news.ycombinator.com/item?id=11152880 where you will find Matthew Garrett xyrself participating in the discussion.
[1]: https://www.phoronix.com/scan.php?page=news_item&px=UEFI-rm-...
https://www.phoronix.com/scan.php?page=news_item&px=UEFI-rm-...
It's also possible to bridge from a higher level to a lower level, typically when flashing your BIOS, chances are you're doing it from the OS (it's usually possible to do it at a lower level though).
In this case it looks like a kernel module that does exactly that has been enabled before being ready from prime time. Woups.
The BIOS is not to be messed with by user software or the OS. It should be signed and only changed by the manufacturer tool to update it.
From never-fixed bugs in manufacturer's firmware to setting up things to happen on the next boot, there are innumerable reasons to write to the BIOS from a higher level.
:(
[1] https://github.com/torvalds/linux/blob/master/Documentation/...
[2] https://github.com/torvalds/linux/blob/8afda8b26d01ee26a60ef...
This broke a few programs I use, [2] mainly the screenshot application "Shutter". Most users don't realise it became the default, so their apps just stopped working... bad decision in my opinion.
[1] http://www.omgubuntu.co.uk/2017/08/ubuntu-confirm-wayland-de...
[2] https://askubuntu.com/questions/971273/screenshot-feature-of...
Owner of a $3k Lenovo P50 here that was "Linux compatible" but a stake xeon that had been useless on Linux until the latest Fedora.
The main problem is Windows because it allows manufacturers to get away with the 'it runs Windows so it's good enough to release' mindset.
Lenovo supports redhat and ubuntu, that's already a lot. https://support.lenovo.com/gb/en/solutions/pd031426
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1734147...
----------------------
Dell BIOS: 1.5.1
i7-6700HQ CPU @ 2.60GHz
16GB RAM
Intel® HD Graphics 530 + NVidia
512GB SSD
The real problem, imo, is that all of the solutions work badly (they either work at a performance penalty (as with Bumblebee) or require a separate X server for each GPU).
This is entirely a result of NVIDIA's refusal to implement decent support. The Nouveau drivers offer proper PRIME support, which yields an experience more like that on Windows in terms of workflow and the amount of setup (virtually none) required, but they have their own performance problems. Hybrid graphics with AMD GPUs apparently works nicely with whatever drivers you like, granted that the software stack is sufficiently up to date on your system.
However if you really want to, everything pxc says below is spot on...
There were people in the bug report reporting issues on the XPS 13 and many more brands than just lenovo. Try waiting 6 months until Ubuntu can sort their shit.
edit: there is a guy complaining that it screwed his XPS 13 a few messages up.