System76 ME Firmware Updates Plan
blog.system76.com
blog.system76.com
Relevant source code can be found in the following places, keep in mind that it is still work in progress:
- System76 Driver with Firmware Update support: https://github.com/pop-os/system76-driver/tree/firmware_artf...
- Firmware Update Frontend: https://github.com/system76/firmware-update
Please ask me anything
Hear that, Intel? I will put money where my mouth is. I am not sure I can afford any of these RISC-V chips and they are still alpha quality beyond the Talos, but those of my ilk will do their damndest to make you pay for tone deaf reaction.
You are still buying Intel, so not sure how they will care...
So IF they had a Power0 chip it wouldn't be Intel. Though I have zero idea how System76 would put Power9 into a workstation laptop but they do put a desktop cpu into their Serval lineup. https://system76.com/laptops/serval
My system was delivered with a BIOS issue that made it only boot perhaps 1 out of 30 cycles. Delivered on a Thursday night, after spending a lot of time troubleshooting, I had to wait until their support got in on Monday to get anything done. Then I had to mail it in and wait a long time to get it returned with a reflashed BIOS (I'd be lying if I said I remember exactly how long, just remember it feeling way too long).
I explained that I had bought the laptop specifically for on-site work with a client and that I needed a functioning fix or replacement ASAP. They told me they couldn't make anything happen any faster, and even told me that they had no way for me to pay out of my own pocket for faster return shipping, though IMO they should've offered to do this themselves.
I did eventually get back a mostly-working system which I still use around the house, though it's had a habit of hard-locking when the GPU is under stress the whole time I've had it. But my experience with Sys76 customer support is the biggest factor in deciding not to buy from them again.
Do bear in mind this was several years ago and that it is reported from memory. It is entirely possible they've gotten their act together and/or that I'm misremembering some details, but this was my experience as I recall it.
so much honesty is such a pleasure to read !
Good tools cost more, and are worth it.
Your product just went to the top of my shopping list.
I won't, if it means having to run Ubuntu (and derivatives) or Pop. I'd like to run the distro of my choosing, thanks, but no thanks. It will be possible to disable ME on other laptops/desktops as well.
> You must run Ubuntu 16.04 LTS, Ubuntu 17.04, Ubuntu 17.10, Pop!_OS 17.10, or an Ubuntu derivative and have the System76 driver installed to receive the latest firmware and disabled ME on laptops
Maybe it will run fine on Fedora, but if it bricks your laptop, they don't want to get blamed.
I don't see why you can't attempt to do it manually yourself, without the aid of their driver and firmware update tool. Except that you might forfeit any warranties/support.
System76, can you clarify?
Sadly it's standard to only see official support for Ubuntu or RHEL in the Linux world. Not that I don't understand, it's all very fragmented, and that creates a lot of work as to support multiple distros. People are working on remedies, though.
*I don't have a System76 laptop
EDIT: Crap, pre-order.
https://www.raptorcs.com/content/TL2WK2/intro.html
I have been skimming too light on the market details for that bad boy. Only proves the point that the high-cost might be the price of helping proprietary hardware feel the sting of changing cultural tides.
@jackpot51 system76 folks -
Are your systems currently using opensource boot bios? If not, any plan to do so in near future?
We may be experimenting with coreboot more in the near future
I used to work on embedded systems where I have full control of uboot. With that, I can easily reserve the section of memory in boot uboot and Linux kernel when the keys logging info (such as FTRACE of ISR, sched switches etc) from kernel are kept.
When a kernel panic happens, the system will go thru the warmboot sequence and detected there are logging info in that section of DDR and dump it out before continue to boot.
This "postmortem debugging features" make debugging certain category of very difficult bugs (kernel driver panic) much easier.
With typical close source x86 bios in Linux laptop, it is very hard to enable this kind of debugging.
I've written this elsewhere, but I'll repeat: it seems a shame that folks like System76, Purism, aspiring Kickstarter campaigns et al aren't leveraging the supply chain already in place for Chromebooks. In the case of Purism, for example, x86-64 Chromebooks comparable to Purism's low end models can be had from Dell for 3/4 the asking price of the former. Considering that the driver situation is necessarily a solved problem for Chromebooks, and they're already using coreboot, it seems silly to start a parallel effort to deal in hardware the way System76 and Purism are doing, rather than just approaching Dell about a white label deal to supply you with hardware components that are already known to work.
The business model of system76 was what I was personally looking for and I've been very happy with my latest laptop from them. For me personally, the luster would have been lost coming from Dell.
Alienware was bought by Dell not that they outsourced their hardware.
A white-label, more configurable, plain ubuntu version of the pixelbook sounds like a dream.
Dell makes Chromebooks. These Chromebooks run coreboot. Ergo, Dell makes laptops that run coreboot.
There's nothing in my comment about building off the back of Dell's existing program to sell Ubuntu machines to customers—a program whose success story is reported to be bizarrely hit-or-miss with respect to driver issues, even.
Dell's Chromebooks are a success. They're shipped in large quantities. They have no driver issues. They run coreboot. So my advice is simple: if you're a company trying to sell to folks who want a "Linux laptop", then stop sourcing components/machines from otherwise clueless Wintel vendors and trying to shoehorn a traditional distro onto that. Start with the BOM used in Chromebooks. Approach Dell and say you'd like to place a large order—because that's the business they're in, by the way. Now build your company's products on that platform instead.
Substitute "Dell" there with any other competent vendor that has demonstrated their hardware plays well with the FOSS stack powering ChromeOS .
> Partnering with Google for the chromebook supplychain seems like the better theory/plan.
No, that's an ignorant theory. Google is not known for hardware, and certainly not as a supplier for commodity hardware. Even ignoring all that, the response you should expect from Google in regards to playing along should not be enthusiastic agreement.
There is a demand for this, and almost no one is doing this. The only other company that has done it so far is Lenovo, and if you know anything about privacy and hidden adware and history of spyware - you will not buy any Lenovo product.
ECC memory would require a Xeon, but it would be possible to underclock it to bring it to a mobile heat emission profile.
If I balls to the wall it I can kill the battery in 90 minutes (though they are swappable) but I average ~6 hours under actual work conditions (VMs and Intelligent).
It spends the vast majority of time sat at a few percent utilisation but the processing power is there for those few second operations you run infrequently (build from scratch, indexing a large project etc).
I love it frankly, not turned desktop on at home more than once or twice since July.
- Release ME source code
- Remove ME from consumer products
- Have a provable method of disabling the ME entirely
Intel can just give the source, but you can't trust Intel to just give you the same compiler they used, because the compiler might insert a backdoor. You'd need the compiler source, audit it for backdoors, then compile that with a trustworthy compiler. Then use the resulting compiler to compile the ME source.
(Compiler compiler compiler)
Of course that only helps if you trust the toolchain.
(Reflections on Trusting Trust)
Thanks, I really enjoyed reading it!
It is possible (though somewhat time-intensive) to audit binaries, too. If there is real demand for this, it should be possible to crowdscale the auditing problem over a large group of OSS enthusiasts.
I am wondering. Have you considered using the [Linux Vendor Firmware Service](https://fwupd.org/) ?
Source: Am the LVFS and fwupd maintainer.
Still, I have been changing our firmware updater to better suit LVFS and fwupd for when you finally decide to reconnect in a more positive way.
Not trying to tone police you, but you run a project that necessitates bringing vendors to the table. Odd to publicly bash one that's trying to work with you.
It's possible that at some point (depending on if it gets open sourced) I'll be grabbing some of your work and trying to wrangle it to work on my desktop. Is this an area where I seriously risk bricking my CPU, or are there safeguards that cause these kinds of instructions to either work or safely fail?
If you have BootGuard, you will brick your machine, unless you have the manufacturer signing key. We have the signing key for our laptops, so we are able to sign updates for the four systems with BootGuard: galp2, galp3, lemu7, and lemu8.
As far as I'm aware, there's no particular state involved for you to brick the CPU, but you can easily write a firmware which will prevent boot, or fail some time after boot (In the case of removing critical ME blobs). You could brick your board temporarily, but you're going to be using an external flasher (with a convenient little SOIC clip :- ) anyway, so fixing it is a matter of writing back the old/working image.
This is, of course, assuming you are able to write anything that works to the flash, which you might not be able to thanks to BootGuard.
I'm aware that you're working on a distro-agnostic tool. In the interim period, however, if someone with a System76 laptop wants to use Fedora while still disabling ME, can they simply load stock Ubuntu/pop_OS long enough to install the driver and disable ME and then wipe the drive and reload their OS of choice?
Would it also be possible to support consumer-controlled signing keys, placing control of the system in whose hands they belong?
Signing keys are fused into the CPU for boot guard when the CPU is attached to the motherboard during production, for soldered CPUs like the U and H class. Having a customer signing key would seriously complicate BIOS updates, as only one key can be utilized, meaning our firmware updates would not work.
It would also make returns much more difficult, as all CPUs with customer keys would have to be replaced.
Right now, boot guard is only used on the Galago and Lemur.
Yes, I expect as a general solution for your customers it might be overkill, but as this is basically the cutting edge of public domain research on these topics it would nice to know what's possible for customers who would take the risks.
Understood, I have more reading to do in to how the EFI shim/flasher works.
Awesome, I finally get to see the punchline. ;)
(It's never going to happen, I know. But a fellow can dream!)
The other thing I can't give up from my Lenovo is having two batteries in the machine (it has a 4 cell 32WH and 6 cell 72WH)
Side note: PopOS wasn't ready for release. I had lots of problems getting the initial account created and on corporate wifi (PEAP).
For example, there may be two drugs that can treat my illness; taking both might not be better than taking one or the other.
I've been considering system76 for awhile. I see Dell also offers Linux laptops now and has for awhile. Any reason why I should pick system76?
Would be a good place to start, as far as not supporting dell goes. Linux is also a first class citizen w/ system76 - Dell has the factory linux xps13 e.g. but I've heard from a handful of friends who use one at work that its obvious Linux was just kinda slapped onto the laptop at the end of manufacturing or whatever. Support is also probably much better with System76.
For my particular case, the Dell has superior hardware (also, it's about a year newer), but its Ubuntu install has strange problems, and I get crash notification popups somewhat often (including immediately upon logging in for the very first time). The Kudu's install has been rock solid.
System76 has far superior support.
is Redox partially sponsored by System76 via allotting time for you to work on it?
For example, the graphics library orbclient was used in the firmware updater, and the UEFI library from the firmware updater may be used for a Redox UEFI bootloader.
It feels NIH and is exactly why I'll be buying Dell and never System76. I can one-click flash my Dell forward without configuring or installing anything.
For that matter why is there a "HiDPI Service" running? Why is the driver a set of udev rules that could've been upstreamed?
Here is what the intelmetool [1] says on my laptop: https://i.imgur.com/yKTt5ga.png
What's even more interesting is that there is a simple, automatic, no-frills method to clamp a SOIC clip connected to a Raspberry Pi zero to any Chromebook and it will clear out the Intel ME automatically with no user interaction. It's not even hard, all you have to do is to configure the Pi to enable SPI, and then make the pi automatically run flashrom to pull out the ROM from the Chromebook's flash chip, run ./me_cleaner on the ROM image, and then flash it back.
It can be done safely and automatically, and you wouldn't have to risk frying your laptop, so everyone could do it provided they can open their laptop. However, I'm too lazy to document it properly: either by providing the image tool to create the said Raspberry Pi Zero image + h/w instruction or by providing a premade hardware to do it.
I have reasons to believe that downloading a random binary image from a random guy nicknamed jimmies and use it to flash the firmware to your laptop because you don't trust Intel is probably not a great idea. The act of creating such a script to customize Raspbian, and testing out to make sure that it works for every Chromebook or laptop, and make a hardware compatibility list is quite a daunting task. I was talking about that briefly to some of the security people, but then as a grad student trying to graduate it got to the pile of TODOs. So if anyone is interested in it, let me know and I can provide some more details.
Currently, I'm running a Dell 13 inches Chromebook that can be had for $300 and does everything I need.
1: https://github.com/coreboot/coreboot/tree/master/util/intelm... - disclaimer: I contributed a patch to the intelmetool to make it work on the Chromebook.
For the rest of the population, there's value in there being a vendor who sells magic black boxes, certified to be freshly purged of all known evil demons.
Sure, but there are things that I like about my Chromebook that the System 76 laptops, despite being way more expensive, are unable to to provide. I like how mine has a backlit keyboard, has a FullHD 1080p IPS screen, and an 10 hours battery life, all that for $300.
I just looked at the number of people who upvoted my post and I am really think that this actually could work if there is a person who builds the hardware, which I think is trivial. The problem is that not many are comfortable with both the electronics (wiring the clip) and be able to compile a big software package with dependencies on the Raspberry Pi. Make the frustrating 2 hours job to a 5 minutes job of opening the laptop and identify the chip, I think it might just be what many people want.
I talked to a very smart netsec person and he just said that because he doesn't understand electronics, he couldn't have done it himself, but if that's something prebuilt or foolproof, he would absolutely do it.
Perhaps I will actually take the plunge and publish the work over the next couple of months as a Show HN post.
This already exists for "chipping" games consoles and the like. I think it's just that the average person can see the cash value to them of pirating games while the ME has no effect on their daily life.
That sounds very compelling. Which model are you using please? I'd like to take a closer look but I don't seem to see one with these features/price - but I am in the UK.
I'm just hoping for a 100% software exploit that will do this.
Having the ability to automatically push new firmware of your own creation to customers' machines is more power than you ought to want. My security threat model as a System76 customer shouldn't have to include you (perhaps with you being hacked or coerced) pushing me malware that's undetectable to my OS after it's been installed.
Please reconsider this feature (of automatic firmware updates). Firmware updates are rare enough that it should be fine for them to be explicitly opt-in. It's great to want to make Intel's firmware more secure. But replacing Intel as a possible attack vector with yourselves is strictly worse for the machine's security.
But on the other hand, System76 customers are trusting that System76 hasn't been hacked or coerced to ship malicious firmware from the factory in the first place. These updates are signed and verified with industry best practices.
jackpot51 (the System76 engineer currently working on this) could probably detail it better than I can, though.
> But on the other hand, System76 customers are trusting that System76 hasn't been hacked or coerced to ship malicious firmware from the factory in the first place
I think it's reasonable to feel differently about those two risks.
Most notably, you only get one chance to load malware at the factory, whereas you have an infinite number of chances to push malware as a software update after that. It's harder for you to avoid being compromised forever than to avoid it at one specific moment. One person on your team could probably get malware signed and distributed as an targeted update without anyone else knowing, whereas doing it in the factory might take more coordination.
It's also tidier from an attacker's perspective to deliver malware just-in-time to a specific user, rather than to everyone, or to a machine that you hope will end up in the hands of the target weeks/months later. It's less detectable.
If you have a way to avoid being able to infer (e.g. by their IP address, correlated with other records) which human is asking for a firmware update file (or any update file) at the time it's downloaded, I recommend taking steps to deny yourself that knowledge.
Why push the firmware before asking?
>Not only aren't updates initiated without permission, I wrote the code to make that literally impossible without changes to the installed python code.
The code is available at https://github.com/pop-os/system76-driver and https://github.com/system76/firmware-update
To be clear: I think this is a great default configuration, and it's been a long time coming - but it's just not true afaik that Debian/Ubuntu won't auto-update at all in the default configuration.
[ed: Debian: https://wiki.debian.org/UnattendedUpgrades
"As of Debian 9 (Stretch) both the unattended-upgrades and apt-listchanges packages are installed by default and upgrades are enabled with the GNOME desktop."]
The process was so byzantine that I very much doubt more than a small fraction of home users would get through it, or even bother starting.
The correct steps were (1) flash a newer bios, (2) install the Intel ME driver for windows, (3) run the actual vulnerability patching tool. Discovering those steps required a bunch of trial and error and navigating Asus's really terrible website full of badly named downloads.
This is why I don't buy "enterprise users" as a reason for having IME. I've never once worked in a company that patched firmware, even though they have specialists capable of it. They want the option to perform enterprise wide upgrades with ease but they aren't willing to pay the true cost of having this ability.
> Discovering those steps required a bunch of trial and error and navigating Asus's really terrible website full of badly named downloads.
Gets even worse when your international. Half the links will be to a US address that will then try to redirect you to a localized one which will then not have the resource you were looking for. Then you've got some really byzantine export restriction procedures and you have to create an account but it still probably won't work. I've had these issues with ordinary drivers too, it's the biggest reason I support the linux in kernel tree and no stable ABI model, it's better for users.
I'll admit I was a little afraid of bricking the PC, as with any kind of BIOS modification, but it worked like a charm.
> System76 will investigate producing a distro-agnostic command line firmware install tool. Follow us on your preferred social network for updates.
https://www.reddit.com/r/linux/comments/5umefi/system76_refr...
Things like getting newer drivers working (and contributing that openly back to the community) plus getting ME disabled are some pretty big value added. If they had a 4k model I might be more interested.
Galago Pro (https://system76.com/laptops/galago) is probably our most popular, and its resolution is 3200×1800—not technically 4K, but the right resolution for that small of display.
Oryx Pro usually has a 4K HiDPI option, but it looks like it's currently sold out. Our WS models also have 4K options, but they're pretty specialized machines.
For me probably the biggest issue is that they usually have mediocre keyboards and sometimes they even have keyboards that drop keystrokes. It seems like every model they release should have a top-notch keyboard because their target market is mostly developers.
fwupd updated my Logitech "Unifying" receiver firmware :-) And Dell uses it to provide updates of some of their laptops and servers.
Ubuntu is the distribution that once sent everything you typed into the desktop search box to Amazon so that it could deliver you ads. Current versions may not do that but it's clear that Canonical prioritizes profit over privacy.
It's disappointing that if you choose not to run Ubuntu you can't take advantage of their firmware update tool.
If we criticize people this strongly for things they've already fixed, how can we expect them to listen to us when we ask them to fix things?
Ubuntu may still be a good OS today and if so you should use it but that doesn't mean their previous actions didn't happen.
> System76 will investigate producing a distro-agnostic command line firmware install tool. Follow us on your preferred social network for updates.
Keep in mind, testing this type of update is pretty important. I'm sure they went through this several times with multiple versions of all their hardware.
Honestly for the sake of not bricking a laptop, I think they should release a bootable USB image for other Linux distros and operating systems. Flash it to a USB stick and now you have a completely stable environment, with the kernel you've tested against, and remove potential unknowns.
Symbol versioning will actually permit the binary compat (unless you have a rather clueless approach), and not break it by some kind of "oh my god I don't know what is this black magic and I would rather not be bothered to learn anything about it even if it is my job to release binaries people can run, so let's just forever rant a little about symbol versioning and pretend GNU systems are hard to target".
Now in order to release a binary which will work on multiple systems, obviously you can't rely on symbol versions that are only present on some of those systems but not all.
If you intent to at least test it on those targets (and well, you... should? ) then obviously you have access to them and obtaining a toolchain on the "oldest" one is trivially an apt-get (or equivalent) away.
Basically, if you want the simplest worryless solution that will work 99.9999% of the time: you build your binary there. Done.
You might even be able to find some compilation flag or something like that if you insist on building on a recent distro, while still targeting old ones. I have not checked; I would if I had to do that.
Bit hypocritical to pan Ubuntu while giving Firefox a pass. At least for Ubuntu there are many alternatives.
This was probably a misguided attempt by Canonical to generate some revenue, they recently had to lay off a significant number of employees. Criticism is valid but dismissing the entire Ubuntu project because of a single bad decision taken and reversed years ago is uncharitable and over the top.
Ubuntu used to do the exact same thing with Amazon, its an affiliate link and does not 'display ads', that is complete FUD. Please understand the issues before commenting.
It's a search scope for amazon so you can get amazon search results in the unity search bar if desired. Firefox has also done a privacy busting ad deal with Cliqz where user's browsing history is sent to cliqz servers for targeted advertising. This is even worse for user privacy.
> System76 will investigate producing a distro-agnostic command line firmware install tool. Follow us on your preferred social network for updates.
For a single software update? With whatever else a normal human has going on in life? There's no way.
This isn't a software update, it's a firmware update. I don't expect updating firmware to be painless; and I'd rather it not be painless. Updating a PROM is a messy process fraught with peril, I want as many checks & balances between my clumsy fingers and that flash as possible.
> With whatever else a normal human has going on in life? There's no way.
Many vendors don't ship ELF binaries update their firmware at all, and yet I still manage to flash their devices. I've lost count of how many times I have to throw FreeDOS a USB drive to flash the RAID controllers I use. Putting Ubuntu on a flash drive to install a firmware update is hardly any more complicated than that. (At a minimum: at least you'll have a reasonably competent shell and functional networking w/ an Ubuntu live image.)
Clumsy fingers won't change the fact that a flash image is read (flashrom -r), modified (mecleaner) then written (flashrom -w)
Running a given distribution or another won't change that process. If anything, I could understand using statically linked binaries.
Of course, you can monkeypatch the VM's bios all you want! But that would be erased the next time you start a new vm.
The only other way is to find a security hole in KVM, qEMU, or XEN, and then exploit it to break out of the VM[1] and get access to base hardware. Hard to do, but it does happen.
My first thought was "Ah ha! AMD CPU!", but they seem to be in on it too. What's a "normal person" to do?
* It might be worth looking into: ARM, Intel Atom, or POWER9 processors
* Intel chips without VPro seem to omit much of the ME functionality, including AMT which provides (all of?) the remote access capability. AFAICT, chips with the Small Business Technology implementation of ME are purposely designed to not have remote access. Maybe use one of those.
* Flash the chip to enable HAP
Soon™: RISC-V, we hope:) Or Talos, or probably a handful of others.
Edit: not to detract from system76! It's excellent that they're doing this too.
It also makes me nervous that they're putting resources into phones when they obviously haven't solved all the issues with non-free components of laptops. It gives me Firefox OS and mobile Ubuntu vibes.
By not using x86 (the board is going to use i.MX8), they remove a whole series of issues that plague modern laptops (no ME, no EMC, etc). They have a very long run-down on why they chose i.MX8, and the freedom status of the components[1].
I'd still like to get one, but there's better stuff around at those prices.
They are little more than repackaged laptops, and not that good in terms of reliability from what I've heard.
[1] http://blog.system76.com/post/159767214983/entering-phase-th...
Their hardware is usually jointly developed with safer/clevo, which sager is free to sell barebones
If system76 goes out, you can be pretty sure that sager Linux compatibility goes out with them (along with many other brands that use the same wifi chips, etc)
Most laptops work out-of-the-box, and there is still Dell.
pick two
The Galago Pro is almost something I'd get, but I really dislike actual buttons on the trackpad: most of them are (rightfully) clickable anywhere on the trackpad.
Are you referring to installing Ubuntu on the WSL, or actually wiping out Win10 and installing Ubuntu natively?
Because to my knowledge, Linux support on the Surface/Pro/Book line is still wobbly at best. I've seen some people running Arch, but problems with battery life and sleep mode abound. I can't imagine the Surface Book's detachable base in tablet mode would be working yet.
According to what I've read on https://www.reddit.com/r/SurfaceLinux/, installing a few packages is enough to get most things working.
I would pay more for a premium Ubuntu support. Unfortunately, System76 doesn't ship to my country.
The process is probably the same for most other vendors, so if your vendor provides a Windows ME firmware updater (no pun intended :P) you might be able to make it work that way.
The system won't boot without it.
It seems to me that using ME as an _actual_ backdoor would be only an occasional thing, (maybe once in a lifetime thing), but it would be so cool to at least know when it is happening and maybe capture some packets.
http://www.amd.com/en-us/innovations/software-technologies/s...
ARM TrustZone
From what I heard the consensus was that it couldn't be turned off.
Sad, because I reeeeally want a System76 laptop.
Just so you don't misunderstand me, I am one of those people, and what Purism and System76 are doing is great.
I just don't think it's going to affect Intel in any way whatsoever.