Linux boot partitions and how to set them up
0pointer.net
0pointer.net
There are serious concerns with SecureBoot and being able to unlock your own bootloader - but UEFI itself is a nice universal base to target in 2022.
But its boot mechanic, and the added crypto stuff (secure / attested boot) is really nice.
Using UKIs is like boot should have worked since the beginning: You just copy a (signed) boot image onto the (firmware managed) boot partition. Done.
https://github.com/zbm-dev/zfsbootmenu/wiki/Portable-ZFSBoot...
I've been on systemd-boot for a long while now. For a while I was just hand maintaining vmlinuzs & loader entries, copying & editing stuff on /boot/efi. Easy but inelegant & I'd forget the very simple steps.
I've been copy pasting (well, its in my ansible now) /etc/kernel/postinst.d/ hooks from stackoverflow, which writes these files, & that greatly simplified life. It's a jank hurdle & I wish my os would actually support this wonderful easy to use tool. Systemd-boot is so much less obtuse, such a breath or fresh air, after years of grub (and many of uboot as well, but that's a different sector).
I made the jump ~two years ago to a single partition, just the ESP partition. Theres a warning abiut not being able to set permissions properly but its worked fine & been so much more pleasant to operate. Very strongly recommend. It just worked for me on Debian, no real fiddling.
https://github.com/filakhtov/kernel-postinst-d/blob/master/9...
Also it would be nice to be able to interact with the boot loader on a modern laptop display without having to get out a magnifying glass. Another problem that is deigned to be the fault of the firmware.
One of the great things about open source operating systems is that people step up to provide these sorts of improvements and I think it's a shame that systemd-boot will cause regression here.
> I think it's a shame that systemd-boot will cause regression here.
If you use grub2, yes you need the magnifying glass.
Meanwhile, I just press F12 and use the BIOS (which selects a readable font) and pick the UEFI entry I want.
My recollection was that GRUB when in graphical mode looks readable. Maybe I'm misremembering. The systemd-boot menu was definitely far too small and there was bug that was basically closed with "it's the firmware's job to give us a usable text output protocol".
Effectively, as shitty as the legacy PC-BIOS boot protocol was, it did at least allow open source software to jump in early and work around the gaping deficiencies in the vendor's firmware. UEFI has effectively replaced all that with a proprietary boot process. The vendor is now in complete control; want a usable text console, or to access the firmware via an external monitor? Vendor doesn't give a shit. Want to access the boot loader via the serial console? Oh you must be an enterprise, let me direct you to our sales staff who won't give you the time of day unless you've got £thousands to drop into their pocket.
Don't get me wrong, I'd love to replace GRUB with systemd-boot, but these are very real impediments to me ever doing so.
Yes you could, if the firmware did let you change the Unicode string (some firmwares let you do crazy things)
> UEFI has effectively replaced all that with a proprietary boot process. The vendor is now in complete control; want a usable text console, or to access the firmware via an external monitor? Vendor doesn't give a shit. Want to access the boot loader via the serial console? Oh you must be an enterprise, let me direct you to our sales staff who won't give you the time of day unless you've got £thousands to drop into their pocket.
You are over dramatizing the current situation. Just boot to a UEFI shell and you can do all that. There are many.
If all you need is to edit the cmdline parameters, add an initrd containing a busybox + kexec to show the currently entered parameters and either keep booting as is, or edit them and kexec the kernel with the new parameters.
It's something a bit like what zfsbootmenu does to let you select the root filesystem you want, and it's much more powerful than a UEFI shell since you get the wide set of drivers linux suppors, and the even wider set of tools you can run on the commandline.
> Don't get me wrong, I'd love to replace GRUB with systemd-boot, but these are very real impediments to me ever doing so.
The #1 impediment seems to be your lack of desire to reclaim freedom, something easy to do if you spend a few hours learning about UEFI payloads, initrd, kexec etc.
Get in touch by email and I'll help you.
/boot (and ESP) management always feels hacky at best.
Or you could put key files in places as yet undescribable. That's a worthwhile advantage for Linux, your choices are less limited.
With legacy BIOS doing the job, it's always been smooth & straightforward to use the DOS and/or NT boot loaders to choose between multiple Windows and Linux installations on different partitions. Even though only one out of a maximum of 4 primary MBR partitions per HDD are capable of being marked bootable at a time, you can still boot from there to as many logical MBR partitions, as you can GPT volumes under UEFI. Either way HDD space is the only limitation, UEFI offers no advantage here; when Microsoft orignally claimed it did they were obviously lying.
As it stands now, the Microsoft UEFI boot loader will still not boot Linux, so it is not yet as advanced as the Microsoft BIOS boot loader was 20 years ago.
Therefore to multiboot Windows and Linux by UEFI, you now need to use a Linux boot loader for Windows, when you never needed to do that before, and that basically means Grub2.
Too bad Syslinux 6.04 was badly fragmented in its misguided rush to attempt UEFI without deep co-operation with the actual Syslinux Project itself. So the various 6.04's are somewhat of a shitshow even though there are some nice features that hopefully will condense into stability someday when UEFI becomes truly well-handled. One can only hope.
Syslinux 6.03 has been highly stable and very feature complete since 2014 and no urgent work should be expected, but that is only for FAT32 BIOS booting not UEFI. Along with Isolinux of course for CD/DVD booting, and Pxelinux for netbooting.
Anyway Grub2 has some forks that will read Syslinux config files if desired. This can be good if you want to stick with the tradition of manually creating and editing your own text config file from better documented references the Syslinux and Grub1 way. Compared to the Grub2 approach which highly discourages touching the config file and autogenerates it occasionally instead in a somewhat mysterious way which is much more poorly documented.
But the stable Grub2 is just fine with its own kind of config file, and now works great on UEFI for multibooting Windows.
Anyway, when Windows 7 first appeared the HDD was still structured with a single NTFS partition spanning the whole HDD, with a boot loader file in the root of the volume along with a BOOT folder containing the BCD, just like Vista. XP has a couple boot files and BOOT.INI in the root with no need for a separate boot folder.
So the boot files and folders were in the same drive volume that the Windows folder was installed to up until that point but this was never a required layout. This was recognized by multibooters since additional Windows installations made to different partitions required no additional boot loaders of their own. Further Windows installations to the same PC simply add an additional boot entry to the default bootmenu in the Windows config file (BOOT.INI or BCD), triggering the menu to display upon startup when there is more than one entry.
A number of months after the first release of Windows 7, the boot files & folder disappeared from the Windows volume and began to be located in a small (often 100MB) hidden FAT32 first partition preceding the following second NTFS partition spanning the remainder of the HDD. Although OEM layouts often had third or fourth partitions somewhat hidden following the main Windows volume for built-in recovery and/or restoration, when you installed Windows from scratch you would not have these low-GB final partitions unless you took the effort yourself.
Or in my case leave some large final partitions for various NTFS, EXTx and FAT32 formatting later.
All you ever do then is add entries to an already functional bootmenu in a hidden volume.
Plain to see. /s
Not enough people saw it, but this was when there was finally way more space on the HDD than Windows needed, leaving plenty for Linux. Or other Windows versions, VMs will only get you so far. Judicious partitioning was as good or better than having a second HDD dedicated to Linux.
PC started out with a Windows bootloader/bootmenu and with MBR/BIOS you can just add enties to boot Linuxes which are installed onto their EXTx partitions if you wanted. Or you could switch to Grub on the hidden volume and accomplish the same thing, or Syslinux since it was FAT32.
But a regular Windows PC would boot to a Linux partition(s) from its original Windows bootmenu, and you had complete manual control of the config file from Windows (or Linux).
Once the UEFI system descended from this structure to depend on a similar but more obscure FAT32 ESP volume containing boot files, preferably the first partition on the HDD, it cemented the sensibilty to continue adding new boot entries to the default structure, rather than restructure, except in very particular situations.
Which is why you... Just use the system boot manager to pick which loader to load. Easy, problem solved.
And you can edit the system boot manager entries from Linux and Windows, set bootnext from either one. It's really strictly superior to what you're describing.
The payload can be managed by your UEFI bios and efibootmgr, allowing more advanced thigs like signing your payloads with your own keys for a true secureboot instead of the hackish "MOK" that's just using unprotected UEFI variables
parted -s "${diskpath}" mklabel gpt
parted -s "${diskpath}" mkpart primary 1MiB 2MiB
parted -s "${diskpath}" set 1 bios_grub on
parted -s "${diskpath}" mkpart primary 2Mib 202MiB
parted -s "${diskpath}" set 2 esp on
sleep 1
mkfs.fat -F 32 -n "boot" "${diskpath}2"
mount "/dev/disk/by-label/boot" "/boot"
grub-install --target=i386-pc "${diskpath}"
grub-install --target=x86_64-efi --efi-directory=/boot --removable --no-uefi-secure-boot
grub-mkconfig > "/boot/grub/grub.cfg"
(This is part of a larger Debian imaging script I made)
Shell redirection is fine, but you could just use `grub-mkconfig -o` to set the output file.
Using -o makes failure nondestructive.
Or the program could start out by deleting my file and starting to line buffer write to it as it executes. Then the failure would be the same as shell redirection. Or it could buffer differently making program failure even more fun.
My point is that I don’t know what the program will do, or even what this particular version will do, without looking at its code.
Shell redirection has predictable semantics. And I can output to a different file, run diff on the two, then rename myself, if this is truly critical. And muscle memory will be faster than looking up the specific option or argument for this specific program.
Why do you still need the legacy BIOS boot logic?
I might understand doing that just in case when preparing a bootable flash drive. Why complicate things for permanent installations where you know your current hardware and your next system after 5 years is unlikely to much worse than current one?
Most of the systems I have ran into were in SCADA[0], PMS[1], and BMS[2].
[0]:https://en.wikipedia.org/wiki/SCADA
[1]:https://en.wikipedia.org/wiki/Power_management_system
[2]:https://en.wikipedia.org/wiki/Building_management_system
There's a whole other problem, though, of my wanting to use secure boot. Ubuntu was the easy out for that. At the moment, it's just disabled on my machines, which is far from optimal.
I have a setup that's almost like what Poettering proposes. Only that I don't use any external bootloader as it seems completely unnecessary.
All you need for secure boot in such a setup is signing the UKI with your keys.
That's the simplest boot setup ever!
It's only one file, so no moving parts. It just works. No LiLo and config, no grub and config, not systemd-boot, no nothing. Just the signed UKI on the EFI partition, and a efibootmgr entry pointing to that single file. That's all needed to boot a modern system.
parted -s "${diskpath}" mklabel gpt
parted -s "${diskpath}" mkpart primary 1MiB 2MiB
You can use a descriptive string here instead of "primary" (e.g. "boot partition", or "esp partition").
https://manpages.debian.org/unstable/parted/parted.8.en.html...
#!/bin/bash
set -ueo pipefail
# Remount EFI partition read/write and restore to readonly when done
trap 'mount /sys/firmware/efi/efivars/ -o ro,remount &>/dev/null || true' EXIT
mount /sys/firmware/efi/efivars/ -o rw,remount &>/dev/null || true
# Remove all existing Arch Linux entries
efibootmgr | grep 'Arch Linux' | grep -Po 'Boot\K\d+' | while read -r bn; do
efibootmgr --delete-bootnum -b "$bn" &> /dev/null
done || true
# Install boot entry
efibootmgr --verbose \
--create --disk /dev/disk/by-id/nvme-abcdef --part 1 --label "Arch Linux" \
--loader /vmlinuz-${_linux} \
--unicode "initrd=\\intel-ucode.img initrd=\\initramfs-linux.img OTHER-KERNEL-BOOT-PARAMS"
$ stub_line=$(objdump -h "/usr/lib/systemd/boot/efi/linuxx64.efi.stub" | tail -2 | head -1)
$ stub_size=0x$(echo "$stub_line" | awk '{print $3}')
$ stub_offs=0x$(echo "$stub_line" | awk '{print $4}')
$ osrel_offs=$((stub_size + stub_offs))
$ cmdline_offs=$((osrel_offs + $(stat -c%s "/usr/lib/os-release")))
$ splash_offs=$((cmdline_offs + $(stat -c%s "/etc/kernel/cmdline")))
$ linux_offs=$((splash_offs + $(stat -c%s "/usr/share/systemd/bootctl/splash-arch.bmp")))
$ initrd_offs=$((linux_offs + $(stat -c%s "vmlinuz-file")))
$ objcopy \
--add-section .osrel="/usr/lib/os-release" --change-section-vma .osrel=$(printf 0x%x $osrel_offs) \
--add-section .cmdline="/etc/kernel/cmdline" \
--change-section-vma .cmdline=$(printf 0x%x $cmdline_offs) \
--add-section .splash="/usr/share/systemd/bootctl/splash-arch.bmp" \
--change-section-vma .splash=$(printf 0x%x $splash_offs) \
--add-section .linux="vmlinuz-file" \
--change-section-vma .linux=$(printf 0x%x $linux_offs) \
--add-section .initrd="initrd-file" \
--change-section-vma .initrd=$(printf 0x%x $initrd_offs) \
"/usr/lib/systemd/boot/efi/linuxx64.efi.stub" "linux.efi"
Also a secure boot setup is much more difficult this way.
I for my part love the UKI. Never had a simpler boot setup!
Is it? Don't you just sign the bootable kernel image that already has the initrd and command-line built in?
Oh, I guess if you're using Microsoft as a CA I can see why that would be tricky.
In case of a UKI it's very simple of course. Just sign the boot image.
That's why I love the UKI. :-)
Or Microsoft is trying to make dual booting easier without using the simplest solution of making a larger ESP the default.
I see no problems with secure boot from a freedom perspective as long as the owner of the computer can install their own trusted public keys. There might be industry players that would like to remove that possibility. But why Poettering's work would make that easier I fail to see.
Do you think once a real secure Linux boot is possible they will remove machine owner rights and you have to buy a signed Linux from Microsoft? Or from a Microsoft/IBM/? consortium super monopoly?
It was not, ironically because of Microsoft. Shim is by policy effectively only allowed to boot grub2. So systemd-boot can't be used OOTB on secure boot enabled systems, you'd have to enroll your own key.
It has been possible for years. I have used it for 4 years myself and I was not the first adopter.
> Shim is by policy
I am not talking about shim, you don't have to use it.
Here's how to do it:
1. You erase the whole key database from UEFI (That possibility was some agreement between antitrust authorities and Intel and/or Microsoft that such possibility must be provided because Wintel was in a a dominating marketing position. On Arm devices it is often not possible, Windows / ARM is not in a dominating position and antitrust was not relevant.)
2. You generate your own key pairs.
3. The public ones you install into the secure boot database of UEFI.
4. You sign your UEFI application with your private key.
I have done that with `systemd-boot` and with the Linux kernel (containing the UEFI stub). Works in both cases. I used the instructions from https://wiki.gentoo.org/wiki/User:Sakaki/Sakaki%27s_EFI_Inst... 4 years ago and still do it the same way today.
Deleting the key database can not be done programmatically (unless UEFI has a bug). In all machines I have looked it's an option in the BIOS. So that should be clear case of warranty repair. Which of course does not help you if you do it after warranty has ended.
It's an option in the UEFI.
I guess it remains unclear whether it was a firmware bug that has since been corrected or whether it depends on how exactly the user installs their own keys.
The reply the UEFI itself would be signed and if you delete the matching keys from the relevant DB UEFI would no longer start does not sound right to me.
Good the see that the option exists for AMD, too. I guess AMD had no dominating market share when secure boot was introduced. So they would probably not be legally obliged to provide it? Hopefully market power of those requiring independence of Microsoft is big enough to keep it that way.
I can't exactly figure out how they'd use that level of control over the Linux boot, but I'm not a huge fan of them having it.
The better that Linux conforms to this range of layout structures, the more overall effectiveness you will have when multibooting, especially the massive multibooting possibilities unique to the UEFI boot method.
You're not going to figure this out on only one manufacturer's hardware, and not over a single-year period.
100MB boot partition is not enough, 500MB either. My first FAT32 partition is always the full 32GB that regular FAT32 is made for, but it's the same HDD structure. I've got plenty of space.
In UEFI I like to put my Linux kernels all on the FAT32 volume where I can handle them along with my Grub files entirely from Windows if I want. I carefully engage the Grub menu autoconfig so my custom bootmenu containing my up-to-date kernel choices is not lost or mangled.
Fundamentally in different Windows versions the particular Windows kernels are stored on the different partitions, and the bootloader just picks a partition. With Linux the different distributions are installed to their own separate EXT partitions, but are not strictly tied to the exact kernel they were issued with. When you update the kernel you can just copy it to the FAT32 volume, then put it on the bootmenu to replace the previous one, or as an alternative. Whether or not you want to update the entire remainder of any distro.
Each Grub bootmenu entry specifies which kernel will be loaded first from which folder on which partition, followed by which full distribution will be loaded next from the same or a different partition.
But this is the forbidden ESP partition, U may not want to go there but EFI already is.
To tame the beast you're going to have to change its identity (temporarily) so you can edit it willy-nilly.
Change or edit its partition GUID designation from ESP to Basic and a proper mainboard UEFI will still reboot by finding the EFI folder on the FAT32 volume as before, but in Windows you will now be able to assign a letter volume to the ordinary FAT32 partition and see the EFI folder there.
If it doesn't boot EFI without the strict ESP GUID in place, you would have to edit it "offline" from externally booted media.
As a very basic bare-metal layout all you need on a FAT32 volume is an EFI folder containing a BOOT folder (EFI\BOOT) containing a bootx64.efi file and it's off to the races.
But bootx64.efi's are squirrely beasts.
It's almost like no two are alike, god I hope these are real code and not generated.
Each OS install can have its own predictable way of overwriting the previous bootx64.efi with its own, sometimes more predictable than others.
A Windows bootx64.efi will then look for an \EFI\Microsoft folder, drill down to EFI\Microsoft\Boot\BCD and run the Windows bootloader
A Ubuntu bootx64.efi will look for \EFI\ubuntu and run the Grub found there
Each OS installed adds its own specific \EFI\XXX folder to the ESP and they are all independent.
But by convention the latest installed OS takes over as the bootloader going forward, simply by overwriting bootx64.efi with its own, and you may need to rely on an autodetect process to update your new bootmenu in its corresponding \EFI\XXX folder, to list all OS's that were previously installed to their partitions.
Otherwise you may need to fill out the config file bootmenu entries custom or manually, or switch back to one of the previous bootloaders which update Grub to include additional distros more ideally.
You will see that different distros sometimes have Grub fully in the ESP, which can include the kernels or not, and others where a good number of the Grub files are on their EXT volume, but mounted interestingly after you have booted. When kernels are only present by default on the EXT volume, I leave them there but place a copy into the working ESP folder(s) additionally.
>Boot Partition Discovery
>The traditional boot partition was not recognizable by looking just at the partition table. On MBR systems it was directly referenced from the boot sector of the disk
Actually not so. The partition table is only the final few bytes of sector 0, and whichever of the 4 possible primary partitions is currently marked for bootability has always been clear to see. It will be the one with the 80h, all others are supposed to have 00 as a boot flag.
Therefore you can (I advise) have duplicate or alternative boot files on each primary legacy partiton, each of which can boot its own OS, or boot to any of the other primary or logical volumes from the flagged partition's own unique (or identical) bootmenu. You choose which primary partition's boot files will be "Active" by placement of the boot flag, and it reboots accordingly. All other boot files remain dormant on their non-flagged partitions.
Except for the partition table at the end, the bulk of sector 0 is the Master Boot Record itself for that HDD, which is the code that runs to find a primary partiton marked Active. Control then transfers to the active partition's first sector which is supposed to be a Volume Boot Record commonly known as a boot sector.
A DOS bootsector then seeks IO.SYS on that volume in a filesystem it understands, an NT5 bootsector would seek NTLDR, and an NT6 bootsector seeks BOOTMGR from there before parsing any multiboot menus contained in config.sys, boot.ini, or BCD, respectively.
In Linux a Syslinux bootsector seeks ldlinux.sys, Grub seeks grub.cfg.
>ESP can be auto-discovered from the partition table, traditional boot partition cannot.
Nope, not true, this is the same misconception as above.
Over the years, I've been pleased to see that more and more distributions are writing their disk images and the like to the ESP. (Previously, dd'd USB images for distro installing _required_ the creation of a /boot partition)
The logical next step would be to standardize everything through systemd, and ensure all boot images are autodiscoverable and automatically bootable.
It's been somewhat frustrating for distributions to install GRUB, hijacking the previous prioritized boot PE, and have entries for other installed Linux distributions missing.
See systemd-boot and BootLoaderSpec, both mentioned in OP.
https://www.freedesktop.org/wiki/Software/systemd/systemd-bo... https://systemd.io/BOOT_LOADER_SPECIFICATION/
Same. The boot partition is a relic of the past, and you can easily do without it when you remove other relics like grub2: simply make EFI payloads the Arch way: https://wiki.archlinux.org/title/Unified_kernel_image#Manual...
> The logical next step would be to standardize everything through systemd, and ensure all boot images are autodiscoverable and automatically bootable.
As much as I like systemd, I think it's not necessary here: I have a EFISP that's about 16G: it also contains a few .ISO that can be loaded in case system maintenance or a full reinstall is needed. GrubFM and others like Ventoy let you boot directly on say Windows11-22H2.ISO, Ubuntu22-10.iso etc.
When I buy a new drive or computer, I just copy this partition: it's much faster than playing with thumbdrives or PXE Boot.
A second issue is that since FAT doesn't have permissions any user ID on the system can potentially mount, read, and possibly write to the ESP.
A separate (possibly encrypted) /boot/ file-system can be hosted on a RAID device or even loaded over the network.
With a 'regular' boot process (UEFI > ESP boot-loader core > LUKS unlock > /boot/ boot-loader 2nd stage > /boot/ kernel+initrd > root-fs) the only thing lost is the boot-loader core, which can quickly be switched for a USB boot, set options appropriately to find and use /boot/, and for the boot process to use alternate RAID device(s) and continue with a degraded boot.
On UEFI most distros now do not use os-prober to populate GRUB's own menu with other bootable OSes (GRUB_DISABLE_OS_PROBER=yes) since it expects those OSes to have added themselves to the UEFI boot menu.
As for replacing the default boot entry, this is caused by efibootmgr not GRUB. GRUB calls efibootmgr with "--create|-c" which adds a new BOOTNUM and puts it at the head of the list.
GRUB could be taught a new option that then does several efibootmgr operations: parsing text output of "efibootmgr" to get current boot order, --delete-bootorder then --bootorder A,B,C,OS to ensure OS is added last.
That is obviously fragile so best approach would be to teach efibootmgr a new option --create-append|-C that adds the new entry to the end of BootOrder and then simply teach GRUB a new variable GRUB_UEFI_BOOTORDER_FIRST=yes|no to use efibootmgr --create-append.
It is possible to make ESP part of MDRAID array of level 1 with metadata v1. True, the boot starts with single disk and only later Linux activates the array. So if your boot fails before that happens, reboot and try the other disk.
Or alternatively, don't set up MDRAID array for ESP, just have the ESP mirrored on all disks.
Isn't that the entire point of utilising MD RAID-1 in the first place?
I've done as you suggest in years gone by with either metadata v0.9 or v1.0 (not v1.1 or v1.2) where the metadata is not at the start of the block device and therefore does not confuse a non-MDRAID-aware firmware or other OS.
Then why did you wrote
> it is not possible to make it part of a software RAID device
You are saying it is both not possible and that you've done it.
It requires manual intervention of a knowledgable person to boot a UEFI system into the default OS if the ESP is broken or missing.
You could substitute Boot for Root in this sentence and flip it around on Poettering.
See https://github.com/AsahiLinux/docs/wiki/SW%3ABoot for example.
The unification of the operating systems for both devices continues with every macOS release. (And that's not only about the unification of the naming scheme…)
Instead, I would prefer a different approach. An approach that can be seen in Windows and macOS, that is: no user serviceable parts inside. It would work like this:
* Keep two partitions.
* ESP contains a simple program (let’s call it stub), whose only job is to call the real boot loader on /boot.
* The stub is simple (minimum user interaction) and doesn’t need updating very often. It has drivers/support software for storage media and some reasonable file systems (VFAT, Ext4). It may also support some simple form of disk encryption if desired.
* The real bootloader (as well as any kernels) live on the separate /boot partition. The bootloader can do all the fancy things it wants, it can display a fancy wallpaper, support mouse input, and so on.
* The ESP is not auto-mounted, might not even be listed in /etc/fstab.
* Whenever the stub is updated (which would happen rarely, since it’s meant to be simple and minimum), some post-install scripts would mount ESP in any location they please (be it /run/{uuid.uuid4()}), copy over the new stub, and immediately unmount it.
Simpler, safer, will make it harder for rogue software or `rm -rf /*` to mess up the booting of the system, and will not require any changes to existing partition tables.
Than this "init Linux" would boot the bootloader, which in turn would load another kernel and initrd?
This looks way to complicated.
How does secure boot work in this setup?
It's much simpler to just "glue together" a kernel and initrd, put it on the EFI partition, and boot that UKI directly by the built in EFI boot-manager.
The EFI partition needs in this setup also be only mounted when updating / adding a UKI. Otherwise it never needs to show up in the running system as it's only used by the EFI boot-manager. Preparing a UKI can be done on the root FS (in its local /boot dir).
I usually have my storage drivers compiled into kernel, not as modules (because why have them as modules if you need them always).
* need to have sshd at initrd stage to enter encryption keys (remote boot)
* need to have recovery tool on hands (going even further: imagine having working graphical browser when storage fails)
* have same crazy storage setup - root on ceph/nfs volume with non trial configuration and etc...
* in case of electricity fault at home - it only system which have exposed connection to internet
My suggestion for booting would be following:
1) having UEFI bootloader which boot some minimal kernel (lets same sort LTS for stability’s sake ) on /boot partition
2) have ability to unlock and read from root filesystem (same as above, rootfs in nfs/ceph/encrypted ir whatever)
3) then scan kernels from /lib/modules/{version}/bzImage (i know there are modules here, but having one additional file would not hurt too much)
4a) kexec kernel with necessary parameters to start system [in this step we do not enter passwords, encryption keys twice!]
4b) or boot without kexec in recovery mode with some GUI (I want ability to have browser in recovery mode)
Or are you trying to say that you're booting without a initrd (which is said to be in general broken for decades).
Also the secure boot question remains.
I still don't get why complicate things in such way.
Imho nothing is simpler than copying a (signed) boot-image onto the (firmware manged) boot partition, and be done.
Yes I'm booting without initrd, because in my case I don't need it - no cryptsetup and I don't need optional drivers because my setup doesn't change the hardware. (e.g. general distro kernel has to have it because they add modules for every hardware there is to support each user).
How is boot without initrd broken? I use it since my first kernel compilation and find it simpler than dealing with initrd.
I have actually almost such a setup since some time.
The only thing I don't understand: Why add something like `systemd-boot` to the setup? It's completely unnecessary! All you need is an UKI on the EFI partition. UEFI has a perfectly sufficient bootloader already. I never found out what additional advantages `systemd-boot` would offer.
Maybe someone could clarify?
Another useful feature would be editing kernel parameters for troubleshooting. Not sure if systemd-boot can do that but GRUB can.
It gives you a UI to choose what to boot and edit the kernel bootline. You don't need it if your UEFI firmware makes it easy for you to do that, or if you have edk2-shell available, but those are not true of all systems.
Editing the kernel command line on the other hand is something you never need except for debugging or recovery. For that you would have anyway an extra UKI installed, with a recovery system in the initrd.
But in normal operation you never ever even see the boot menu.
I still miss to see what vital advantages `systemd-boot` would offer. (And this has nothing to do with systemd as such. I use most of its modules. I just didn't find any compelling reason to use the boot module also. Some very simple hook script that triggers efibootmgr when needed is imho perfectly sufficient).
There is no "the UEFI bootloader" and there is no "universal" boot menu. The firmware only has to boot EFI applications according to the EFI vars.
Who builds PCs without that feature?
Edit: I've had a look at the spec. Indeed a boot menu isn't mandatory (except when it is, for network boot options, see `PXE_BOOT_MENU`).
> UEFI specifies only the NVRAM variables used in selecting boot options. UEFI leaves the implementation of the menu system as value added implementation space.
But it seems to be common to implement this, as the spec also says:
> If the boot via Boot#### returns with a status of EFI_SUCCESS, platform firmware supports boot manager menu, and if firmware is configured to boot in an interactive mode, the boot manager will stop processing the BootOrder variable and present a boot manager menu to the user.
There are at least a dozen more references to a boot menu in the spec.
IIRC, older bootloaders like LILO used a simpler approach: after each kernel update, a userspace program asked the kernel for the list of sectors which contained the kernel file, and wrote the list to a map file; the bootloader then read that map file (its sector hardcoded into the bootloader by the same userspace program), and loaded the kernel by reading the sectors directly. Neither the bootloader nor its userspace installer needed to know anything about filesystems or other parts of the storage stack, and it worked perfectly with RAID 1.
Ah, that stirs memories. That 1,024 something boundary the kernel image had to reside in, hence the need for a separate /boot partition. And every time there was a new kernel available, I had to re-run some command to recreate that map file.
Things have become a lot more convenient since then.
And more complex at the same time. Now I need to do a silly dance to register a GUID with my BIOS so it can find an executable and run it for me (you did remember to add efivars to your kernel, right?), hold a vFAT partition around specifically for this case (what package has mkvfat again?), and worry about how the system maintainers will decide to "improve" what they think is wrong with this hierarchy.
Previously.. the BIOS would find a particular sector on what I told it was the "boot drive" and run it without too much concern over what, if anything, happened next.
And for this change, do I get a BIOS that's more helpful when there's a frustrating boot misconfiguration? A log to show me what precisely went wrong? The ability to save debug information in any relevant format? I could, but zero vendors in the consumer space are going to do that.
Now.. essentially, I just don't have to specify a "boot drive" anymore. Other than that, if you're bringing a system up from scratch, EFI has just as much "janky magic" in it that the old boot sector method did.
I was just speaking of LILO vs grub, though. (I heard that grub is not pretty internally, but as a mere user, it has not given me trouble.)
Am I missing something?
The "Reserved" partition on Windows machines isn't really a boot partition, for any meaningful definition of it. It's just … reserved, and MS being MS. On my machine, it's empty (unformatted, all 0s). It is lightly documented here: https://learn.microsoft.com/en-us/windows-hardware/manufactu...
(I'd expect your typical GPT Windows install to have about four partitions: the ESP, the empty "reserved" partition, a recovery partition, and the main NTFS partition.)
IIRC, that "reserved" partition is to allow converting the data partition which follows it to a "dynamic disk" (which AFAIK, is Microsoft's equivalent to a Linux LVM PV). That conversion needs to grow the partition backwards to prepend some headers, and that extra space comes from shrinking the reserved partition just before it.
Details on whether reserved partition would be created and what would be on it depend on hardware you're installing on, and if separate boot partition is necessary windows installer would inform you about need to create an extra partition.
Two partitions are needed only for Bitlocker; one has to be unencrypted.
Similarly with Apple: they use APFS subvolume for boot files. They do not bother with multiple partitions and static allocations, guessing, what size is going to be OK. They can use as much or as little space as they need.
--
With Linux, I've been using btrfs subvolume for /boot. It works with "normal" distributions, grub complains (it cannot write there; I find that OK). The dynamic nature of the space used is great. It doesn't work with ostree-based distributions (Fedora Silverblue & its ilk); ostree cannot generate proper BLS and grub.cfg for subvolumes.
For the record, and showing again the unfairness of the entire MS monopoly situation, most commercial UEFI implementations out there happen to understand NTFS. This allows e.g. a Windows pendrive to boot no matter how the user formats it.
1. Find physical disks that may be a part of a managed volume group/pool 2. Find logical volumes and file systems on the volume pools 3. Mount a filesystem by some configurable logic 4. Load and execute a kernel from the filesystem
Having to allocate the first "blocks" on the imaginary "sectors" of my SSD (or even worse, a virtual disk drive) for some arbitrary amount of space formatted in possibly the most barebones filesystem still in mainstream use feels quaint and irritating and limits my ability to use that disk in a larger storage pool.
UEFI is an overcomplicated specification with lots of wintel baggage, but most of it doesn't personally offend me. What does is that UEFI had the chance to abolish disk partitioning, but instead enshrined it. And added mandatory FAT32 to add insult to injury.
My main laptop and desktop each have a separate disk for the EFI system partition. The former uses systemd-boot and the latter ZFSBootMenu. This way I have a maximum of one partition per disk. It's not ideal, but I like it better than the usual solution.
The disks in my zpool show up as having partitions 1 and 9, but I consider that an implementation detail since I never need to treat the disks as anything other than entire disks in a pool
I format directly the raw unpartitioned SSD/HDD with a file system that uses 100% of the capacity, with no wasted sectors. At least on Linux and FreeBSD, there is no need of partitions.
For booting the computers, I either boot them from Ethernet or I boot them from a small USB memory that uses a FAT file system for storing the OS kernel, either in the format required by UEFI booting, or, when booting Linux in legacy BIOS mode, together with syslinux, which loads the kernel.
> For booting the computers, I either boot them from Ethernet or I boot them from a small USB memory that uses a FAT file system for storing the OS kernel, either in the format required by UEFI booting, or, when booting Linux in legacy BIOS mode, together with syslinux, which loads the kernel.
Creating boot USB drives (which I think need partitions don't they?) or setting up a PXE boot server would take me a lot more effort than an extra minute with gdisk to create partitions before formatting the disk.
For booting with UEFI, you just need to create the directories with the names expected by the firmware. For legacy booting, you just need to install syslinux, which takes a second.
Then the USB drive can be used to boot any computer, without any other work, for many years.
When you change the kernel, you just mount the USB drive (which is not mounted otherwise), then you copy the new kernel to the USB drive (possibly together with an initrd file), renaming it during the copy, you unmount the USB drive and that is all.
You can keep around a few USB drives with different kernel versions, and if an update does not go well, you just replace the USB drive with one having an older version.
Configuring a DHCP/TFTP server for Ethernet booting is done only once.
Adding extra computers may need a directory copy in the directory of the TFTP server only when the new computers have a different hardware that requires different OS kernels.
Updating a kernel requires just a file copy towards the directory of the TFTP server, replacing the old kernel.
None of these operations requires more work than when using a boot partition on the root device.
There is less work because you make booting USB drives or a DHCP/TFTP server only once for many years or even decades, while you need to partition the SSD/HDD whenever you buy a new one that will be used as the root device.
I partition a new computer once every few years. I upgrade the kernel a few times a month. With the normal way that's a simple `pacman -Syu` or `apt get dist-upgrade` and it's handled, no mounting thumb drives or sftp needed.
That is more work for everybody except if they wanted a) completely encrypted main disk and booting from portable guarded USB, or b) set of computers in school or internet cafe with boot process managed/updated efficiently via network.
This system does provide new special capabilities, but it is not for free. Meanwhile, most users are happy with defaults like 1G partition with bootloader and kernel, which allows easy updates without worrying about having the right USB drive, being in the right port, being mounted in the right path, or losing it.
If it it works for you great but that is a LOT of extra work to regain less than 1% of the storage on a drive.
I am glad this setup works for you, but many people will not want to need a USB drive to boot their desktop, laptop, tablet, phone, et cetera.
Using a single disk with multiple partitions is less convenient than using a separate boot drive, because the separation makes easier the reuse of both the boot drives and of the root drives in other computers, or their copying onto drives of different sizes, when migrating or cloning operating systems.
For a laptop that contains encrypted SSDs/HDDs, using an USB drive to boot it, which is not normally kept with the laptop, can be used for improved security, because only in this case no secret keys need to be stored on the encrypted drive. Even if the secret keys are also encrypted, they must be encrypted with a key derived from a password, which can make their decryption much easier than the decryption of a drive encrypted with a random key.
That certainly works, but I'm pretty sure that moving booting off of your main disk is the only reason you can go without partitions, and I'm also pretty sure that most people don't want to deal with that.
> 100% of the capacity, with no wasted sectors.
You will never have that. SSDs have a large amount of reserved space, and even on HDDs, there are some reserved tracks for defect management.
Obviously, I do not use unpartitioned SSDs/HDDs with Windows. On the other hand, With Linux and *BSD systems they work perfectly fine, regardless whether they are internal or removable.
For interchange with Windows, I use only USB drives or SSDs that are partitioned and formatted as exFAT. On the unpartitioned SSDs/HDDs I use file systems like XFS, UFS or ZFS, which could not be used with Windows anyway.
Any SSD/HDD that uses non-Windows file systems should never be inserted in a Windows computer, even when it is partitioned. When a SSD/HDD is partitioned, it may be hoped that Windows will not alter a partition marked as type 0x83 (Linux), but Windows might still destroy the partition table and the boot sector of a FAT partition. It happens frequently that a bootable Linux USB drive is damaged when it is inserted in a Windows computer, so the boot loader must be reinstalled. So partitioning an USB drive or SSD does not protect them from Windows.
>> 100% of the capacity, with no wasted sectors. > You will never have that.
I thought that it is obvious that I have meant 100% of the capacity available for users, because there is no way to access the extra storage used by the drive controller and also no reason to want to access that, because it has a different purpose than storing user data, so your "correction" is pointless.
Efi+boot partitions usually take less than 2G of space, and can be made like 200MBs total, while mainstream disk capacity is hundreds of GBs nowadays.
This "loss of useful space" is immaterial in most cases. Maybe if you have something like a 2GB drive from 1990s that you want to use (why?) then it makes sense to shave off 1G off that. But it is more work, as you have to buy, prepare and manage the USB drive.
Think of an expensive, super fast but considerably small SSD, and some cheap big mass storage (maybe even on spinning rust) along that.
You'll likely try to use the expensive SSD as efficiently as possible. Every GB counts if you have "only", say, 0.5 TB.
A boot partition on such expensive and small (but fast) media is pure wastage.
Also this kind of setup seems not so uncommon, as I can claim that I've done something similar. :-)
There are even more reasons. It makes things even more simple and less error prone:
The argument that you can swap disks more easy was mentioned already. But that's not everything one gains.
SSDs are very prone to get worn out much quicker and loose at least half of their performance when you mess up the data alignment on them. In case of FDE with partitions (maybe on top of LVM even) the alignment issues isn't trivial. It's quite easy to mess up the alignment by mistake. You can read a lot of docs, try to find out details about the chips on your SSD, do calculations, yada yada, or you just encrypt the raw device and use the whole disk without partitions. That's considerably simpler, nothing can go wrong.
It doesn't. You can make the overhead partition take 200MB. That's immaterial fraction of 0.5TB. You ain't gonna see impact of this loss. Additionally, by partitioning the drive, you protect it from dumb programs who like to create partition tables.
Yes, there are reasons for not partitioning your OS disk, like full disk encryption. But it is more work.
> when you mess up the data alignment on them. In case of FDE with partitions (maybe on top of LVM even) the alignment issues isn't trivial.
This sounds interesting. What are these alignment issues? Why do you think they are present on disk with partitions (I never had those issues) and why do you think they are not present on disk without partitions (may be they are, due to compression/encryption)?
Alignment issues are only really relevant in case of SSDs. The FS blocks need to align with the "physical" blocks of the chips used. (Actually this are also "only logical blocks", presented to you by the SSD controller, but at least this is fully transparent). If the alignment is messed up the SSD needs to consider at least 2 "physical" blocks (as presented by the controller to the OS) when accessing a single FS block. This leads to doubling the wear and halves the performance. (At least, in really unhappy scenarios this can even triple the access effort).
Where exactly a FS block starts and ends in relation to the underlying "physical" block(s) depends on all the "headers" that are "in front" of the FS blocks (or logically sometimes "a layer up", even "physically" this also only means "in front"). Partition tables are headers. LUKS headers are obviously also headers that need to be taken into account. LVM headers (and blocks, groups, volumes) are even one more layer to consider.
To make things more fun, like said, the "physical" blocks are only an abstraction presented by the controller. In some cases their size is configurable through the SDD controller firmware. (But this shouldn't be done without looking at the chips themself). The more interesting part is: The "physical" blocks can have "funny" sizes… (Something with some factor of 3 for example). Documentation on this is frankly spare…
The usual tools just assume some values that "work most of the time". But this whole problematic is actually quite new. Older version of all the related tools didn't know anything about SSD block alignment. (Like I said, they still don't know anything for sure, there is not way to know without looking a the docs and specs of the concrete device, but now at least they try to guess some "safe values"; with a large margin).
If you use partitions you'll end up with those "funny" few MiBs large offsets, which you have seen for sure. (If you don't use offsets it's very likely that the alignment is wrong).
Without partitions the other storage layers are much easier to align. You don't need to waste a few MiBs around your partitions, and especially don't need to remember (and maybe even recalculate) this stuff when changing something.
Not many people know about this whole dance as misalignment isn't a fatal problem. It will just kill your SSD much quicker, and half the performance (at least). But SSDs are so fast that most people would not notice without doing benchmarks… (Benchmarks of the different storage layers is actually the only way to test whether you got the alignment right).
If you don't look into this yourself you can only pray that all tools used were aware of this issues and guessed some values that work by chance properly with your hardware. But if you created partitions without the "safe" offsets (usually by setting values yourself and not letting the tool chose its "best guess") the alignment is quite likely wrong.
I'm came across this issue because I was wondering why Windows' fdisk always added seemingly "random" offsets around partitions it created. It turns out it's a safety measure. Newer Unix tools will do the same when using proposed defaults.
TL;DR: If you don't create a partition table on a NVM device you can just start your block layer directly on block zero and don't have to care about much as long as you also set the logical block size of that layer to the exact same value as the (probably firmware configurable) "physical" block size of the device. If you have a (GPT) partition table in front (which is by the way of varying size to make things even more funny) you need to add "safety offsets" to your partitions. Otherwise you're torturing your NVM device, resulting in servery crippled performance and lifetime.
I hope further details are now easy to google in case anybody likes to know more about this issue.
---
> Additionally, by partitioning the drive, you protect it from dumb programs who like to create partition tables.
The better protection would be do keep drives far away form operating systems and their tools that are known to randomly shred data… ;-)
I've written this to shed light on the alignment issue as I was under the impression that this would be be something completely new to you. ("This sounds interesting. What are these alignment issues?")
> Is there any documented case where physical blocks have size that in bytes is not some power of 2?
Yes, there are examples online. I did not make this up!
It was in fact some major WTF as I came across it…
> I suspect if that exists, it is quite a rare device.
Jop, that for sure.
Also the documentation is very spare on this, like already mentioned.
I think it was the early triple-cell chips that had such crazy layouts. (Did not look it up again; maybe this was only a temporary quirk; but maybe it still exists, no clue).
> Blocks of size 512B, 4K, 8K are the most common case, and correct alignment is completely taken care of by the 1MiB offset which is standard and default in fdisk and similar tools on Linux.
Well, it depends.
This thingies I've read about with some factor of 3 in their block size would need at least a 1.5 MiB offset… (And the default 1 MiB offset would torture them to a quicker death; but most people would likely never find out).
There are devices with much bigger (optimal) block sizes, I think in the MiB ballpark (don't remember the details out of the top of my head, would need to look it up again myself). Also in such cases the 1 MiB would not suffice.
Those devices are usually in some compatibility mode in factory settings, with much smaller blocks than optimal for maximal performance and least wear. You need to tell the firmware explicitly to switch the block size to get best results (which is of course not possible after the fact as it obviously shreds all data on the device).
Also it's not only the offset around the partitions. You need to take the block sizes into account also regarding the block layers "inside" the partitions. Which was actually my point: This makes things more complicated than strictly needed.
> You mention "random" offsets with newer Unix tools - I have never encountered this. Any examples?
By "random" I've meant that the offsets appear seemingly random when you don't know the underlying issue. It's not only the one offset after the partition table. Depending how large the partitions are there may be or may not be additional offsets around the partitions themself.
Of course all this is not rocket science. We're talking about simple calculations. But that's just one more thing to be aware of.
My conclusion form that journey back than was: Just screw it, and don't add partitions to the mix if you don't strictly need them. One thing less to care about!
For example the laptop I'm just writing on has two NVM devices. The bigger and faster one is used as (encrypted) root FS, without any partitions on it, the other smaller and slower one carries the EFI partition and an (encrypted) data partition. If I would have have partitions on the root disk this would not give me any advantages, but additional stuff to think about. So why should I do that? OTOH I need an EFI partition to boot. So I have created one on the other disk. I think this is a very pragmatic solution. Just don't add anything that you don't need. <insert Saint-Exupéry quote about perfection here>
For the system disk I have always partitioned it though. I generally create at least /, /var, /home, and /usr. That way it's less likely that a runaway process can fill up the entire disk, at worst it might fill up /home or /var.
And unless I'm really space-constrained, I'll leave some unpartitioned space as well, for later flexibility.
And if you go that far, you can throw out any FS kmod from the initrd except for what you need for your root partition. Including vfat.
It makes all the sense if you want the firmware to load your bootloader binary and then get lost.
Firmware is not to be trusted to read, write, interpret, analyze or send your boot file system elsewhere. It's a closed proprietary part of your hardware ffs and unless that changes, it should be kept in dark.
When there is hardware with free software firmware we can fix, then we can talk about accomodating firmware vendor needs.
TIL! My NixOS configuration just got a little bit simpler, and more uniform between machines.
Back in the day Netware would boot off a small DOS partition located at the front of the boot disk, and was started from autoexec.bat. Blew my mind at the time.
In the real world, my ESP and boot partition are trusted, since I've installed them and control them and can check them for malware, and the firmware is not trusted since there is no possibility of control, it's vendor blobs.
Mr. Poettering is clearly arguing for interests of the secure boot/DRM complex, not competent owners/users.
As far as your last point, the vast majority of people aren't competent computer owners or users. They don't know what to do on a technical level with malware in their boot process.
I don't, because it's not something I so far cared about. But if I started caring about these sorts of attacks, I would not rely on board firmware to check software on my disks:)
> vast majority of people aren't competent computer owners or users.
Vast majority of people don't read HN and are not interested in the boot process. Those that do are more relevant, and I suspect many of them don't welcome Mr. Poettering's agenda.
> They don't know what to do on a technical level with malware in their boot process.
I'm not suggesting they should. But Mr. Poettering does not discuss origins and possible ways to solve this problem, e.g. using free software from reliable sources only and auditing software and hardware we use. He presents his (an the secure boot complex's) preferred solution to a problem that is complex and probably will try to "gently push it" on distributions like he did with systemd (his words).
UEFI is simply supposed to find an EFI folder containing boot files, on a FAT32 partition.
A proper UEFI firmware will find the EFI folder regardless of whether the layout is for MBR or if it is GPT.
So you just use conventional MBR layout, format with FAT32 and throw on a (carefully crafted) EFI folder.
This is how Windows live (startup) media is normally done, so you can boot to either type and install Windows to the PC. On recent Windows versions the install.wim file has crept up in size beyond what FAT32 will support, so NTFS is needed for them.
Linux live distros do real well booting from FAT32 using Syslinux (ntfs is also an advanced option), just like the monolithic ISO's do using Isolinux. But to install the distro on the HDD I want an additional EXTx partition to hold the Linux, and still boot to it from the FAT32 boot volume.
The main partition can be whatever, but that's not typically available until after the kernel & initramfs are loaded. (As it is typically initramfs that does the prompt for the password, to decrypt it.)
Adding out of tree ZFS to the boot mix sounds hella complicated.
...I say, as someone who does in fact leave my boot filesystems on VFAT:)
Since 22.04, zsys (the tool that did the snapshoting) is not installed by default.
Er, are they not snapshotting the root filesystem or ex. /home by default then?
It's really nice because you get to boot from multiple datasets in the same pool so upgrades and downgrades can be very smooth.
- Windows
- A Linux kernel with zfs zupport sufficient to boot up and load zfsbootmenu
- rEFInd boot menu
The default target is rEFInd which shows this snazzy menu to choose between Windows and Linux
https://i.redd.it/n51sxvv8xfw61.jpg
If you choose Linux the zfsbootmenu kernel boots up imports available pools and lets you boot your current system or a prior snapshot of same. If you have regular snapshots especially automated ones before upgrades its pretty hard to mess it up.
The whole works is one small partition and is fairly easy to understand
Code quality of the firmware in typical systems is known to not always be great. When relying on the file system driver included in the firmware it’s hence a good idea to limit use to operations that have a better chance to be correctly implemented.
Remind me again why I should cater to people who insist on writing and running terrible code.