Arch Linux Btrfs with hibernation in a swapfile | Hacker News Reader

Arch Linux Btrfs with hibernation in a swapfile

nwb.sh

86 pointsnwb9943 comments

I just did this last week and ran into an issue with LUKS: https://lore.kernel.org/cryptsetup/CAD-Ua_hPjBLHK0cyEz3k7KYB...

LUKS wouldn't accept my password, no matter how simple it was.

Turns out a BIOS update somehow messed up RAM settings which caused memory errors. And as LUKS needs a lot of memory for some of it's description operations bit flips can cause all of that to fail.

That was frustrating!

I feel you. God I wish there was a company taking ownership in building, selling and supporting a batteries-included Desktop Linux distro where all these quality-of-life niceties that are working so well by default on Windows (and MacOS?), would be working out of the box on Linux without any user maintenance or need to read tutorials or wikis, and also be tested together on various hardware combos so that I have assurance that one of those components updating wouldn't brick my system.

I'd pay a $100-200 yearly subscription for such a Linux distro.

I remember wasting over an hour trying to get hibernate working on my company provided Ubuntu laptop and failing miserably. Or Gnome not having the option to disable the touchpad when an external mouse is plugged in, and the extension made to fix this not working because I'm using Wayland. I love Linux on the server side, but I just don't have time to deal with frustrating bullshit issues caused by this bazaar-engineering on the desktop side, even if it's FOSS. I'd rather pay and have something that works.

The closest thing I found was Gecko Linux (based on Opensuse Tumbleweed) or Nobara Linux (based on Fedora) but even there YMMV.

> I'd pay a $100-200 yearly subscription for such a Linux distro.

Same. Same for a lot of my devices really. I'd pay $100-$200 yearly for another year of Android updates or a viable alternative phone-os altogether.

> I love Linux on the server side, but I just don't have time to deal with frustrating bullshit issues caused by this bazaar-engineering on the desktop side, even if it's FOSS. I'd rather pay and have something that works.

Which is fair. As an enthusiast that enjoys nitpicking my config files to make it all work perfectly, I've slowly come to understand people that don't want to do that. Sometimes you just want to get your work done, and it can definitely be frustrating when every utility has a different config file format, some don't even have config files but instead use gconf, some require a restart and some don't, some rely on new libraries that your distro doesn't have yet, every application has a different-looking UI... it can be a frustrating experience if you just want a MacOS or Windows-like experience. And unfortunately that level of simplicity isn't quite a thing yet in the Linux world.

Fwiw I no longer use disk swap files for anything, but ZRAM swap instead. It's mature, it's in the linux kernel, it's faster, and less wearing on SSDs, and RAM is cheap. I don't have a linux laptop though, so don't know how it affects hibernation. Anyone know how/if hibernation works on a laptop using ZRAM swap instead of disk-based swap?

Hibernation requires a copy of your RAM to be stored on disk. Compressed RAM alone can be used with traditional sleep mode, but hibernation won't work.

That said, you can combine ZRAM and swap files/partitions. You need to make sure the priorities are in the right order, though, and maybe you need to script your system to disable and flush zram to swap on hibernate and enable it again on resume, but that way you can do hibernation together with ZRAM.

ZRAM is amazing for cloud VMs which typically have strict disk IOPS limits which means any significant swap usage is going to destroy IO performance or even lock the system. ZRAM w/o swap completely prevents this IO condition and is much faster than disk swap anyway.

Shouldn't a cloud VM have swap disabled by default if it's using networked block storage? It is on AWS at least.

Azure provides what they call "Azure Temporary disk" at typically /dev/sdb for most instance sizes; it's *not encrypted* and it's ephemeral, but is garanteed to be co-located (at least rack-level) with your CPUs.

Linux generally gets very squirrely when it's in a low-memory state. I think it's a bit safer to turn swappiness down and use a small swapfile and/or zram just to keep it from getting completely bricked up. That way you're sacrificing IO for a few seconds/minutes to intervene.

RAM is pretty cheap but it's not free and not always possible to upgrade, and my personal computers keep ending up with gigabytes of data in memory that won't be touched for hours or ever. I'm happy spending a negligible amount of SSD lifetime to clear up that space.

Seems silly to set aside ram to use when your ram runs out. What am I missing here?

You don't set it aside, unused pages just get compressed to save space.

Okay, I think I don't know how swap works on Linux. Is it:

When there's no more ram, swap is used

When ram hasn't been used in a while, it is sent to swap

Or maybe a mix of both?

Both, yes.

The second one depends on your configuration, and your configuration depends on your hardware and your use-case. For example I have some 4GiB VPSs with a single vCPU and slow IO, so I configure the virtual memory to aggressively compress unused ram with lz4; decompressing them takes no CPU power and is orders higher throughput and lower latency than any disk swap can ever be, even when using just 1 vCPU. Then it means I have gigabytes of RAM that is used as file cache, instead of wasting that space with unused ram, and when everything is needed at the same time it just drops the cache. Having all that cache availability also means that it can take in a lot of files without locking up.

The first one also happens when you have too much RAM allocated for your things. When it's temporary until your applications finish it'll swap around and slowly finish eventually, but depending on your over-allocation and your swap's speed it can take significantly longer.

Repeat after me:

ZRAM is not swap.

ZRAM uses the same OS interface as swap does, and that interface is called "swap" but really is more about composing system memory out of tiered backing memory. One of those tiers can be swap.

With ZRAM and no swap, all your system memory is volatile. As long as Linux requires nonvolatile system memory to hibernate, it will be impossible to hibernate in this setup.

Same here. I was using zswap at one point merely for hibernation but got tired of the dice roll on whether my system would wake up, would wake up but with locked up graphics, would show me a black screen, would show me a blinking cursor or would kernel panic (on my hardware with an Intel iGPU at least). I just gave up on hibernation altogether and been happily using zram ever since.

Interesting!

I went on a similar journey recently with debian and FDE on luks2, which involved a fair amount of learning and messing around, but I got there. No Btrfs though, just plain old ext4+swap volumes using lvm on luks2, and a small uefi partition.

My /boot is encrypted so I enter the LUKS2 passcode before grub can even access its config, and I've rolled an unlock key into the initramfs (which is loaded by grub from the encrypted partition) so that I don't have to enter it twice. The only cleartext stuff is /boot/efi.

Next project is secure boot, but so far I have failed to add a MOK to my system (it just doesn't seem to persist for some reason), and I've not got any further than shim->grub->FAIL.

Hibernate is less interesting, and apparently unsupported using secure boot anyway.

For secureboot you may want to took a look at this project [0]. Don't think it has ever gotten easier to sign UKIs than that though systemd should have a new project (systemd-ukify) that aims to make it more integrated.

Hiberation is not supported in lockdown mode because I'm assuming the kernel (maintainers) expect most people to have an unencrypted swap partition. If you have secured your swap, you can patch [1] the kernel to allow hibernation.

[0] https://github.com/Foxboron/sbctl

[1] https://gist.github.com/kelvie/917d456cb572325aae8e3bd94a9c1...

Thanks, yeah I think I came across sbctl when I was playing around with it. Haven't tried it yet... it got a little confusing when I was reading various guides to try to achieve what I've been trying to achieve, and they started referring to various different tool sets!

I'll have a go with it to enroll some keys and see if they persist. mokutil, and then rebooting into shim to persist them, has failed me. It all seems to go OK and then they're just not there on the next boot.

On hibernation, that makes sense. I hadn't read into the reasoning, just got as far as "MS allows it in secure boot mode, linux devs consider it insecure by design" or some such thing.

You don't need any other tools sets when using sbctl to enroll and sign your keys. It's a one-stop shop for creating UKI bundles and signing them. I use systemd-boot with UKIs created by it and it has no issues detecting the UKIs. Maybe your problem is holding on to grub(legacyware IMO) which has poor support for what you are trying to accomplish.

Hibernation support in lockdown has nothing do with the MS politics around secure boot. You can generate and use your own keys to use with secureboot. The issue is that "accessible" unencrypted hibernation files invalidate secureboot when you can break into RAM and modify system images/files.

> Maybe your problem is holding on to grub(legacyware IMO)

My problem is that I can't enroll keys, going through the enroll procedure (which doesn't involve grub) results in ... nothing.

I'm not specifically wedded to grub, and a UKI signed with a key is a fine idea if I can get a key installed. As such, I'll try sbctl but I have no particular reason to think it will work where shim/mokmanager fail if there's a motherboard issue of some sort.

There may well be challenges using grub down the line, but right now I'm not even getting far enough for that to be an issue.

It is not because of an unencrypted swap partition. It is because, even if it is encrypted, you know the key and can thus replace the image with an arbitrary modified one, or, in theory, with a hacked version of a Windows boot loader, which would break DRM.

There were some movements to remove this restriction, on the condition that the encryption key is properly sealed in the TPM and is not extractable.

Except you'd have to know the encryption passphrase to unlock the swap partition. Only after this step can you use the stolen key to manipulate the state of trusted images.

TPM has never been a pre-requisite for secureboot nor kernel_lockdown. Infact the proposal you are speaking of sounds very exclusionary since TPM hardware is still relatively new and not ubiqitous.

Correct. But you do know the passphrase, and, from the viewpoint of Secure Boot and the locked-down kernel, you (the legitimate owner) are also the attacker who tries to run some unapproved kernel-mode code and will stop at nothing in order to do that.

An alternative to MokManager/shim is to enroll your own key. You'll also need to convince your kernel package to create a UKI under /efi instead of regular initramfs under /boot, and to sign the UKI with your key.

Eg on my OpenSUSE system, the kernel package runs dracut to build the initramfs in /boot, and it's three lines of config to co-opt dracut to build a UKI in /efi and sign it with my key instead. So installing a kernel package automatically creates the UKI in /efi, and systemd-boot automatically notices it on next boot.

Debian doesn't use dracut by default, but it does have hooks in /etc/kernel/postinst.d which should let you do the same thing.

> An alternative to MokManager/shim is to enroll your own key.

I was under the impression that that was exactly what I was doing with mokmanager/shim? dropping into a utility that would perform key enrolment?

Perhaps I have seriously misunderstood!

I'd expect "MokManager" to... manage the MOK. Never used it, though. For one PC whose UEFI was recalcitrant, I've had to use another tool, called "KeyTool". It may or may not be the same thing, but mine let you change multiple kinds of keys, and, in particular, the platform key (aka the "root" key).

There are several steps. First, you usually have to put the UEFI in "custom mode", where you install your keys and end with the Platform Key. You typically get some kind of warning when enrolling the PK similar to "this will get the system out of custom mode".

I've found the Arch guide to be fairly clear: https://wiki.archlinux.org/title/Unified_Extensible_Firmware...

You should be able to adapt it to other systems.

MOK is short for Machine Owner Key, which is just a way of saying "a key you enroll in your UEFI secure boot variables that is under your control".

Or that's what I thought anyway, if it's not actually doing that, and is just enrolling them in some sort of shim-internal DB, then it's a lot less interesting and I'll abandon that approach...

I don't necessarily want to change the 'root' PK ... or maybe I do if it's the only way to install a machine-owner trusted key in the real UEFI... I'll look at KeyTool in case it makes saving a key under my control to the db easier/actually work. Though the keytool I'm familiar with is a java pkcs#12 manipulation tool!

(on further reading, it looks like shim/mokutil creates a new key-set in nvram, separate to the UEFI stuff, that it manages itself and uses internally. This is not what I was trying to achieve and seems very much like vendor lock. I surely shouldn't need to use shim to boot if I'm signing stuff with my own keys... and it doesn't seem like it has even managed to persist this custom keyset. Back to the drawing board!).

As you already discovered in the other comment, the root Secure Boot key is the PK, and the MOK is a concept that shim made up. shim was created so that people didn't have to enroll their own PK, if their firmware even let them do that in the first place. Every distro that uses shim was able to get it signed by Microsoft (MS would not sign each and every bootloader in existence, and especially not any version of grub), and then shim does its own verification of the next stage application using the MOK DB.

What's the point of encrypting /boot exactly? From an Encrypt All The Things! perspective I get it, but practically... it seems overkill?

Presumably to avoid tampering of the kernel and initrd, since GP doesn't use secure boot.

As the other poster mentioned, without secure boot, there are no guarantees about a kernel or initramfs that are sitting out there in plaintext (and yes, someone could mess with my grub install).

It was mostly because "this should be possible, right?"

So yeah - Encrypt all the things :)

You don't need to encrypt anything to verify those images, you just need to sign them. See how Heads does this.

https://github.com/osresearch/heads

Sure, and you could do that with more or less your whole installed system.

From the other angle - if it’s not a lot more difficult than encrypting the rest, why would I specifically exclude having these stored on an encrypted volume?

> Hibernate is less interesting, and apparently unsupported using secure boot anyway.

That's not the case. I have a similar setup to yours (/ on ext4 with separate swap, on LVM on LUKS, separate /efi) and my box hibernates just fine with secure boot and auto-unlock via TPM.

The difference with your setup is I don't use grub, but have the UEFI load a signed unified kernel image directly. Since this works so well, I never had a reason to mess around with yet another moving piece (grub or other bootloader).

As another commenter said, I haven't attempted to mess around with the MOK. I just replaced all the secure boot keys with my own, and I've also signed MS's Windows key (but not the 3rd party one) for my dual-boot needs.

---

For specifics: This is an up-to-date Arch Linux install, running on an HP EliteBook 840 G8 (11th gen intel). I know Debian may have older components than arch, but this setup has been working for more than a year now.

IIRC, the most significant change was brought by systemd 251 which started supporting auto-unlocking LUKS with the TPM. Before that, on an older computer with the same general setup, hibernation worked well, too. I just needed to input the unlock password (which I was too lazy to do, so I just used my yubikey - see https://github.com/agherzan/yubikey-full-disk-encryption).

Fascinating! How do you sign your kernel every time you recompile it? Do you need to store a new key into the UEFI every time?

I use sbupdate [0] to build the unified kernel image and to sign it with my keys. It's run by a hook in the arch's package manager whenever the kernel, the initrd or the firmware images change. I saw the other day that systemd recently got an utility to do this, but I've never looked into that. sbupdate has been working fine for me for several years now.

It doesn't store a new key in the uefi, it signs the new image with the key that uefi already knows about.

See [1] for the whole setup and [2] for the signing part specifically.

[0] https://github.com/andreyv/sbupdate

[1] https://wiki.archlinux.org/title/Unified_Extensible_Firmware...

[2] https://wiki.archlinux.org/title/Unified_Extensible_Firmware...

What's the point of auto-unlocking using the TPM? Does this mean people may be able to access your data without a password?

The point is that the PC boots up fully unattended. People could access my data without a password if they stole my machine and somehow managed to log in to it, say through a bug in the desktop manager on the login process. The LUKS key is in memory, so I guess if they can somehow read that, they could also access the drive. But I figure the convenience is worth it, I don't expect to be a high-level target for this. And if I were, the 5$ wrench attack would probably work better.

If they stole the hard drive, they wouldn't be able to access the data, the TPM wouldn't release the key.

OpenSuse's got a neat patch for grub that copies the password you enter at boot into the initrd. This way you don't need to store a key in your initrd and you don't need to worry about securing those initrd against theft.

https://build.opensuse.org/package/view_file/openSUSE:Factor...

Side note, you can use this same technique to add the LUKS password to the initrd just before doing kexec. This lets you reboot a remote system protected this way without having to pull up a VNC to enter the password.

My script for doing kexec with a preloaded LUKS password is here: https://gist.github.com/webstrand/381307348e24c28d5c4c9a5981...

As a daily arch user for over a decade I can't help but be that guy; why not just RTFM? Isn't that arch/gentoo's thing? good docs? about 15 seconds of searching turned up all the info.

To be fair, I use a btrfs swapfile on an encrypted/compressed filesystem so I knew what to look for. But still, I find the wiki very reasonable to navigate and nearly always up to date.

https://wiki.archlinux.org/title/installation_guide

https://wiki.archlinux.org/title/Power_management#Suspend_an...

https://wiki.archlinux.org/title/Power_management/Suspend_an...

https://wiki.archlinux.org/title/btrfs#Swap_file

Added note, there is currently a bug in systemd's suspend-then-hibernate behavior (doesn't ever wake up and hibernate). To use it you'll have to downgrade to the arch-provided systemd version 251.7-4.

edit: formatting

That's exactly what I did to make all of this work. While the Arch Wiki is the best thing since sliced bread, some things on it are spread out or aren't clear: given some of the sections actually have some talk going on. That was my experience trying to figure out the topic of my blog post. Anyway, I produced the blog post for fun and in hopes it may help someone who is confused or unclear as to why something they do may not be working. I 100% cited those exact wiki articles in my blog post and recommended the reader to go through them prior to doing anything else (especially if the configuration was slightly different, i.e., UEFI vs BIOS). While I agree that RTFM is a good attitude to have, it has too many gate-keeping vibes and can discourage new users from learning the ropes. Some people need a little bit of hand holding to get going before being let loose. I was once that person, and I wanted to share my experience with others. Nonetheless, I still agree with your comment. Just thought I'd clear it up some.

Reply on news.ycombinator.com