Hacker News Reader

Comment by gizmo | Hacker News Reader

Pretty cool, although I'm pretty sure I would never use something like this.

What has saved my skin on a number of occasions is the ability to boot remote servers into rescue mode and chroot into the broken system. That way you can use package managers, all your diagnostic tools, and everything else the boot image doesn't provide.

Basically you just mount the different partitions and then chroot just swaps /proc /sys /dev of the rescue image with the real ones, and BAM you're back in business.

For details see threads like: http://superuser.com/questions/111152/whats-the-proper-way-t...

I know that for many of you this isn't rocket surgery, but for those who don't know you have to google for "chroot" when you boot into a rescue image and discover you can't do anything, you might just remember this post.

View on HN

Haha yes, this is the default install procedure for Gentoo! :D

Arch also does it. The first time I encountered this was when I accidentally deleted my EFI partition and was unable to boot. Boot into live media, chroot, run grub-install and rejoice.

Yeah as an Arch user I've used it a few times where an pacman -Syu has caused something to break and it's pretty easy to chroot into your system and fix the broken package.

Something like this has been my default way of installing Linux for years - and I've installed Debian, Ubuntu (with debootstrap), Gentoo and Arch this way. Typically I just create another partition to my LVM volume group, chroot and install over there and then reboot without removing the old OS install and I run the install procedure from the old OS (instead of booting from a Live CD / usb stick).

I do this because I know it works and there's no guesswork involved in what the OS installer does. They aren't really intended for installing beside another system and the default partitioning options aren't always that great (RAID, LVM, crypto, etc).

Basically you only need 3 things to run Linux: kernel, initramfs and rootfs.

You really only need two (kernel and initrd/initramfs), at least to get into a basic running state. This requires building your own initrd, of course, but it's pretty common in non-graphical Linux installers (Slackware's install media still does this, IIRC).

I think it's even possible to embed the initrd in the kernel binary itself, but I've never really investigated that.

You can ditch the initrd if you compile in the kernel the drivers you need to boot the system, as a minimal example SATA/SCSI, EXT4/JFS, that usually are compiled as kernel modules.

How most sane distros do it.

Build a kernel with the most common hardware built into it, and use that to bootstrap. No need for messy things like balled up temporary rootfs in a ram drive.

Initrd/initramfs have become an excuse for piling on complexities that frankly should be added by the sysadmin after initial install.

The purpose of the initrd to have the initrd bootstrap to rootfs. This involves userspace tools like mdadm, lvm2, cryptsetup, zfsprogs, etc.

It's not very common to need kernel modules (drivers) on initrd on typical hardware.

It's not common to need them, perhaps, but it's still useful if you don't want to recompile your kernel just to add on-boot support for various devices. Slackware's 'mkinitrd' tool is one example of this sort of approach; you can add various modules (like for your root filesystem, keyboard, etc.) by adding to the $MODULE_LIST variable defined in '/etc/mkinitrd.conf' or by running 'mkinitrd -k $MODULES'.

You can of course accomplish similar things by just recompiling the kernel (which Slackware makes very easy to do), but if you still need to use 'mkinitrd' anyway (perhaps because you're using LVM or softraid or LUKS), it's often more convenient to just throw in the modules you need while you're at it.

The keywords there are 'typical hardware', but it can get worse. How about 'badly selected' hardware on a short time frame, in situ, waiting for an install and application port process to be performed by non-technical users following a recipe. Proprietary drivers for network and soft raid with a large enterprise linux vendor support and site license basically voided. There are worse things than starting an install,loading storage drivers, creating the lvm partitioning, hup'ing disk druid, installing, rebooting and adding the network driver to initrd, and having to explain and document that manual procedure to the same people who ordered the hardware...but I've forgotten them.

This only works if you stick to a simple filesystem, not crypto, RAID, etc. Otherwise you'll need to have file system utilities (lvm2, mdadm, cryptsetup, zfsprogrs, etc) on the initrd to get your rootfs mounted.

Damn, you are absolutely right!

It never crossed my mind that LVM and similar needs utilities to access the filesystem and to be honest, I thought initrd/initramfs contained only kernel modules, not executables and scripts... ^__^;

Yes, it's a fun exercise to build a tiny Linux install that's fully on the initrd. Not that you'd want to have that kind of system in daily use outside of special applications.

> I think it's even possible to embed the initrd in the kernel binary itself, but I've never really investigated that.

Yes, the kernel config has an option to embed the initrd in the kernel image. I'm not sure if there are any advantages to this.

I use the initramfs method to put an small ssh server in that I can use to unlock full disk encrypted headless boxen, so I could see initrd in the kernel being used in a similar way.

The advantage to embedding the initrd is usually that you don't have to worry about a separate initrd (for example, if you're reliant on some really basic bootloader). It's also more in-line with, say, OpenBSD (AFAICT) which doesn't use a separate initrd/initramfs (not sure if this is true of all BSDs).

> I'm not sure if there are any advantages to this

This comes to mind: http://kroah.com/log/blog/2013/09/02/booting-a-self-signed-l...

Having the initrd merged with the kernel is slightly less hassle when netbooting to test during kernel development

CoreOS's "system volumes" are just rather large initramfs images. This means you don't really have a rootfs partition on CoreOS, just a boot partition (containing a newer and older set of kernel + initramfs) and a "state" partition (containing whatever you like.)

This choice creates a very nice upgrade-management strategy for CoreOS clusters: rather than letting each machine have its own boot partition and asynchronously downloading updates into it, you can just stand up a PXE server to serve the current CoreOS kernel+initramfs and tell your machines to boot from that. Now the entire OS (~200MB) is streamed to each machine on boot; to "update" a machine, you just reboot it. (And, unlike a shared NFS/iSCSI rootfs disk server, you don't have to be careful when updating to tiptoe around the machines that haven't rebooted yet; they just stay happily on the old OS until you kick them over.)

As an added benefit, if the programs you're running on those CoreOS nodes don't need any persistent storage either (i.e. they're not database nodes), then they can be entirely diskless, and just let the rootfs tmpfs hold everything.

> Basically you only need 3 things to run Linux: kernel, initramfs and rootfs.

And networking, fully functional networking.

If you've missed out the FW packages for your wifi drivers, you will have a hard time using your package manager to get those missing bits ;)

Probably not an issue for servers, but for those who'd like to do "complex" desktop/laptop install with ZFS/btrfs roots or whatever, it's an easy mistake to make.

It's funny, I knew how to do that when I borked my systems back in 2004 only because the Gentoo installation instructions were like that.

Sort of the same for me but I learnt it from the Arch Linux installation process.

This is one of those things you want to have practiced before it's critical.

I like the term rocket surgery, I will steal it from you for sure.

If I'm not mistaken, credit goes to Chris Rock (who is not me).

Pretty sure the conflation has been around ever since rocket science and brain surgery have been used as a comparison. :)

I don't tend to link youtube videos on HN, particularly of the comedic type, but I'm just going to leave this here for anyone who wants a good laugh for their morning.

https://www.youtube.com/watch?v=THNPmhBl-8I

I was thinking the exact same thing.

That was great!

While funny, I still cringe because of the http://www.dict.cc/?s=fremdschaemen

But it's not exactly brain science, now is it?

I work at an IT Helpdesk and we build a custom GRML image which we write to our pen drives. We use it, for example, to quickly live boot Debian/Ubuntu PC's and run a script that mounts all necessary file systems (including pseudo file systems). This makes it really easy to quickly chroot into a broken system right at a users desk.

On our Linux installations we have separated most of the directories into different LVM partitions. Such as '/home', '/usr', '/var' and of course '/' (the root). Just last week I helped a user live boot his system and ran a fsck on his '/home' partition. We had "frozen" the system and hard reset his computer which led to a couple of corrupt blocks and inodes.

I recommend to copy a rescue disk iso image or/and network installation image to boot partition and add it to GRUB menu. This way it is possible to boot into working environment without risk to damage root partition.

https://wiki.archlinux.org/index.php/Multiboot_USB_drive

That usually doesn't work for remote servers because you can't access the grub menu over SSH and you want something that works even in those cases that grub or your boot record are trashed. So you just want to do a netboot from a generic rescue iso. Zero dependencies. Works even when your hard drives are severely damaged and you just need to /bin/dd some raw sectors.

Works even when your hard drives are severely damaged and you just need to /bin/dd some raw sectors.

That reminds me of the time when some partitioning utility overwrote the first block of my filesystem and then set the filesystem start to the wrong block. I thought I was hosed, but I managed to use hexdump to find a backup superblock, calculate the correct filesystem offset from that, and dd the backup superblock into the primary location. I may be misremembering some details.

> That usually doesn't work for remote servers because you can't access the grub menu over SSH

Depends on your environment. My hosting company offers a remote console you can enable that gives access to a virtual serial console via SSH; with that, you can access your bootloader.

Reply on news.ycombinator.com