Takeover.sh – Wipe and reinstall a running Linux system via SSH without reboot
github.com
github.com
What has saved my skin on a number of occasions is the ability to boot remote servers into rescue mode and chroot into the broken system. That way you can use package managers, all your diagnostic tools, and everything else the boot image doesn't provide.
Basically you just mount the different partitions and then chroot just swaps /proc /sys /dev of the rescue image with the real ones, and BAM you're back in business.
For details see threads like: http://superuser.com/questions/111152/whats-the-proper-way-t...
I know that for many of you this isn't rocket surgery, but for those who don't know you have to google for "chroot" when you boot into a rescue image and discover you can't do anything, you might just remember this post.
I do this because I know it works and there's no guesswork involved in what the OS installer does. They aren't really intended for installing beside another system and the default partitioning options aren't always that great (RAID, LVM, crypto, etc).
Basically you only need 3 things to run Linux: kernel, initramfs and rootfs.
I think it's even possible to embed the initrd in the kernel binary itself, but I've never really investigated that.
Build a kernel with the most common hardware built into it, and use that to bootstrap. No need for messy things like balled up temporary rootfs in a ram drive.
Initrd/initramfs have become an excuse for piling on complexities that frankly should be added by the sysadmin after initial install.
It's not very common to need kernel modules (drivers) on initrd on typical hardware.
You can of course accomplish similar things by just recompiling the kernel (which Slackware makes very easy to do), but if you still need to use 'mkinitrd' anyway (perhaps because you're using LVM or softraid or LUKS), it's often more convenient to just throw in the modules you need while you're at it.
It never crossed my mind that LVM and similar needs utilities to access the filesystem and to be honest, I thought initrd/initramfs contained only kernel modules, not executables and scripts... ^__^;
> I think it's even possible to embed the initrd in the kernel binary itself, but I've never really investigated that.
Yes, the kernel config has an option to embed the initrd in the kernel image. I'm not sure if there are any advantages to this.
This comes to mind: http://kroah.com/log/blog/2013/09/02/booting-a-self-signed-l...
This choice creates a very nice upgrade-management strategy for CoreOS clusters: rather than letting each machine have its own boot partition and asynchronously downloading updates into it, you can just stand up a PXE server to serve the current CoreOS kernel+initramfs and tell your machines to boot from that. Now the entire OS (~200MB) is streamed to each machine on boot; to "update" a machine, you just reboot it. (And, unlike a shared NFS/iSCSI rootfs disk server, you don't have to be careful when updating to tiptoe around the machines that haven't rebooted yet; they just stay happily on the old OS until you kick them over.)
As an added benefit, if the programs you're running on those CoreOS nodes don't need any persistent storage either (i.e. they're not database nodes), then they can be entirely diskless, and just let the rootfs tmpfs hold everything.
And networking, fully functional networking.
If you've missed out the FW packages for your wifi drivers, you will have a hard time using your package manager to get those missing bits ;)
Probably not an issue for servers, but for those who'd like to do "complex" desktop/laptop install with ZFS/btrfs roots or whatever, it's an easy mistake to make.
That reminds me of the time when some partitioning utility overwrote the first block of my filesystem and then set the filesystem start to the wrong block. I thought I was hosed, but I managed to use hexdump to find a backup superblock, calculate the correct filesystem offset from that, and dd the backup superblock into the primary location. I may be misremembering some details.
Depends on your environment. My hosting company offers a remote console you can enable that gives access to a virtual serial console via SSH; with that, you can access your bootloader.
On our Linux installations we have separated most of the directories into different LVM partitions. Such as '/home', '/usr', '/var' and of course '/' (the root). Just last week I helped a user live boot his system and ran a fsck on his '/home' partition. We had "frozen" the system and hard reset his computer which led to a couple of corrupt blocks and inodes.
In short, the answer details how to switch your running system to use an in-memory only root filesystem, without restarting. This allows installing a new OS, resizing the OS disks, etc.
It's a risky operation, but the linked answer covers many pitfalls that you might run into - I recently used it to shrink the root partition on a remote server, very much appreciated the detail.
Infact, I will be using it later today, to replace a Debian system with Gentoo, no less. The README in OPs link is spot-on here (the last few paragraphs).
* use btrfs, and create your main root filesystem as a btrfs partition subvolume and another btrfs subvolume for snapshots (also a sub of master btrfs partition)
* to start any experiment (e.g. installing whole gnome and 500 different packages you MIGHT WANT TO REVERT in the future) create before the risky operation snapshot (btrfs subvolume snapshot / /.snapshots/yournameofsnap) of current filesystem
* experiment in any way :)
* switch between old root (snapshot you created) or the new one with (btrfs subvoulme set-default)
* delete any of them (btrfs subvolume delete)
btrfs copy-on-write allows all of these commands to happen instantly without (almost) any actual copying. Also booting from both volumes is possible without any additional steps as long as master btrfs partition is the one to be booted from UEFI.
If you have remote console access, a similar thing can be done for OpenBSD by dd(1)'ing a miniroot ramdisk install image.
Edit: it doesn't really do the same thing. vps2arch could be implemented on top of takeover.sh for better reliability.
On the other hand, this seems like it would be an incredibly easy thing to screw up and potentially leave yourself with a corrupted or unbootable system.
IMHO, systemd in container mode can be used instead of fakeinit.
Look up the chroot-install method for Debian/Ubuntu, as an example of this in a user-accessible manner. The process uses debootstrap, but as I understand was originally developed via a chroot shell on a running system and either partition or ramdisk-based bootstrapping of a new system.
What's interesting about takeover.sh, though, is that it goes a step further and causes the chrooted system to actually replace an existing OS without a reboot and (theoretically) without involving additional boot media.
If that doesn't work (kernel panic or whatever) you can disconnect the drive and connect it via a SATA-to-USB adapter in which case hot plugging should definitely work.