OpenBSD 7.5 via QEMU on Hetzner physical machine (no phys. access / KVM console) | Hacker News Reader

OpenBSD 7.5 via QEMU on Hetzner physical machine (no phys. access / KVM console)

hackmd.gfuzz.de

148 pointshoschi_47 comments

Nice write-up. It should really not be that hard but unfortunately it is. And it is getting worse.

It made me sad to see that Hetzner had discontinued the FreeBSD rescue system. But it seems to be correct: https://community.hetzner.com/tutorials/freebsd-openzfs-via-...

How much did it really cost them to have the mfsbsd image available?

For Hetzner and other dedicated servers, there is a prepackaged script to bootstrap FreeBSD from Linux: https://depenguin.me/.

Similar to what some other commenters said, booting into the Hetzner rescue system and installing FreeBSD from there is pretty straightforward.

I have some notes on doing that, including notes on my setup for disk encryption with FreeBSD on Hetzner:

https://gist.github.com/ctsrc/9a72bc9a0229496aab5e4d3745af0b...

Well, an alternative approach for their bare metal boxes is to request a KVM and link whatever image you in the KVM request form. They will then download it, put it into a USB stick, and plug in the KVM and USB stick. In my experience, you will have it all set up within the hour.

This is of course easier (although the KVM software is a heap of proprietary arse with a web interface or ancient Java blob), but if you are truly paranoid you will have to trust the Hetzner staff and their software stack not to bring something undesired along with your provided image.

Just a sidenote, in my experience, over the part year or so, Hetzner has finally switched to more modern KVMs with good HTML5 consoles. The Java applet was indeed a pain.

I know as well that they are often impressively quick to respond.

But it is not the initial setup I am really worried about. It is those fan meet fecies moments. Being able to respond to a situation instantly. Mfsbsd was a godsend.

Trying to fix problems on one OSis from a different OS is a much harder issue. Especially under pressure.

I would much rather have seen they added OpenBSD to their PXE environment. Just the vanilla stuff - no fancy optimizations.

Tbh, it's not that "hard" nor is it hard to make it more convenient.

I've actually built a small tool that has the UX of regular SSH but under the hood reboots into rescue, configures the keys and opens an SSH session. Once you close it, it reboots back into regular mode.

I've wrapped that tool again to then build a tool that takes just an ignition config and automatically images a hetzner server with fedora coreos using that config.

You could easily build your own tool that reboots into rescue, installs a VM, mounts all the necessary devices, boots the OS image, and exposes the serial console of that VM to you.

It took me a while to learn to love Hetzner's solution, but I prefer it over having to use shitty proprietary KVMs.

To be fair, it's only that hard because the author wanted to do it without KVM. As they note getting KVM access for a Hetzner server is only a short message to support away. Last time I requested it (to change BIOS settings) I got it within 5 minutes.

On the off-chance that the author of the original text sees this, please know that I have passed it onto a few colleagues here at Hetzner. For those here who have posted other workarounds, thanks for that. And thanks to others for other constructive comments here. --Katie (Hetzner)

I tried a couple times to get OpenBSD 7.4 and 7.5 booting on EC2 and GCE with no luck (hang during the bootloader). I ended up going with https://openbsd.amsterdam but it would be nice to have working images in the major cloud providers.

Just out of curiosity, what are you running on OpenBSD?

httpd with some critical files; I'm taking a diversified approach of offline storage and various OS hosts to store the keychains and other things I'd need to bootstrap connectivity to the remaining online services, and maybe I'll host some personal pages as well.

I got annoyed with the openssh bugs this summer and figured I should have some hosts I can mostly not worry about.

In general, I think I want OpenBSD to be the Internet-facing hosts as much as possible, so looking into proxy options for TLS+QUIC.

Out of interest, do you put relayd in front of httpd? (I’m not knowledgeable enough to know whether that would improve security or not)

Why would it?

I've used relayd in front of a ruby on rails app that spawns 4 sockets, but I don't see why you would do it for httpd.

Note that httpd is based on relayd and likely still shares a lot of the same code.

Vultr has OpenBSD images too.

Hi there neighbor!

Ended up there for more or less the same reason. Shame there isn't more hosted BSD options around.

American Cloud has an OpenBSD beta out. I'm sure they'd be happy to have more testers.

This seems unusually complicated. With Hetzner's infra you should be able to just boot the install kernel ("miniroot") and proceed from there with a normal networked installation via their serial console. Same method also works with Oracle Cloud and Scaleway - I keep OpenBSD VPSes with both of them.

nixos-anywhere is a tool that allows you to do something similar (?) to install NixOS in-place on a remote machine (using kexec “magic” under the hood).

https://github.com/nix-community/nixos-anywhere

I tried it on a Hetzner VPS and was honestly pretty surprised that it even worked. What makes it even cooler is that you can continue to rebuild the machine’s config remotely even after initialization (thanks to NixOS).

I recently finally rented a VPS from Hetzner, and to my pleasant surprise was done and ready with my config (shell, utils, services) in under 5 minutes. It took me longer to read about nixos-anywhere and disko in aggregate. NixOS lends itself pretty well to these kinds of magic works.

Reading about it is a one-time thing. Once your config is setup, the single command takes less than 5 minutes to run.

Of course, if what you already have works and/or you aren’t using NixOS in the first place, this probably is not the right tool for you.

Hetzner offer Nixos isos. I got an arm server there running 24.05.

Oh nice, is that a recent change? Because I don’t think I saw the option a few months back.

You can see this option on Hetzner Cloud by navigating to a cloud server you want to use it on, and then to "ISO Images" in the menu at the top and then navigating through the alphabetized list of images. --Katie (Hetzner Online)

Pretty clever to boot a virtual QEMU VM and pass through the physical disks for installation. Nice that it can be done without a support request.

Did the same to install Proxmox. Pretty straightforward though I probably wouldn't have thought of it myself had I not read about it.

Struggling right now setting up various qemu on a hetzner, specifically with IPv6 on bridges etc. I used to have Proxmox servers for work before but never had to worry about networking.

Any issues or recommendations considering the Proxmox route? You do port forwarding or multiple ipv4?

Thanks for putting this idea in my head!

I use Proxmox on Hetzner for a few of my toy VMs, I just rented a few extra IP addresses.

I also have an instance of PFSense running and any VMs that don't need a full IP address to themselves can just be port forwarded through the firewall.

I know you can do this with iptables but I am too lazy to learn how it works.

I once did similar black magic to have a dual-OS set up (Linux & Windows) where the _same_ Linux OS can be either booted into or ran in a VirtualBox VM from within Windows. I probably spent more time reading about it & getting it to work than I ever actually used the OS, but it was a fun learning experience.

Well, I did it 11 years ago. There is little magic. Boot linux rescue system. Fire qemu. Go with auto-install. Even with raid setup if you want to. Done.

The only tweak -- auto-detection of swap space, as it is derived from RAM available and you cannot give all 100% RAM to qemu. So you need to adjust for it.

https://www.dim13.org/Install-OpenBSD-on-remote-host-without...

Linux/Unix Kernels allow you to do some insane stuff. My best current example was swap on VRAM but this is even better.

Discontinuation of FreeBSD rescue system catches me off-guard when I ruined boot of my FreeBSD system by inaccurate ZFS operations (not ZFS or FreeBSD fault, but operator's one).

Trick with qemu works, but is veeeeery slow if you need a lot of disk access (ZFS zmirror scrub, or ZFS `send | receive` pipe or something like this).

This site is built on CodiMD, which seems like a really cool platform for editing and publishing markdown: https://github.com/hackmdio/CodiMD

There is also Hedgedoc, which seems to be another fork of the original hackmd (https://hedgedoc.org/history/)

Website seems to have gotten HN’s hit of death, but the headline sounds fun.

I’ve also got OpenBSD 7.5 running on a Hetzner server, but it runs “natively”. By which I mean it’s still a VM from Hetzner, but I don’t have my own nested QEMU layer or anything.

I host a virtual machine with Hetzner and it runs OpenBSD: the image was right there in the "ISO Images" tab. It seems that setting up a physical server with OpenBSD is much more complicated, though.

It's not particularly complicated. You boot into the rescue system and from there "dd" the OpenBSD install kernel to the host's boot disk. Reboot and attach to the host's serial console, then follow the installation script as usual.

I do the exact same for Illumos, just ripped ideas from depenguin.me (which is how I previously installed FreeBSD after they discontinued the rescue system).

QEMU most likely is not required. OpenBSD's installer is inside a single 4.5 MiB [1] ramdisk kernel image. Chainload or netboot it, or download the ramdisk to ffs on sd0 and run installboot [2]. Once the ramdisk kernel is loaded you can erase the disk containing it.

1. https://cdn.openbsd.org/pub/OpenBSD/7.5/amd64/bsd.rd

2. installboot also needs /usr/mdec/biosboot and /usr/mdec/boot from base75.tgz.

At least for Hetzner's VPS, I get away with the following and then a reboot:

    wget -O - https://cdn.openbsd.org/pub/OpenBSD/X.Y/arm64/minirootXY.img |
        dd if=/dev/stdin of=/dev/sda

This is presumably fine for an initial install, as long as it auto-configures correctly via DHCP.

However, if you ever have issues and need a rescue image, you'd need to figure out how to do something like the OP, and do it while learning how to do it for the first time rather than having had a practice run when you first installed it.

Nice! I didn't know about miniroot*.img. It's actually just bsd.rd, boot, MBR+PBR, and bootx64.efi. Nothing that can't be safely overwritten while the ramdisk kernel is running.

Taking the "curl | sh" pattern to the next level.

Thank you, that one cracked me up.

Pardon my potential ignorance, but as someone that usually does the right thing security-wise, is there really much of an advantage to signify(1) and Sha256 if we are pulling the key and hash over the same HTTPS connection as what we are about to verify? It is not like with sysupgrade(8) where we have a trusted key already on disk.

Signify was created so that a human can relatively easily eyeball the hash and make sure it's legit.

If you're just relying on HTTPS alone it means you're essentially trusting the certificate store that Hetzner put there for you.

arm64 or amd64? Does Hetzner offer ARM servers?

They offer both, but there can be differences between locations. Arm64 was the cheapest last time I checked.

https://www.hetzner.com/cloud

Reply on news.ycombinator.com