The wrong way to switch operating systems on your server | Hacker News Reader

The wrong way to switch operating systems on your server

figbert.com

109 pointsohjeez73 comments

These experiences are relatively common, we just don't see writeups that often. Kudos to the author for writing this up. I do disagree with some of the lessons here.

You want to switch operating systems on your server?

1. Set up monitoring. If you don't have monitoring already, hack something together with simple scripts.

2. Start up a new server.

3. Migrate services and data from your existing server to your new server. Do this at whatever pace you feel is appropriate. Point the monitoring scripts at your new server.

4. Switch DNS records.

5. Wait. You are not in a hurry to turn off the previous server. Why not wait one or two months?

6. Turn off the old server.

7. Wait some more, and then delete the old server.

The idea here is that steps which might take your site offline are easily reversible. For example, switching DNS records. It's trivial to switch the DNS records back if your migration unexpectedly failed. As much as reasonable, you want the ability to go backwards and undo the steps that you've done to get back to a known good setup.

In particular, I would say that backups are usually not the right tool for migration. This is missing from the lessons at the bottom of the article. The way you get more confidence in your backups is by doing restore tests into a sandbox environment, by adding automated monitoring to your backups, etc. Trying to address a lack of confidence by increasing the backup frequency doesn't make sense. The backup frequency is the most trivial thing to adjust and doesn't address deeper issues, like the fact that you need to dump/restore databases properly and shouldn't copy files from a live database. These issues are discovered through restore testing.

The saying goes, "Nobody wants a backup system, everyone wants a restore system." If you are making backups but not testing restores, you're gonna get bitten. Test the part of backups that you care about--the ability to restore data--and don't test it live. Test it in a sandbox.

This approach saved my bacon on a migration just yesterday. I had gotten to testing files after having done a big baseline rsync, stopping file services on the old server, and doing a catch-up rsync incremental. Oh shit - one veracrypt container is corrupted and won't mount. Turns out rsync diff updates and mounted containers being written to don't play nice together.

Since the old machine was still sitting there with all the data accessible, I was able to just blow away the corrupted volume, confirm it was unmounted on the source, then copy the whole thing over. If I just had a one-time copy that I'd thrown at b2 or something, I would have been very sad.

So, yeah, test restoring your backups. Even fancy checksumming filesystems and shit won't save you from bad assumptions about the integrity of your data.

Anything you do in production should always be reversible at least one step back with a single command. You should always be able to roll back to the last-known-working version at the first sign of trouble with the new version. (Not that you should necessarily respond to all trouble with a roll back but you should always design your processes to keep this as one of your options.)

The way I usually do this is that I have a production PUSH and ROLLBACK script. There is a single symlink on the production server that points to the current working version of the code. PUSH makes a complete local copy of the current working version, changes the symlink to point to that, then pushes the new version, then changes the symlink to point to that. ROLLBACK just changes the symlink back to the backup copy. This is robust and easy to tweak to allow you to roll back as many levels as you like, though I've never had to go back more than one. If a problem doesn't manifest itself immediately you probably want to fix it by going forward, not back.

In the corporate world, this is also a great way to hold vendors accountable.

Story: Company Red hires company Blue to take over the company blog as part of a marketing initiative. Their point of contact was the marketing team at company Red, and I'm sure they thought, "We've nabbed a big customer, this is going to be great."

However, the switch from the internal blog to the external blog goes through the operations team at company Red. The operations team tells company Blue, "Here is the plan for rolling back to the internal blog at the touch of a button."

Company Blue was suddenly much more responsive to questions from company Red.

I do a variation of this when moving to a new developer machine. I virtualize the previous one and keep it around for a long time.

Yeah I was legitimately confused why there was a problem until I realized that author had installed the new OS to the same server as the old one. I can't understand why that was done. Previously I wouldn't even have called your method of moving services over one by one while keeping the old setup available "best practice" because I thought it was so obvious. I hope the original author realized that this should really be what is learned from the story.

I would also setup a reverse proxy on the old server so it would forward all traffics to the new server, and finally kill the old server once the reverse proxy no longer see any legitimate requests.

Oh man, I totally forgot about that!

Wow, that's much more technically advanced than I was as a teenager! Way to go!

To print progress with tarsnap 1.0.39, send it a SIGUSR1 or SIGINFO. On FreeBSD, you can do this by pressing ctrl-t. On Linux, you have to use the unfortunately-named `kill` or `killall` command, such as

killall -SIGUSR1 tarsnap

https://www.tarsnap.com/tips.html#check-current

(Note that Tarsnap is not responsible for naming the unix `kill` or `killall` commands.)

In the unreleased git version of tarsnap, there's a `--progress-bytes SIZE` command, which prints a progress message after every SIZE bytes are processed.

As a general note: the tarsnap-users mailing list is a great place to ask for tips. As you mentioned in your lessons learned, some of the options could have helped a lot (such as `--recover`) https://www.tarsnap.com/lists.html

(Disclaimer: I'm employed by Tarsnap Backup Inc.)

"employed by"

<grin>

Note that that's gperciva not cperciva. I believe the person's name is Graham Percival, and I am fairly sure Colin Percival is still called Colin Percival. They may well be related, but I'm pretty sure that's a bona fide "employed by" :-).

Yes.

Yes. For the curious,

https://github.com/Tarsnap/tarsnap/graphs/contributors

I feel like a lot of this article really should be about how bad tarsnap is. Defaults to no progress updates, has no builtin multithreading, has failures around large files, can't backup sym links properly, no builtin way to detect an in progress restore so you have to manually tell it to resume, etc.

If tarsnap didn't have a bad UX, this entire article would have instead been 'the time I forgot to backup my .env file', none of the other issues would have occurred.

I agree. Used to use tarsnap but the speed is garbage.

It’s so bad that actually restoring anything of any size is just not viable.

Don't consider it a backup until you've successfully restored the data from it.

The first thing I do after setting up a new data backup is test a restore of the data. Only after that will I feel confident that the backup procedure works right.

In the article author's case, an attempt to restore would have caught the problem of the missing .env files and the large movie files.

As for the Ctrl-C on both the backup and restore, you should check your I/O (network and disk) before terminating a process. Doing that would have confirmed that the process was still going, and indicate the rate at which the process was going.

+1. `strace` can be very helpful here, to see if a process is stuck waiting on something or whether it's just zooming along with no output.

`progress` (https://github.com/Xfennec/progress) and similar can be very helpful too depending on the backup utilities being used (in my case often involving rsync) even if the processes normally have everything set to quiet so no progress information is automatically forthcoming.

Wow this looks great, didn't know about this tool. Thanks for the recommendation!

> Don't consider it a backup until you've successfully restored the data from it.

Ouch! I back up every 2 hours - should I REALLY restore from each of those?

No, not ALL backups, just the first backup you do: you test that you can restore the data from it.

Then you can be confident that following backups will be restorable.

Corrolary: If you introduce a new folder for another service in your backup, make sure you can restore it too.

:itsatrap:

It needs to be done periodically.

No, not if they're written by the same hardware using the same process. You need to restore data from one of them, and then another one every so often. Not every two hours, though.

Maybe, it depends on your data-loss and time to restore flexibility. Test regularly enough that you are confident it works. If you don't test at all you can't be confident that it works at all.

For instance, if you have a full/diff/log backup cycle for a database for instance, perhaps test restore each full backup. Just be aware that if a full backup happens daily your comfort zone for data loss in the case of a disaster needs to be at least a day worth of work.

Also, if you backup a range of data and your method allows for partial restores, you might do partial restores of key information far more often than you test everything. You could spread the testing load temporaly: check the really import parts every time and cycle through the other bits less often.

Also if your concern is man-time, automate the process as much as possible. My mail server (running Zimbra) has a small replica in a VM that restores itself from the latest backup once a day and sends me a message to say what the last message in its queue was. This way if I don't get the message, or it says the last message was too far ago, I know something has gone wrong and the backup or the restore failed. I manually log in regularly to inspect a little deeper too, this is slow as the VM has far lower resources than the main box. If properly separated from other resources (so there are no single points of failure that can take it out along with everything else - mine isn't currently that well arranged) then this copy becomes an extra secondary single-snapshot backup itself.

I really meant to restore in order to test your backup process. It's important to test your backups periodically to ensure your process wasn't inadvertently broken, but you don't need to test each and every backup.

I'd even go a step further. It's not a backup until you've restored it using different hardware. You really want to know that your tape (or whatever) can be read by a different tape reader than the one that wrote it.

> I woke and the backup was finished! I wiped the VPS

Just this single line made me scream in horror.

Let me get this straight. He launched some backup command, it didn't output anything for hours, he suspected it hadn't done anything, aborted with ctrl-c, and then learned that he aborted it at 90%. Wipes the partial backup, starts again.

After _that_ experience, he blindly trusts the result of _the same tool_, blindly wipes everything? Wtf.

Spoiler: That's not even part of his "lessons learned".

I know whom I won't hire for my company IT or devops or whatnot.

Brutal honesty can be a wonderful trait all of its own in a team however. We can assume the author is not a BS artist from his very candid self-evaluation. This is a good thing he should hang onto.

"My haphazard strategy resulted in three days of stress and frustration as I clambered to restore a self-hosting empire that I myself had reduced to ash."

Plus he's able to reflect on his own mistakes and articulate what he learned from them. Would hire.

One of the articles they link off to about choosing a different provider explains that the author is a student. I think it's reasonable to assume they don't have much experience with this type of work. Hopefully they'll learn a lot from this process and maybe from the comments here as well. Maybe lets not write someone off before we understand more about their experience and background, especially after they shared an honest account of something they screwed up and learned from.

Do you normally hire high school students at your company?

They cost less.

Good idea to let him graduate from highschool before hiring him !

i did things like this in high school tool...

Given you phrased this "for my company IT or devops or whatnot" I'm going to guess you have very little if any input into who gets hired into those positions, or at least shouldn't if you yourself don't know what the terminology for them should be.

Secondly taking shots at people like this does not encourage people to share learnable lessons like this - instead it encourages them to hide it away. Again if you are person in management you'd think you'd know better than to encourage people to hide mistakes instead of owning them and learning from them.

Finally - you do realize this is a kid right? Not some seasoned professional?

> I’m FIGBERT, an Israeli-American high school student

They've got probably half a decade of youthful mistakes to make yet before they'll darken your recruiting door. Give em a break.

(At their age I think I was still bashing on Locksmith to copy AppleII game floppies, and I turned out OK enough to be managing the devops team where I work now...)

> After _that_ experience, he blindly trusts the result of _the same tool_, blindly wipes everything? Wtf.

You imply that there's something wrong with the tool because he aborted early the first time? Why?

I would check the backup too but why emphasize "the same tool" like that?

It has a -v flag.

My current backup strategy is to backup the whole filesystem. I run services in actual VMs, not docker containers, with disks mounted from LVM volumes which allows me to take snapshots and back them up live without needing to shut them down. I'm using bup to do the actual backups to a server I'm keeping in my home. I wrote a few custom scripts to backup and restore servers, and keep a history of the last x days, y weeks and z months. That way it gives me more time to figure out if something's wrong, as it's hobby stuff that I'm not checking every day.

My advice for OP would be to 1/ ditch tarsnap. A backups tool that runs for hours without any feedback ? A restore tool that fails if the files are too big ? Everything extremely slow ? Just forget it. 2/ keep more than 3 days of backups. It's too short if you make a mistake, it took 3 days to recover from this one already. 3/ backup everything. Don't try to pick and choose files, you're likely to forget something, and if not now then some time later when you create a new file but forget to add it to the list of things to back up.

> My terminal sat empty for hours. There were no changes – the process was running, but there was no feedback. I was nervous.

> What if it failed silently? How can I check? What should I do?

On Linux, find the process ID and run e.g.

    ls -l /proc/12345/fd

which will show you all the files currently open by the process. For something like a backup of a whole directory, or something generating a lot of output, run it again a few seconds later. If it's opened different files, then you know it's making progress and it's not stuck.

If it's something that operates on a single file, find the number corresponding to that file in the list (the file descriptor), and run e.g.

    cat /proc/12345/fd/3

which will output a "pos" field showing the position in the file, in bytes. Compare it with the actual size of the file, and also run it again a few seconds later to see how fast it's making progress.

(You can also use strace, but that slows down your program and potentially changes how it behaves in extremely unusual cases, so it isn't the first thing I'd reach for unless I really think the program is misbehaving and I want to see what it's doing in more detail. And there are tools like iostat too, but they're systemwide.)

Also there's the simple method of using iotop or a network equivalent.

Far and above, the best strategy is to spin up the new server, scp/rsync the data to the new server directly from the old server, and then boot services, and only decommission after you’ve moved all DNS over and confirmed the new site is working. Using Tarsnap for this is not only time consuming but needless unless you already have it setup and working.

Using tarsnap has one big advantage: it proves that you can recover from your backups. Using this method caused the OP to realize the backups were there but the secrets were missing!

I agree with keeping the old server in place until the new one is working obv

My strategy is to setup the new server, and once everything is ready, stop the service on the old server and replace it with a reverse proxy to the new server, and THEN finally update the dns record. When the old server no longer see any meaningful traffic, I'll decommission the old server. This way I don't have to deal with both server out of sync.

Why tarsnap when wasabi or backblaze would be significantly cheaper? You can just encrypt by yourselves anyway.

Also I run my own personal infra and here’s what I do:

* treat servers as cattle, not pets. This is really important. Have mandatory reboots, never be afraid of reboots.

* preferably do things with an automation method, I use ansible for n=5 but pick whatever you like

* have SOME monitoring. It’s not too hard to throw up prom+grafana so get on it early.

* VPN instead of securing internal services. Attack surface is way too high if you’ve too many services. Just throw them all behind a vpn and expose selectively. I use WireGuard.

* personally: don’t self host critical infrastructure. I can’t afford downtime on email etc so I rather just pay someone to host that. Personal infra is for fun, not a second job (and I’m an SRE).

> Have mandatory reboots, never be afraid of reboots.

Yes! It is not a badge of honor to have a server that's been online for 365+ days -- it just means you haven't applied any kernel security updates for a long, long time. :-p

> Yes! It is not a badge of honor to have a server that's been online for 365+ days

The badges come at 1000 days and every 500 after that. If you have a minimal kernel, and a limited software set, many kernel security bugs found won't apply to your system and you don't need to reboot for them.

Not that you need to be scared of reboots, and you should probably schedule some reboots so you can be confident your systems can reboot, but you also don't need to be scared of uptime or stability, either.

1000 days of uptime is 1000 days of configuration drift since the last time you confirmed your services still boot properly.

I don't want to wait for a power failure to find out whether my services still recover from a cold boot.

If your push process is reasonable, you don't need to confirm boot time config changes on all machines.

When you make a boot time config change (which, in my mind, includes kernel version and OS version to some extent), decide if it's nice to have or needs to have. If it's needs to have, well everything gets rebooted and no badges for a while. If it's nice to have, group hosts by kernel version and hardware spec and reboot one or two hosts with median uptime; your config will be tested, and you can still earn badges.

If you can't tell what configs are boot time or not, just reboot a median uptime host in each group once a week. If you can't tell what's important enough to reboot everything, I guess you can just reboot everything every 49.7 days, but you'll never get badges that way. You also won't get production data comparing current kernel to older kernel to see if any new software would have worked better on older kernels, so you have some data to start with when you hit new bottlenecks.

On my server, I have three 'root' partitions. One for general day-to-day use, one for a backup if something catastrophic happens to my main system, and one for experimentation. The extra disk space taken up for my having two extra 'root' partitions is a miserable 40-50 gig.

But I know that I can swap operating systems over almost instantaneously and then back again just as quickly if I did something wrong.

Great peace of mind for practically no cost.

The second error I see in the article is to use software that we haven't used previously for something important. My first wife had the habit of trying new recipes when we had a dinner-party. I tried to tell her repeatedly to try the recipe on us first, then she would have it down pat when she wanted to impress.

The third error of course was the need to have restorable daily backups and the use of them to restore the system when need be, associated with modularity of the system.

I back up my whole system daily. but the most important part of that is not backing up the distro itself (we have re-installs for that) but backing up all the config files, all the databases, all the local binaries, and a current list of all of the installed distro packages. I can replace the whole operating system from a complete wipe-out in less than two hours.

I store these backups in a pseudo-exponential policy. I have more recent backups, fewer older backups. Currently I have 15 backups covering 9 years, with five of those covering just the last 3 weeks, and four covering the last five days. To augment this, I have a monthly snapshot backup also stashed away.

The other stuff, personal docs etc, is deliberately kept small. Total daily backup of base system and /home is approximately 12 gigs. That is easily transported on a USB stick.

I don't store music, photos, magazine .PDFs, old software, etc in my /home directory. That stuff all goes in an archive directory that's write-once, and store (practically) forever. That gets rsynced to two external USB drives daily. Most days, there's practically nothing that gets transferred out.

Having several times lost much valuable data, I suppose I am really paranoid, but I still think I haven't been paranoid enough.

Without a sweat, I recently changed from Ubuntu to NixOS on a server I have access to only via SSH (so mounting an ISO wasn't an option) using[0], so the switch happens live(!) and when you reboot you end up in NixOS. I've also done it to CentOS 7.5, but manually via[1].

It was the case here that backups were not necessary, but it's pretty incredible what's possible when the system configuration is declarative and only /nix and /boot are needed by NixOS to boot. I highly recommend people new to running servers to try NixOS.

[0] https://github.com/elitak/nixos-infect

[1] https://nixos.org/manual/nixos/unstable/index.html#sec-insta...

Someone tell me if I'm missing something, but isn't the whole point of hosting things in a virtual environment so that when you want to switch/upgrade OSes, you stand up a second server and start migrating apps over one at a time?

I can't understand why that doesn't done here.

Well the title is "The Wrong Way to Switch Operating Systems on Your Server" so it's unsurprising that it's describing completely the wrong way to do things.

The concerning part is that the author seems to have learned the wrong lessons from doing things the wrong way.

Yep, the correct title would be "There is no Right Way to Switch Operating Systems on Your Server" Just don't do it. Fire up a new one, and migrate everything over.

This is a bit dumb and I feel like if as much effort had been made doing it as writing the blog post, a more positive outcome would have resulted.

For me: rsync > tarsnap

I have a backup rsync script that parses a file I have that lists every path I want backed up. Yes, this considers dotfiles, so the poster's .env file would have been backed up. My script runs locally, backs up to my main (home) server, and then does another rsync to a 'cloud' server. Want to backup a new file or path? Add it to the manifest file. Adding another server or device? Build another manifest script, have rsync write to the same dir on the server, it'll automatically get sync'd to the cloud server too.

Spent three days fighting with this and didn't think to try the -v flag on his apparently-hanging process?

The author may not have the necessary familiarity with it. Experience is gained via mistakes too.

My first thought, also always run long running jobs like this in Screen or Tmux! As it is, it's a good learning post that others should be able to build on (and don't hit control-c just bc it's taking too long!)

Every server of mine has "history" in a form of script that can rebuild it from the scratch on brand new clean machine. Every time anything is added / modified the script is ran on clean machine to test if all still works as supposed. My databases are all automatically backed to a different backup servers and are restored when needed by the same script.

So for me setting up new server involves following steps:

1) Modify config portion of the script (basically change default values if needed) and run said script on a new server 2) Test 3) Put old server into "read only mode" 4) Run script again with the option to restore the database only. 5) Switch DNS

There are/could be slight variations of course. Practically, I did recent migration of the new server for new product that was running on my location in Toronto to Hetzner. It took a "whopping" 15 minutes of which 13 been taken by restoring database from backup.

What a nightmare! I swear I had something similar with rsync on a mac once, I was very certain it finished, ran again and it reports it's done. I migrate and I'm missing all these files! Although it was probably my fault, I really don't trust rsync anymore... Maybe it had to do with HFS+ and those strange aperture libraries but man it ruined my day (week).

Sure migrating is 100 times more relaxed when you have the old system running, but sometimes you need to reinstall. I had only one MacBook, now I only have 1 server.

What you could do in that case is just install a new disc, unplug the old one until the new systems is running. It's worth the money and effort.

I'm looking to install NixOS to my Home Server next week. All my personal infra is in Docker compose, on Ubuntu 20.04 atm. I only have one m.2 slot in the server and I don't want to buy a second drive just for this... So I'm sweating already. Maybe I should first migrate to my nuc, then back to the new server... hmmmm...

I had exactly the same problems, and I decided to change not my backup strategy, but my deployment strategy. I wrote a small tool to deploy everything in a single directory, using Docker Compose:

https://gitlab.com/stavros/harbormaster

It allows you to separate important state (data) from non-important state (caches). This way, all you need to do is back up the data directory, and then you can restore the Harbormaster config file (along with the data) on the new server and you're done.

The problem with the conclusion is that he implies his error was to not be careful / smart or knowledgable enough… but his error was to not use a fool/error-proof method.

Moving containers between hosts is not a use case that is well supported by docker. The tenor being that you should rather delete and recreate. While not a replacement for docker, with LXD moving containers between hosts is a single command. In many aspects I find LXD's UX more polished and its capabilities more complete, so I often do recommend it to people. Even if just to containerize docker itself to gain some of the missing functionality...

if you get nervous and ctrl-c things with no output, try using -v mode

This is fine, except for the annoying tools that think they need to then print their software version, and exit.

The moment I see: - shutdown - backup + restore anew - bring up

This triggers shivers. Nononono.

- backup (while your current gif is running) - restore and test—verify (via a separate vhost) - delta-backup, delta-restore and test-verify - stop, delta-backup, delta restore, start at new place

This way you minimize the amount of unexpected. Of course can be infeasible in some contexts…

One big WTF in my mind is the default lack of feedback with tarsnap. The -v should be implicit and automatic, and should need a flag to turn OFF, not on.

I mean, that’s just UI/UX 101. _Always_ provide feedback on progress, unless explicitly and intentionally silenced.

Tools that are intended to be used in scripts often have fairly quiet default modes.

tar, cp, and many other *nix tools in that general realm tend to only speak up when something goes wrong, at least by default.

That said, I'm on your side and I think quiet mode should be the option with a reasonable human-friendly verbosity being default. Or even better, some form of detection of if it's being run interactively versus in a script.

Exactly - it’s very possible to detect of being run interactively and at least provide a hint - or even when Ctrl-C is hit (I know some programs will print the status on ctrl-c and tell you to hit it twice to kill it). At least when exiting and saying 90% give a hint how you could have seen thy without killing it.

He forgot one mistake: never take down your existing infrastructure before the new one is up.

I've not used tarsnap before but it doesn't seem to be worth the frustration being caused here. I think rsync works pretty well and it's end to end encrypted over SSH with multiple ways to perform compression.

Tip: have your "data" on a separate file system mounted outside the operating system (OS) root.

Then you can just remount it on the new OS

Reply on news.ycombinator.com