Hacker News Reader

Comment by sugarpimpdorsey | Hacker News Reader

Fair warning: the Trixie update does not allow you to roll back. It is in theory possible but practically it not only fails every single time, but leaves the system in an inconsistent and broken state. (Code for 'soon to be unbootable').

What this means is when you find out stuff breaks, like drivers and application software, and decide the upgrade was a bad idea, you are fucked.

More notably, some of the upgrade is irreversible - like MySQL/MariaDB. The database binary format is upgraded during the upgrade. So if you discover something else broke, and you want to go back, it's going to take some work.

Ask me how I know.

View on HN

The page about upgrading [0] does have this warning:

  Back up your data
  
  Performing a release upgrade is never without risk. The upgrade may fail, leaving the system in a non-functioning state. USERS SHOULD BACKUP ALL DATA before attempting a release upgrade. DebianStability contains more information on these steps.

[0] https://wiki.debian.org/DebianUpgrade

Yet Windows will let you roll back an upgrade with a single click within 10 days.

Of course anyone can restore from backups. It's a pain and it's time consuming.

My post serves more as a warning to those who may develop buyer's remorse.

This is what LVM/btrfs/ZFS snapshots were invented for.

Windows is using Volume Shadow Copies, which for the purposes of this discussion, you can think of as roughly equivalent.

I always find the rough edges on upgrading windows (and macos), I've had several computers that take 3-4 hours to hit a roadblock, give a inscrutable error message and rollback. I feel spoiled using nixos (once you get over the learning curve)

You might like snapshot based solutions like Snapper

I once tried Spiral Linux, light fork of Sid with bundled Snapper stack. Switched to a giant-userbase distro, Fedora, mostly because Plasma was bad with 5k screens. Are there mainstream distros with easy rollback?

openSUSE should be one of them.

as much as I love Debian (been a faithful user since 25 years or so, no more Windows at home since then), that Windows ability is just really cool and Debian is still not on par I believe...

If you keep your /home on a separate partition, you can basically reinstall a whole system without much efforts. It's good to do that from time to time. etckeeper is very helpful too. Lots of desktop apps are AppImage nowadays, so if you keep them in the home directory, they'll persist.

You know imaging your machine is still an option...

But you can't do that on a live system as you can with Windows or macOS. Not a problem for pre release upgrade perhaps. But I'm so missing this feature from macOS.

You can if you're using LVM. Take a snapshot of the logical volume your system is on, then run `dd' against the snapshot, as it's essentially a frozen point-in-time.

I've used this trick many times in a live, rw environment.

Depends on your filesystem. For example, I certainly can as I’m using btrfs. I’m also using Timeshift for easy management of snapshots. As others have mentioned, there are other choices too like Snapper that all work well.

You can snapshot the filesystem if you're using BTRFS, ZFS, or another Copy-on-Write filesystem.

Linux Mint offers rollbacks, I have snapshots going back a point release and a major version.

That has never been supported: https://wiki.debian.org/SystemDowngrade

I said possible, not supported.

Too many bits of 'advice' on Stack Overflow, etc. claiming it's possible as top Google results.

I'm here to say unequivocally: it does not work, will not work, and will leave the system in an irreversibly broken state. Do not attempt.

Well... then it's always been possible, if you use LVM and create a snapshot before upgrading, then you can revert your snapshot in case it breaks.

But this isn't something that would be 'out of the box'... but that's why we make backups, but I can't remember an dist-upgrade ever significantly bricking my systems.

In all fairness... How would that work? Not even just on Debian; in the general case, I don't see how to avoid that other than full filesystem snapshots or backups of some sort. Even on, say, a NixOS system where rolling back all the software and config (basically, /, /usr, and /etc) to exactly its old config is as easy as rebooting and picking the old generation, databases will still have migrated their on-disk format.

Indeed. Snapshots. And they are a breeze on operating systems where ZFS for everything is available. It's not like the Windows feature of the same name, which I suspect is in part what makes people wary of the idea. That works rather differently. A ZFS snapshot completes in seconds.

> Ask me how I know.

What problems did you have that made you want to roll back the update?

I had some containerized application software break and start misbehaving in odd ways which was indicative of a deeper incompatibility issue. Possibly GPU related. No time to debug, had to roll it back.

This was complicated by the fact that the machine also hosted a MySQL database which could not be easily rolled back because the database was versioned up during the upgrade.

Wait, the containerized application broke because of a host upgrade? That's some leaky container you have there.

This sounds like a business setting, so this sounds like a good opportunity to advocate for testing hardware, a testing budget, a rollout plan, and a sound backup strategy.

For me it's usually been GPU driver compatibility.

You should be using an lvm snapshot. You are not even making a valid complaint.

Reply on news.ycombinator.com