What part of it breaks for you? It's not that I don't believe you, and I've seen this sentiment before, but I've used Arch as my daily driver for about five years, and can't recall a single time where it broke from a system update.
- Arch will release a new version of Xorg before the graphics card vendors release new drivers that are compatible with it. Same for kernel versions too.
- I've had gedit start crashing in new minor versions of gnome due to a setting being incompatible, needing to track that down and unset
- If you have python virtualenvs for development and system python is upgraded to a new major version, all your virtualenvs break
- If arch upgrades the major version of glibc during some random package install (rather than a system upgrade), and you don't upgrade the whole system at once, every app that didn't get upgraded will fail to start due to soname mismatches... and that can mean that pacman, sudo, etc are all busted (this has actually happened to me)
- If you do a full system upgrade and have AUR stuff installed, you need to be sure to upgrade the AUR stuff otherwise it could break due to being incompatible with any library that was upgraded.
- In general (not specific to arch linux), new versions of software break stuff all the time. You tend to hit way more of this on arch if you keep your system up to date.
Okay I just wanna point something out here, partial upgrades in Arch are broken by design because they simply can't work. Don't do it.
That being said...
> - Arch will release a new version of Xorg before the graphics card vendors release new drivers that are compatible with it. Same for kernel versions too.
... kernel ABI is stable and even ancillary interfaces are usually stable, so usually it's quite A-OK to not immediately upgrade the kernel.
That said, the vast majority of the time only upgrading a particular application/package (and its dependencies) will work just fine. It's just that there are no (official) guarantees.
Debian's way of working is labour intensive, requiring packagers to fork, follow and maintain fixes and security issues in the version of the software they're packaging. They generally do a great job, but this is not a sustainable approach for smaller distributions. Arch Linux on the other hand follows upstream in real time, so you get the latest fixes directly from upstream, but there is no real Arch Linux "version buffer" that allows to freeze the versions of (parts of) your system. You move with the stream, that's sort of the philosophy of rolling distributions like Arch Linux.
The real trick is that Debian packages have sonames in their versions, so ABI compatibility is encoded in package dependencies. So when I try to downgrade a library (to a known older version from snapshot.debian.org), it knows precisely which packages depend on the new version and forces them to be downgraded as well.
This is entirely orthogonal to following upstream vs. backporting patches. Sure, Debian (stable) does backport patches, which makes it more likely that single package downgrades don't downgrade half of the system, but it really is a different thing. Debian testing/unstable follow upstream to a larger extent than Debian stable, and upstream fixes are usually preferred to patch backports. Still, partial upgrades and downgrades almost always work without trouble.
> - Arch will release a new version of Xorg before the graphics card vendors release new drivers that are compatible with it. Same for kernel versions too.
I imagine this is mainly a problem if you have an Nvidia card (which would explain why I haven't had the problem).
> - In general (not specific to arch linux), new versions of software break stuff all the time. You tend to hit way more of this on arch if you keep your system up to date.
More often, sure. But for software delivery from the production side, the common viewpoint seems to be that deploying more often leads to less pain in total, because you're making smaller increments so the individual failures are smaller as well.
As such, aside from integration issues like Xorg and the drivers, I would expect Arch to have fewer major breakages than (e.g.) using LTS releases of some distro and upgrading every second year.
This is mostly true for production infra (at least for your own software in production infra) but I'm not sure this logic flies for personal machines and software you don't directly interact with. I have used arch for at least 7 years and I have had upgrades render my system unbootable (or nearly unbootable, like Xorg/GDM/Gnome failing to start). These issues were mostly not due to my configuration needing to change, they were due to breakages between different packages that would end up being fixed in some later minor version.
As an end-user upgrading distros every 2 years, I really doubt I'd hit many major breakages each time as all of those pieces of software will have been in the wild for a bit and major bugs will have likely been fixed. I think the system-level issues that are resolved by frequent deploys are stuff like "systemd deprecated setting X in service unit files, so I need to update my config" or "library xyz changed some API so I need to update my app", etc. With linux distros that have coordinated releases like fedora/ubuntu/debian/etc my software's interaction with the distro may break, but for the most part the major inter-package relationships within the distro get some amount of coordinated testing which doesn't happen with rolling release distros.
Put another way: deploying more often is great for quickly discovering problems introduced in software that you write. Deploying every single dependency in a linux distro more often is going to cause you to hit every bug destined to be fixed in some minor version of the software in pieces of code that you are very far away from and lack the context to quickly debug. So you will hit a much higher sum total of bugs that would have otherwise ended up getting fixed whether you personally hit them or not. However, if you are developing a linux distribution itself, then yes your CI infra should be constantly upgrading dependencies.
I'm not convinced that an arch-paced rolling release at a distro level will ever reach a point where inter-package dependencies do not cause totally unexpected bugs given just how many inter-library dependencies there are. The entire OSS ecosystem would need to write a ton more tests for this to be a reality, and distros would need to run that full suite of tests any time and dependency is upgraded. And even then, there's still so much possibility for breakages given that shared libraries don't do a fantastic job of versioning and the upstream vendors can't possibly ensure their software works against versions of a library they never ran it against. There's a linus torvalds post about this: https://lwn.net/ml/linux-kernel/CAHk-=whs8QZf3YnifdLv57+FhBi...
Despite all of this, Arch is actually amazingly stable, and the entire community is probably better off due to the existence of arch and arch users. We might be the best integration test there is for OSS software.
> - Arch will release a new version of Xorg before the graphics card vendors release new drivers that are compatible with it. Same for kernel versions too.
You accidentally put a plural in "vendors", but in practice this is just Nvidia.
If you are forced by circumstance to deal with an Nvidia card, use the LTS kernel and most of your problems go away, and you can also pin your X.org version and manually update it. The real solution is to use a GPU vendor that has mainline drivers, the only valid reason nowadays not to do that being CUDA or being unable to acquire hardware with mainline support.
> - I've had gedit start crashing in new minor versions of gnome due to a setting being incompatible, needing to track that down and unset
That's a gedit bug. Complain to gedit devs or stop using it.
> - If you have python virtualenvs for development and system python is upgraded to a new major version, all your virtualenvs break
You should not be using system python for non-system tasks. Use asdf, Nix, Docker, pyenv or a similar tool for projects requiring their own non-system environment.
> - If arch upgrades the major version of[...]
Partial upgrades are explicitly unsupported by the distro. Pacman allows you to do this, but you're on your own.
> - If you do a full system upgrade and have AUR stuff installed
AUR is not Arch, it's up to each AUR maintainer to keep their scripts up to date and you as a user to keep up with those external dependencies. A common approach is to use an AUR helper to handle both system and AUR upgrades.
> - In general (not specific to arch linux), new versions of software break stuff all the time. You tend to hit way more of this on arch if you keep your system up to date.
And this is why as an Arch user you will be nudged by experience to use software that gives a crap about quality.
> Partial upgrades are explicitly unsupported by the distro. Pacman allows you to do this, but you're on your own.
Definitely correct, the issue is when you're trying to install a package to get something done, and suddenly your faced with the proposition of either upgrading your whole system while trying to get work done or attempting to upgrade just the required libraries. The latter often carries less risk, but occasionally is very problematic.
> And this is why as an Arch user you will be nudged by experience to use software that gives a crap about quality.
The issue is that this is true of so many very major pieces of linux software. Upstream vendors don't necessarily test their stuff in a ton of contexts, plenty of bugs occur in the kernel/xorg/wayland/gnome/glibc/python/etc. All of these are very mainstream projects that are difficult to avoid.
I found more problems using e.g. debian stable. Upgrades were totally black box and anything could break spectacularly. This was most commonly caused by mixing up to date software you build yourself and the stable repos. Arch on the other hand updates happen daily, and for me, unattended. If I ever do find issues (rare to never), backups fix it.
For any definition of stable.
I may be wrong, as my default is to -Syu any package I install, but doesn't -S install the version of the package compatible with the last time you updated your repos? It's -Sy that can cause problems.
> Partial upgrades are explicitly unsupported by the distro.
The problem is obvious. Sometimes you might have to pin a package version because the newer version does not work with your hardware driver, as you said, but then again Arch calls this unsupported.
I had the same problem in the past, not with Nvidia but with DisplayLink. For that reason I always leave a USB stick with a bootable Arch ISO in my Thinkpad, so that I can arch-chroot and repair the system. It happens rarely, but it happens to me and others as well.
this was what convinced me to switch to Debian back in the 90ies. It was the only distribution that could do this without throwing a tantrum. I was coming from RedHat and SuSE and both had huge problems with this. Debian presented a higher learning curve (at least that was what people said). Debian really only lagged behind back then with their graphical installer and the overall lack of integrated DE back then (compared to others).
I really like the Arch wiki as a Debian user but never used Arch itself. Not going to change because old dog, new tricks ...
* after "BootHole" GRUB vulnerability, I've read that upgrade requires re-installation of the bootloader. And just in case, did run `sudo grub-install …`. After reboot, system didn't boot. Had to use installation USB to restore.
* More recently, during routine upgrade pamac (Manjaro's pacman alternative) GUI showed me some "transaction can't be completed" error message. Shuddered it away - few days later pamac wasn't starting at all. Starting it from terminal showed error message about some missing *.so file. Googling showed that this is a required file for pacman (Arch package manager) to function. Typing `pacman` in terminal showed "command not found" error message. Restored the missing *.so file from snapper snapshot (thanks btrfs!), after that pamac started fine and happily upgraded my system.
I'm not sure what happened in second case and why pamac left system in broken state (looks like it wanted to upgrade pacman by first removing old files and then putting new files in place, but aborted in the middle), but first one might be quite distro-independent.
Also, reading through recent Arch news, I believe this could bite someone:
https://archlinux.org/news/sshd-needs-restarting-after-upgra...
> After upgrading to openssh-8.2p1, the existing SSH daemon will be unable to accept new connections. When upgrading remote hosts, please make sure to restart the SSH daemon right after running pacman -Syu. If you are upgrading to openssh-8.2p1-3 or higher, this restart will happen automatically.
FWIW, as a vanilla Arch user, I have not encountered this issue. I remember a time when pacman updates were kind of iffy (and in fact, pacman itself asked you to update it first before proceeding with the rest of the updates), but since 5.0, all subsequent updates have been completely unremarkable in the best way.