I highly recommend it to anyone else like me, who is generally cranky about new things. They did a really great job with Arch. This policy you mention is a great example of what I like about it.
* after "BootHole" GRUB vulnerability, I've read that upgrade requires re-installation of the bootloader. And just in case, did run `sudo grub-install …`. After reboot, system didn't boot. Had to use installation USB to restore.
* More recently, during routine upgrade pamac (Manjaro's pacman alternative) GUI showed me some "transaction can't be completed" error message. Shuddered it away - few days later pamac wasn't starting at all. Starting it from terminal showed error message about some missing *.so file. Googling showed that this is a required file for pacman (Arch package manager) to function. Typing `pacman` in terminal showed "command not found" error message. Restored the missing *.so file from snapper snapshot (thanks btrfs!), after that pamac started fine and happily upgraded my system.
I'm not sure what happened in second case and why pamac left system in broken state (looks like it wanted to upgrade pacman by first removing old files and then putting new files in place, but aborted in the middle), but first one might be quite distro-independent.
Also, reading through recent Arch news, I believe this could bite someone:
https://archlinux.org/news/sshd-needs-restarting-after-upgra...
> After upgrading to openssh-8.2p1, the existing SSH daemon will be unable to accept new connections. When upgrading remote hosts, please make sure to restart the SSH daemon right after running pacman -Syu. If you are upgrading to openssh-8.2p1-3 or higher, this restart will happen automatically.
FWIW, as a vanilla Arch user, I have not encountered this issue. I remember a time when pacman updates were kind of iffy (and in fact, pacman itself asked you to update it first before proceeding with the rest of the updates), but since 5.0, all subsequent updates have been completely unremarkable in the best way.
- Arch will release a new version of Xorg before the graphics card vendors release new drivers that are compatible with it. Same for kernel versions too.
- I've had gedit start crashing in new minor versions of gnome due to a setting being incompatible, needing to track that down and unset
- If you have python virtualenvs for development and system python is upgraded to a new major version, all your virtualenvs break
- If arch upgrades the major version of glibc during some random package install (rather than a system upgrade), and you don't upgrade the whole system at once, every app that didn't get upgraded will fail to start due to soname mismatches... and that can mean that pacman, sudo, etc are all busted (this has actually happened to me)
- If you do a full system upgrade and have AUR stuff installed, you need to be sure to upgrade the AUR stuff otherwise it could break due to being incompatible with any library that was upgraded.
- In general (not specific to arch linux), new versions of software break stuff all the time. You tend to hit way more of this on arch if you keep your system up to date.
Okay I just wanna point something out here, partial upgrades in Arch are broken by design because they simply can't work. Don't do it.
That being said...
> - Arch will release a new version of Xorg before the graphics card vendors release new drivers that are compatible with it. Same for kernel versions too.
... kernel ABI is stable and even ancillary interfaces are usually stable, so usually it's quite A-OK to not immediately upgrade the kernel.
That said, the vast majority of the time only upgrading a particular application/package (and its dependencies) will work just fine. It's just that there are no (official) guarantees.
Debian's way of working is labour intensive, requiring packagers to fork, follow and maintain fixes and security issues in the version of the software they're packaging. They generally do a great job, but this is not a sustainable approach for smaller distributions. Arch Linux on the other hand follows upstream in real time, so you get the latest fixes directly from upstream, but there is no real Arch Linux "version buffer" that allows to freeze the versions of (parts of) your system. You move with the stream, that's sort of the philosophy of rolling distributions like Arch Linux.
The real trick is that Debian packages have sonames in their versions, so ABI compatibility is encoded in package dependencies. So when I try to downgrade a library (to a known older version from snapshot.debian.org), it knows precisely which packages depend on the new version and forces them to be downgraded as well.
This is entirely orthogonal to following upstream vs. backporting patches. Sure, Debian (stable) does backport patches, which makes it more likely that single package downgrades don't downgrade half of the system, but it really is a different thing. Debian testing/unstable follow upstream to a larger extent than Debian stable, and upstream fixes are usually preferred to patch backports. Still, partial upgrades and downgrades almost always work without trouble.
> - Arch will release a new version of Xorg before the graphics card vendors release new drivers that are compatible with it. Same for kernel versions too.
I imagine this is mainly a problem if you have an Nvidia card (which would explain why I haven't had the problem).
> - In general (not specific to arch linux), new versions of software break stuff all the time. You tend to hit way more of this on arch if you keep your system up to date.
More often, sure. But for software delivery from the production side, the common viewpoint seems to be that deploying more often leads to less pain in total, because you're making smaller increments so the individual failures are smaller as well.
As such, aside from integration issues like Xorg and the drivers, I would expect Arch to have fewer major breakages than (e.g.) using LTS releases of some distro and upgrading every second year.
This is mostly true for production infra (at least for your own software in production infra) but I'm not sure this logic flies for personal machines and software you don't directly interact with. I have used arch for at least 7 years and I have had upgrades render my system unbootable (or nearly unbootable, like Xorg/GDM/Gnome failing to start). These issues were mostly not due to my configuration needing to change, they were due to breakages between different packages that would end up being fixed in some later minor version.
As an end-user upgrading distros every 2 years, I really doubt I'd hit many major breakages each time as all of those pieces of software will have been in the wild for a bit and major bugs will have likely been fixed. I think the system-level issues that are resolved by frequent deploys are stuff like "systemd deprecated setting X in service unit files, so I need to update my config" or "library xyz changed some API so I need to update my app", etc. With linux distros that have coordinated releases like fedora/ubuntu/debian/etc my software's interaction with the distro may break, but for the most part the major inter-package relationships within the distro get some amount of coordinated testing which doesn't happen with rolling release distros.
Put another way: deploying more often is great for quickly discovering problems introduced in software that you write. Deploying every single dependency in a linux distro more often is going to cause you to hit every bug destined to be fixed in some minor version of the software in pieces of code that you are very far away from and lack the context to quickly debug. So you will hit a much higher sum total of bugs that would have otherwise ended up getting fixed whether you personally hit them or not. However, if you are developing a linux distribution itself, then yes your CI infra should be constantly upgrading dependencies.
I'm not convinced that an arch-paced rolling release at a distro level will ever reach a point where inter-package dependencies do not cause totally unexpected bugs given just how many inter-library dependencies there are. The entire OSS ecosystem would need to write a ton more tests for this to be a reality, and distros would need to run that full suite of tests any time and dependency is upgraded. And even then, there's still so much possibility for breakages given that shared libraries don't do a fantastic job of versioning and the upstream vendors can't possibly ensure their software works against versions of a library they never ran it against. There's a linus torvalds post about this: https://lwn.net/ml/linux-kernel/CAHk-=whs8QZf3YnifdLv57+FhBi...
Despite all of this, Arch is actually amazingly stable, and the entire community is probably better off due to the existence of arch and arch users. We might be the best integration test there is for OSS software.
> - Arch will release a new version of Xorg before the graphics card vendors release new drivers that are compatible with it. Same for kernel versions too.
You accidentally put a plural in "vendors", but in practice this is just Nvidia.
If you are forced by circumstance to deal with an Nvidia card, use the LTS kernel and most of your problems go away, and you can also pin your X.org version and manually update it. The real solution is to use a GPU vendor that has mainline drivers, the only valid reason nowadays not to do that being CUDA or being unable to acquire hardware with mainline support.
> - I've had gedit start crashing in new minor versions of gnome due to a setting being incompatible, needing to track that down and unset
That's a gedit bug. Complain to gedit devs or stop using it.
> - If you have python virtualenvs for development and system python is upgraded to a new major version, all your virtualenvs break
You should not be using system python for non-system tasks. Use asdf, Nix, Docker, pyenv or a similar tool for projects requiring their own non-system environment.
> - If arch upgrades the major version of[...]
Partial upgrades are explicitly unsupported by the distro. Pacman allows you to do this, but you're on your own.
> - If you do a full system upgrade and have AUR stuff installed
AUR is not Arch, it's up to each AUR maintainer to keep their scripts up to date and you as a user to keep up with those external dependencies. A common approach is to use an AUR helper to handle both system and AUR upgrades.
> - In general (not specific to arch linux), new versions of software break stuff all the time. You tend to hit way more of this on arch if you keep your system up to date.
And this is why as an Arch user you will be nudged by experience to use software that gives a crap about quality.
> Partial upgrades are explicitly unsupported by the distro. Pacman allows you to do this, but you're on your own.
Definitely correct, the issue is when you're trying to install a package to get something done, and suddenly your faced with the proposition of either upgrading your whole system while trying to get work done or attempting to upgrade just the required libraries. The latter often carries less risk, but occasionally is very problematic.
> And this is why as an Arch user you will be nudged by experience to use software that gives a crap about quality.
The issue is that this is true of so many very major pieces of linux software. Upstream vendors don't necessarily test their stuff in a ton of contexts, plenty of bugs occur in the kernel/xorg/wayland/gnome/glibc/python/etc. All of these are very mainstream projects that are difficult to avoid.
I found more problems using e.g. debian stable. Upgrades were totally black box and anything could break spectacularly. This was most commonly caused by mixing up to date software you build yourself and the stable repos. Arch on the other hand updates happen daily, and for me, unattended. If I ever do find issues (rare to never), backups fix it.
For any definition of stable.
I may be wrong, as my default is to -Syu any package I install, but doesn't -S install the version of the package compatible with the last time you updated your repos? It's -Sy that can cause problems.
> Partial upgrades are explicitly unsupported by the distro.
The problem is obvious. Sometimes you might have to pin a package version because the newer version does not work with your hardware driver, as you said, but then again Arch calls this unsupported.
I had the same problem in the past, not with Nvidia but with DisplayLink. For that reason I always leave a USB stick with a bootable Arch ISO in my Thinkpad, so that I can arch-chroot and repair the system. It happens rarely, but it happens to me and others as well.
this was what convinced me to switch to Debian back in the 90ies. It was the only distribution that could do this without throwing a tantrum. I was coming from RedHat and SuSE and both had huge problems with this. Debian presented a higher learning curve (at least that was what people said). Debian really only lagged behind back then with their graphical installer and the overall lack of integrated DE back then (compared to others).
I really like the Arch wiki as a Debian user but never used Arch itself. Not going to change because old dog, new tricks ...
I've also been using Arch on a "for messing around; I don't care if it breaks" laptop for about two years, and haven't had any major breakages.
For me, the most noticable difference has been that Debian will install new kernels as new packages, will suggest removing old kernel packages, but won't suggest removing the currently booted kernel. I like this behavior. By default, Arch will swap your currently booted kernel and modules out from under you. You don't necessarily need to reboot right away, but if you don't, you can run into issues loading modules or recovering from hibernation.
You can work around this once you realize what's happening, but it seems like a needlessly dangerous default.
My favorite documentation is the arch wiki and there has never been an issue due to it not being compatible to the "Debian way"
When upstream devs package for a distribution they usually put some effort into it. Just because a package gets published to Sid/unstable doesn't mean the package is unstable. There are some devs that mostly work on higher layer GUI and user-lamd applications who are perhaps inexperienced or reckless who do sometimes introduce breaking changes. It's rare though since most people understand that packaging for a distro means a potential huge number of issues if they're not careful
My 70+ year old aunt also runs sid since a couple of years with unattended upgrades since I installed it for her (and she has KDE). She never had issues with stability or things not working and she uses her computer daily for writing (libreoffice), printing and research (probably reddit idk, I didn't ask :)).
[1] my install is fairly minimal: not much gui, server systems etc are started with docker so hardly anything actually runs on bare-metal which can cause issues during upgrade jeopardizing the things I work on (e.g. due to config file changes). And I don't have a huge DE like gnome/KDE. My sway/wlroots and i3, and tooling like x-terminal/wofi/Waybar/etc are always compiled from git/source. All my dev tooling is the latest and I can still install older versions of clang/gcc or other environments with my package manager if I have to.
When several of us raised the issue in the forum...I'll just say we didn't get a warm reception. That was more than a decade ago, so things may be completely different now, but it left a bad taste in my mouth.
I've been using Arch for years and I never experienced any breakage. I update it every month or so and it's still stable even after these huge updates. There's nothing for me to do other than merge .pacnew files.
For the topic I think is good to have dfsg and to patch any software with the goal to provide better integration with the system and for user's freedom.
As engineers we have no one but ourselves to blame for this one
This has drawbacks too though, in that it's now up to app makers to take care of keeping supporting libraries up to date and secure. That's not necessarily their top priority though, which is a risk for the end user. Also, unnecessary duplication of libraries increasing memory use, when the distro is able to share most of them. Plus the poor middleman has an incentive to set user-friendly, privacy-preserving defaults that I wouldn't trust as much when the package is provided by a commercial company with different incentives.
It's great to have the option, but overall I would still prefer to use distro packages except in the rare special case.
The nice thing about Arch packages, though, is that they're basically just bash scripts. And if all you need is a simple version bump, it's usually quite easy: download the package tarball from [0] and change [1] to the version you want, then do a `makepkg -s` in the directory of the package. It will build the package in a (usually reproducible) chroot. If it builds without errors, then you'll end up with a tarball that you can `pacman -U ${pkg}.tar.zst` to install the produced files.
If you need help, makepkg documentation on the wiki[2] is pretty great. And don't forget to send a patch to arch-dev-public[3] and CC the maintainer. At the very least, it'll kick off a discussion that will get the package updated.
Rolling your own packages is easy in contrast to, say, Red Hat - where compiling an RPM is easy if you've done it a bunch, but really difficult to get bootstrapped on.
[0]: https://archlinux.org/packages/extra/x86_64/jdk-openjdk/
[1]: https://github.com/archlinux/svntogit-packages/blob/packages...
https://lists.archlinux.org/pipermail/arch-general/2021-May/...
https://lists.archlinux.org/pipermail/arch-general/2021-May/...
This is incorrect. Packages in both core and extra are maintained by the core Arch developers. You're probably thinking of the community repository, which is maintained by a group called Trusted Users. These are still heavily vetted, it's not just anyone from the community. Or maybe you're thinking of the AUR, which is a completely untrusted repository of build scripts, which anyone can submit to.
In this case, the issue is that anthraxx, one of the Arch developers [1], has not updated many of their packages in some time. I don't know if a reason is known, but you might find something in the mailing list links someone else has posted.