Linux package managers are slow | Hacker News Reader

Linux package managers are slow

michael.stapelberg.ch

46 pointsericdanielski63 comments

Compared to Windows, they are all blazing fast. I still wonder how it can take over an hour just to search for the latest Updates on a Windows 7/8.1 and/or Server 2012 (R2) Box.

An hour if you're lucky :-)

I think I haven't been able, in the last years, to update a Windows 7 SP1, without using hacks.

I'm not sure what you two are doing. My Windows 7 machine has no problem finding new updates in ~1 min and installs them just fine also.

Well, like I, they simply launch "Search for new updates" and "Install new updates", and each of those operations can take dozens of minutes. I mean, in the lucky cases the first one completes. The downloading step is the most infuriating, with a progress bar that never progresses for several minutes, although the download is just supposed to be a few tens or a few hundreds of Mo.

Fresh installs have a large (for Windows Update standards, which are rock-bottom) number of updates to download, compared to a currently updated one.

AFAIR (not 100% sure) some Windows Update fixes have been released after Windows 7 SP1 (which is the last available image), which means, in order to install a fresh system and update it, one has to install the O/S, then patch, then update.

So this explain at least some cases. Although, I remember that even after patching, WU would hang.

For fresh installs, sure I can imagine it taking a long time. The typical case is an existing install though, for which it should be as fast as I said.

Yes, it's quite a pity that Microsoft didn't put some weight behind it. It causes a never ending amount of pain to the average users too.

Anybody ever used the peer to peer local caching ?

> Yes, it's quite a pity that Microsoft didn't put some weight behind it. It causes a never ending amount of pain to the average users too.

Yes, I never understood why this was never improved across the years (and systems).

The only explanation with which I could come up, is that the team in charge is still the same team who designed progress bars in DOS and Win 3.1 times, and they have been granted the role of tradition keepers.

In addition to taking hours, the Windows update is terrible opaque. Between the 'old' update & new update interfaces, the time spent 'looking for updates' can range from a minute to over an hour which means it's obviously doing more than just querying a server for available updates. During the 'installing' updates phase, the percent complete will stall & jump at random intervals without any indication of why it suddently went from 15% done to 25% done.

And at the end of this long and ardouous process, you only have an updated OS and still have to manually (or use a tool like Chocolatey) update your non-MS software.

It baffles me that this as considered acceptable considering these issues are still present even in domain environments with local WSUS servers.

I'm not entirely sure about this (someone please correct me if I'm wrong), but it seems to me that on Windows the problem is fundamentally harder than on Linux. On Windows, you can (or at least could on Windows 7, where this time spent was particularly awful -- though I think you can still uninstall individual updates) pick and choose which updates you install (and uninstall). The sheer number of combinations of updates installed is exponentially more than anybody can test, yet that flexibility is given to you. On Linux you don't seem to have this kind of flexibility as far as I can tell. Now, precisely what computation has to occur that's so much more work in Windows than on Linux is unclear to me, but it does seem like Windows has to solve a fundamentally harder problem.

All the Linux package managers I've used (admittedly, not that many) allow you to choose which updates to install. It's only dependencies that should be upgraded together, I suppose.

But dependencies is exactly what I mean though. You can randomly uninstall an update to a common dependency on Windows. While keeping a later update that patches the same libraries installed. (!) I don't really get how they handle it (probably SxS magic), but somehow they do, whereas on Linux you can't do this.

Distributions based around static-linking rather than dynamic would allow doing this though, correct?

Or NixOS and Guix that allow you incredibly fine-grained control through configuration? (Pinning multiple versions down to particular hashes.)

Ah, right, that makes sense.

> On Linux you don't seem to have this kind of flexibility as far as I can tell.

I don't think that's true at all. I choose when my kernel gets updated, and which dkms modules I want to update (as some of them are a bit... bleeding edge). I choose which of my apps needs an update.

I can choose just security, or even 'just do this particular security update for this particular program'.

To clarify what dataflow is talking about, two Windows hotfix updates to the same component can be installed indepedently, or one can be installed and the other not.

This ability does not exist in Linux package managers since they deal with packages - there is only one newer package to install, and installing that new package replaces the entire existing package. Hotfixes work by applying patches to the affected components' files, which is why they can be independent of other hotfixes applied to the same components.

Hotfixes are also grouped into larger updates, the largest of which is a service pack. (You released N hotfixes over the last month and expect everyone to apply them in the future. There's no point making every user in the future download N updates where one batched up one would suffice.) So when Windows Update has to answer the question "Is this new update required or is it already installed via a previous update?" it has to do a bit of gymnastics to figure out the answer.

So both approaches have their pros and cons. The hotfix method means you can get targeted fixes with no feature updates and smaller chance of regressions, at the cost of being more complex to solve than "if server's version > installed version, download and overwrite".

For example, consider that users of component Foo discover that it crashes when it tries to access the internet with BarCorp FireWall 2019 installed. There might be a hotfix released that fixes Foo to not crash.

Since it only affects Foo users that are also BFW users, it might happen that the hotfix happens to break Foo users who have QuuxCorp Internet Optimizer installed instead. The testing might not have caught it since the primary purpose of the testing was to ensure that the issue is fixed for BFW users, with minimal sanity testing that it doesn't break non-BFW users.

So QIO users can mitigate themselves by uninstalling the hotfix. They didn't need it anyway. Eventually another hotfix may be released that supersedes the first one and is tested for both BFW and QIO, and this latter hotfix will eventually be bundled up into a general Foo update.

All this happens independently of another Foo hotfix released after the BFW one, to fix the issue where it crashes when you open a .txt that's actually a .bmp.

The equivalent with Linux packages would be to have .1, .2, .3 package releases, but QIO users can't can't keep the changes in .3 and remove the changes in .2.

Some distros allow give you some flexability. RHEL as AppStream which allows you to pin particular sotfware to a specific version while upgrading the rest of the system.

The equivilent of Windows SxS would be like running everything in it's own docker container. Assuming each one was updated concurrently & they used a shared package cache this could still be quite fast compared to Windows.

Even accounting for Window's 'impressive' ability to let you install any combination of updates/hotfixes. It's basic software engineering that the common case should be fast and considering that in 99.99% of Windows instances are going to have all of the updates installed, it's very apparent that the Windows update system & SxS were pooly designed and is an major area at Microsoft suffering from brain drain. I feel the most likely case is that no one currently at Microsoft (and still working as a programmer as many of the old talent are either retired or upper executives) understands it enough to fix the issue.

AppStream still replaces entire packages. It does not work like the hotfixes that I described.

WinSxS is not about hotfixes either, but for having different versions of components simultaneously installed.

Neither of these are relevant to the point of this subthread.

Thanks so much for elaborating on this! :)

That, and that I'm constantly being told that there are new windows updates available, which are really just Defender definitions updates. Just install them for me and stop bugging me!

Because windows update is downloading a multi-GB xml file (before compression) with info about all updates for all windows editions and other Microsoft products

I don't mind packages being installed sequentially, but I do wish they would be downloaded in parallel. It's very annoying to watch ~1500 packages, most of which are only a few KiB in size, download at the rate of one per second on a gigabit connection. (I'm on openSUSE, so zypper. I think yum downloads in parallel now after dnf improvements were merged into it?)

powerpill and apt-fast are handy for this, though they should really do this by default, yeah.

dnf does download the packages in parallel. You can configure the degree of parallelism in `dnf.conf`.

Look for `max_parallel_downloads` here:

https://dnf.readthedocs.io/en/latest/conf_ref.html

I think this is done more for the sake of the server sending the files, not the client side. Most Linux distros are using some form of donated mirror servers, and it’s not nice to slam them with requests.

That probably was the reason when package managers were invented. I don't think it makes sense today, especially when we do have some package managers downloading in parallel (like dnf as I mentioned). That means at least some distros believe their infrastructure can handle it.

As for volunteer servers, if a particular server can't handle >N connections, it can drop them. No reason to gimp clients instead.

And specifically in openSUSE's case, nobody has put forth server bandwidth / limits as a reason not to parallelize downloads. [1][2][3] It's just not been done because nobody (including myself) has cared enough to figure out how to do it.

[1]: https://features.opensuse.org/307862

[2]: https://github.com/openSUSE/zypper/issues/104

[3]: https://github.com/openSUSE/zypper/issues/160

- The "Pain point: no concurrency" thing seems wrong for Nix. By default, Nix isn't using any hooks at all, and it should be concurrent. The confusion might come from the fact that NixOS does have something kind of like hooks, but nothing makes you use them.

- The QEMU results are highly misleading, because there are tons of different configurations possible with QEMU. Nix builds all QEMU architectures while the Alpine one only is getting qemu-system-x86_64. It's apples to oranges.

- The Nix docker image is not actually running NixOS, but instead running Alpine. It just has the Nix package manager installed on top of that.

Pacman aint slow. Seems to basically run at the speed required to download and copy files. Then again it has no concept of required versions, since Arch just requires you to stay up to date. So there's no extensive solving of dependencies over version requirements, just the packages themselves.

When I was on Ubuntu though, which does have version requirements taken into account in packages, I didn't find apt to be slow either.

You know what's slow? The conda package manager. Or installing software - practically any software - on Windows. I regularly budget half an hour to an hour just to install something on the Windows computers I'm forced to use.

That's my experience as well: apt is only "slow" in that sometimes it has to download and write gigs of stuff to disk.

Now brew and conda, they often take minutes to solve for the environment, which is what I think of when you say "slow". That part is probably mostly algorithmic, and it should be fast, unlike network and disk.

The main performance problem in pacman as far as I can tell is really slow xz decompression. This is mostly important for large packages:

    $ time xzcat /var/cache/pacman/pkg/cuda-10.1.243-1-x86_64.pkg.tar.xz >/dev/null
    1:17,33 total

Just decompressing that 1.8GB package takes 77 seconds.

There was a discussion in April to switch compression to zstd - which reduces the decompression time for that package to only 4 seconds (!) while keeping the same file size. That way the disk is the bottleneck again, as it should be. That switch would be awesome, but it looks like it's not happening currently.

If you have a fast connection, xz decompression may be the bottleneck, but in some places, downloading that 1.8GB can easily take an hour or more. So I’m not sure whether this is the bottleneck in the usual case.

I would also be interested to see if it’s using parallel decompresssion when possible. I have an intuition it doesn’t, since I have always found Arch defaulting to single thread decompression, e.g in makepkg or for compressing with mkinitcpio.

link: https://lists.archlinux.org/pipermail/arch-dev-public/2019-M...

Put `apt-get update && apt-get install -y jq` at the top of your CI script which runs hundreds of times a day and then tell me that apt isn't slow. Should I have to maintain a container just for `FROM node:10 ; RUN apt install jq`? If I do so everybody will assume that it's some sort of specialized testing container rather than just a very generic `node:10` and think they can't run test scripts on their local machine. If node:10 was based on Alpine rather than Ubuntu the equivalent `apk install jq` is blazing fast.

1. You can and probably should base your image on node:10-alpine instead of node:10

2. Even if you don't, why would you have to maintain a separate container? If `RUN apt install jq` is indeed the second entry in your Dockerfile after the `FROM ...`, then it should be cached and reused indefinitely -- that's half the point of Docker in the first place. It won't work if you have a command like `COPY` before the `RUN apt`, though.

It's also not fair to show download speeds. Since the whole point of all of these is that there are a ton of mirrors.

Yesterday I had a nearby arch mirror fail and timeout. And then the secondary backup mirror downloaded things at 20M/s.

Also. just tried qemu on arch and there definitely wasn't 124mb of data transferred. (Total download size: 7.85mb) so this can vary heavily depending on which deps you have pre-installed for something elsewhere.

Doesn't seem to be the best methodology.

The author is probably clueless, not dishonest, in this particular mistake, but the Nix/NixOS installation command used is the slow one. Nix doesn't have versions for packages in the traditional sense. A fairer pair of test commands:

  nix-env -iA nixpkgs.ack
  nix-env -iA nixpkgs.qemu

This is because Nix considers an entire tree of Nix expressions at once, rather than examining each package in isolation. When nix-env is run without the -A flag, it must evaluate the entirety of nixpkgs and then search for the desired package amongst the complete tree. With -A instead, it is possible to evaluate only the portions of the tree which are relevant to the named package.

To be fair `nix-channel --update` does take a ridiculously long time:

  $ time nix-channel --update
  unpacking channels...
  created 2 symlinks in user environment
  nix-channel --update  1.92s user 16.58s system 15% cpu 1:56.62 total
  $ time nix-env -iA nixpkgs.ack       
  installing 'perl5.28.2-ack-3.0.2'
  these paths will be fetched (0.06 MiB download, 0.19 MiB unpacked):
    /nix/store/43rvd2kinid0g48ng07d782dsqb497g1-perl5.28.2-ack-3.0.2
    /nix/store/k3f819h2ncadkd6g8yzyzg24wn3vr609-perl5.28.2-ack-3.0.2-man
    /nix/store/p82srjihmh7vnq4szl9q83adcl9r6j2v-perl5.28.2-File-Next-1.16
  copying path '/nix/store/k3f819h2ncadkd6g8yzyzg24wn3vr609-perl5.28.2-ack-3.0.2-man' from 'https://cache.nixos.org'...
  copying path '/nix/store/p82srjihmh7vnq4szl9q83adcl9r6j2v-perl5.28.2-File-Next-1.16' from 'https://cache.nixos.org'...
  copying path '/nix/store/43rvd2kinid0g48ng07d782dsqb497g1-perl5.28.2-ack-3.0.2' from 'https://cache.nixos.org'...
  building '/nix/store/p765c3dxn32zccl4q9xjgr74n2ljgiky-user-environment.drv'...
  created 134 symlinks in user environment
  nix-env -iA nixpkgs.ack  0.19s user 0.24s system 36% cpu 1.158 total

Yeah, this is much speedier!

On my machine, for the original command I get:

   created 56 symlinks in user environment
   real 0m 25.64s

For your command I get:

   created 56 symlinks in user environment
   real 0m 4.39s

Context matters. There are some underlying assumptions not discussed here. You can see them in the content like...

> Pain point: too much metadata

and

> I expect any modern Linux distribution to only transfer absolutely required data to complete my task.

Consider the case of searching for a package. That search can be performed locally and you would need the metadata locally or it can be performed remotely and the results sent over the wire. It has to be performed somewhere.

If performed remotely than the remote endpoint knows what you've been searching for. If you are trying to perform a search across multiple package repos how do you to that securely? How do you not leak information? The remote endpoint can capture, log, and use information about who searched for what. Then tie that back to company and other data. There's a lot of sensitive corporate information that can be contained in these searches. So, do you do perform searches remotely or locally? Many systems choose locally but that entails downloads a dataset.

There are a bunch of things like this to take into account.

Other things, like serial downloads, are likely a product of the package managers age. I would hope modern ones work out parallel downloads.

You both have a point here. As a long time fedora user and occasional packager I always considered the sudden meta data downloads a terrible issue. On a slow connection not would basically disable your package manager due to timeouts.

I never looked into the technical details of yum/dnf metadata, but I suspect there must be a much smarter way to manage that stuff. Incremental updates for instance, or maybe just a different format (is it not xml?).

People are asking 'why does speed matter?'

If you use nix then you'll know that a “nix-shell -p” is basically the url-bar of the browser.

If a browser is a VM in your OS then why not run distri in a VM and speedily launch native apps, horribly useless example.. for now.

If the browser is eating the world anyway then incentives to have native apps be incompatible between operating systems fade away and throwaway envs like nix-shell (and to a lesser extent nixos-shell) become even more interesting. . So long as they are fast enough to feel as effortless as opening a web page.

The author combined fetch and install, which is just silly. Fetch times are affected by any number of things, such as the number of mirrors the package repos have, whether you have a mirror near you, whether your system is configured to use it, and your current connection speed.

Both fetch and install are useful to know, but for very different reasons. Only so much of fetch speed is in control of the distro setting it up (but you can set up your own mirror if you like...), but install speed once the files are already staged locally is also useful to know. Additionally, whether the package manager keeps a local cache of metadata for packages, and whether it checks and updated it every request or only at certain intervals or when requested will affect this as well (and is very important, if you want to know about security updates immediately and not after some interval of hours).

What the author has done is the equivalent to testing how fast certain one mile and 10 mile stretches of highway are, but without controlling for time of day, and not realizing rush hour is invalidating all the results. The first step to measuring something is to learn about what you're measuring so you can avoid issues like these.

Arch has a much longer dependencies list for qemu than Alpine does[0, 1]. Arch has 7 sub-packages to qemu, while Alpine has 82. This suggests to me that Arch's base package includes functionality that Alpine has placed into subpackages. Not all packages named "qemu" are created equal.

Also, without separating out the time spent waiting for the download, the author's choice of mirrors probably has a significant impact.

0: https://www.archlinux.org/packages/extra/x86_64/qemu/

1: https://pkgs.alpinelinux.org/package/edge/main/x86_64/qemu

For me, the thing that's missing from the article is the 'so what?'

If Linux package managers were faster then <it would affect me how?>

Agreed! Free software and the complaint is you cannot download it faster than an Encyclopedia Britannica once a second.

You can customise the sources, pay attention to what you want installed/upgraded... or even run a mirror

Someone else mentioned concurrent downloads which sounds like a good idea, otherwise... thankful.

I'd be happier with not having to wait so long for dnf metadata to download all the time. Pretty much my job (at Red Hat!) is to wait for it a lot of the time.

Isn't the main reason dnf metadata is so massive that it contains the filelists of every package in repository? No other package manager requires that information to work. Does anyone else at Red Hat (such as the dnf developers) consider the metadata size a major problem?

The filelists are a good thing since they let you do:

   dnf install /usr/share/mypackage/file

However they are not downloaded if you're just installing a package by name (even for dependencies) which is what this test is all about.

Then what constitutes the bulk of the metadata? I seem to recall it being much smaller back in the days of yum (~20mb), and the number of packages in the repository has not ballooned that much since then.

The main use cases I can think of for a super fast and lean package manager are (1) speeding up docker image builds (only matters when new dependencies are added), (2) folks with slow / significantly metered internet connections, and (3) embedded systems / low resource machines.

Otherwise 20-60 seconds saved doesn't matter to me personally for a one-time event. I'm guessing it doesn't matter to the package-manager developers either.

Always strive for excellence. For something as fundamental as this, it's a bit shameful that mainstream package managers are so unnecessarily slow.

Let's say there are 30 million Linux users who are wasting an hour every year on slow package managers. 30 million hours means 3424 years wasted every year or the total average lifespan of 48 people. Package managers kill 48 people per year. ( /s )

(There's some related Steve Jobs quote here somewhere.)

Like the article notes, the Alpine Linux package manager (apk) is refreshingly fast.

It's odd that Alpine lets you install e.g. "emacs-nox" on an IoT device connected via crappy LTE, on the other side of the world, faster than you can install it on your local Ubuntu dev machine, with like 100-1000x more network bandwidth, 10-100x more CPU and 100-1000x faster local storage.

Based on these benchmarks, you are including the time it takes to download packages, which is very unreliable since it varies widely based on mirrors. I've installed QEMU on Arch before in a much shorter time. Can confirm though that Alpine has very consistent download times, while sometimes with Arch I will have to change my mirror due to poor speed.

I'm using a local apt-cacher-ng to save bandwidth, and apt really flies - I'd recommended it if you have (obviously) 2+ (debian, Ubuntu or derivatives) machines. Makes even the occasional network slowdown much more tolerable... Just disable it when doing dist-upgrade or do-release-upgrade.

I think to go really fast, you'll have to introduce a new data structure, similar to git's. (I'm thinking it's a trie, but I'm coming up zero evidence for that.) Calculating the dependency graph could be nearly instantaneous with an improved data structure on the client. Syncing the local tree with the remote authority can also be blazingly fast. After that, if you really want to be smart, you can use an rsync style transfer that only copies differences. And if you want to boil the whole ocean, make all the package maintainers conform to a standard that reduces installation to a pure copy operation.

It'd be awesome if updating a node was a sub-second operation, it's definitely a big benefit to cloud based computing.

There's another flaw besides the many flaws that others have mentioned. As the author says, the amount of metadata downloaded is different between package managers. Perhaps ideally, only the metadata needed for a requested package would be downloaded. But I don't believe that's what's happening.

Alpine is fastest not just because it has a more efficient metadata format, but also because it has far fewer packages than the others.

For some reason (probably transparency), rather than using specialized docker images the GitLab auto-devops scripts install needed packages into a generic container. This really slows down the CI pipelines and is super painful

At least the ones where the generic container is Ubuntu/Debian based. `apt update && apt install -y` is really slow. But alpine based containers with the equivalent `apk install` are blazing fast.

GitLab PM here. Auto DevOps makes use of herokuish in order to find the language of your project. Herokuish images are Ubuntu-based, and it doesn't allow the use of specialized images. One can always provide a dockerfile (based on the image of your choice) and Auto DevOps will use that instead.

While a comparison like this may be interesting trivia, it's irrelevant for all practical purposes.

Consider the amount of time it takes to update a system vs. its overall runtime. Now consider what's important in a package manager. Consistency, rollback, configuration management, audit trails, etc. The speed of the package manager is at the absolute bottom of things to be concerned about.

Maybe for you, but there are use-cases and situations where this adds up, and has monetary effects. Container builds, or CI pipelines that run often would be one example. Sure, the things you mention are important, but all other things being equal, everybody would prefer faster installs to slower ones, right? So it's a fair thing to worry about, me thinks.

Meanwhile Gentoo's portage took almost 7 minutes to build qemu...

    zx2c4@thinkpad ~ $ genlop -t qemu
         Sat May 18 15:36:44 2019 >>> app-emulation/qemu-4.0.0-r2
           merge time: 6 minutes and 49 seconds.

Still faster than Windows and Apple updates

Reply on news.ycombinator.com