Using NetBSD’s pkgsrc everywhere I can | Hacker News Reader

Using NetBSD’s pkgsrc everywhere I can

rubenerd.com

104 pointsjayp141879 comments

Other package management systems do one or two of the following things, but I haven't found any that can do all three at once, other than pkgsrc:

1) have good availability of pre-built binary packages for most platforms

2) facilitate building of packages directly from source, whether on non-mainstream platforms / architectures, and / or with or for the purpose of non-default package options

3) unprivileged builds / installations, packages rooted anywhere in filesystem, completely self-contained dependencies

Having to use more than one package management system because one or the other doesn't do one of these things can be a real pain, and can make maintenance rather difficult.

I think Archlinux's PKGBUILDs can do that (https://wiki.archlinux.org/title/Creating_packages#Creating_...):

- With both the community-maintained packages and the AUR packages, you can already find almost any binary that already exists

- If it doesn't exist, or if you need particular options/architectures, a PKGBUILD is basically a glorified Makefile with standard steps, so you can for example get binaries from a http source and use them directly, or pass exotic compilation flags during the build() phase

- It is of course possible to put dependencies inside a given package

- preparing a package is entirely done by your user, no need for root right. Inside the package you put your files in a fake root, and when the package is done you apply the package to your real fs. By default it is installed under /, so of course you need root rights, but you can install it under any root you desire. This is actually what's done when installing Arch: it basically runs pacman (the package manager) but instead of installing packages in the live environment's root, it installs packages in the mounted filesystem that will become your root after rebooting.

We are missing unprivileged builds and clean chroot builds for `makepkg`. Our interal devtools uses `systemd-nspawn` which requires root. But that should be easy to implement with the leftover code we tried to use with our reproducible builds tooling.

https://github.com/archlinux/archlinux-repro/pull/70

Maybe a weekend hack to wrap `makepkg` with this.

nix does all three of these, I think.

It is nowhere near as cross-platform as pkgsrc - it only really supports macOS and Linux.

I love Nix, but this is true. The cross-platform support for pkgsrc is outstanding, and Nix's is still relatively limited.

Nix has better cross-compilation support is a little bit better than its native support, in terms of cross-platform coverage. But even then, I think, pkgsrc's platform coverage is in a different league. :)

It looks interesting. However, documentation is sparse. It doesn't clearly say what platforms are supported other than macOS and NixOS (which implies Linux in general, but doesn't clearly state it). Also, it seems to only target x86 and amd64. No mention is made of any other architecture, nor of any BSD other that macOS.

Do you have more information?

From <https://nixos.org/manual/nix/stable/#ch-supported-platforms>:

> Nix is currently supported on the following platforms:

> - Linux (i686, x86_64, aarch64).

> - macOS (x86_64).

How does one bootstrap nixpkgs from source? Trying to install nix gives:

sh install-nix-2.3.10 install-nix-2.3.10: sorry, there is no binary distribution of Nix for your platform

There's partial support for platforms besides macOS and Linux, but I'm not sure the state of it or how practical it is.

What platform are you hoping to build Nix on?

I guess 3 is sort of anti-nix, but I think nix effectively has this with the way nix-build/nix-shell work.

Unprivileged works now?

I've been using pkgsrc to package our firm's stuff for Mac OS X clients, it works a charm.

https://www.anserinae.net/setting-up-a-pkgsrc-repository.htm...

Currently working on using it to distribute the same software to Linux. Will be great when it gets there, but we aren't quite ready yet.

Very interesting, I didn't know pkgsrc could be used on Linux, I'll check it out. In general the only things keeping me from BSD are non free apps like Zoom so the next best thing is to BSDify my systems.

I'd try nixpkgs before trying pkgsrc if for some reason I wasn't satisfied with ones that come with the distro. Not sure why BSDfying packages is desirable by itself.

One reason to try it might be that pkgsrc mostly solves the problems Nix solves with conflicts in a much simpler way, by allowing multiple co-existing branches installed to different prefixes.

Nothing against nixpkgs, but if the user is curious about how things are done in BSD perhaps he should try xbps before nixpkgs. It was written by a former NetBSD committer who wrote some useful programs for NetBSD's userland. I am biased perhaps but I believe it is faster than any other Linux package manager. Always looking for something faster so I am welcome to being proven wrong on that.

https://github.com/void-linux/xbps

As a pkgsrc committer, xbps is nothing like pkgsrc. The underlying stuff is much more similar to something like Arch or Alpine Linux than anything that exists in the BSD world, and it's much less configurable than pkgsrc.

Care to be more specific than "underlying stuff". The code and design of xbps looks nothing like the code or design of pacman or apk.

Actual package recipes. BSD-style packaging systems are written in BSD make using a mk/ framework rather than Arch-style shell scripts. If a package already exists for FreeBSD Ports I would check that first because it's much more similar to what I'm used to.

Fundamentally pkgsrc is built on very low level tools, awk, shell, make, cwrappers, and pkg_install, although you can manage it with high-level ones (obviously pkgin is there, but there's also a lot of other third-party tooling).

Why are you comparing pkgsrc to pacman. The orginal comparison suggested in the parent comment was between xbps and nixpkgs or other Linux package managers.

pkgsrc boostraps itself using a program called "boostrap" in the pkgsrc directory. It builds a version of NetBSD's GCC toolchain and has no reliance on the host's userland. IME, this is very reliable and is its single greatest strength. Beyond that, pkgsrc is only as good as the build processes chosen by the author(s) of the software being built. These of course vary widely in sanity and depending on the packages one is building the otherwsise sane pkgsrc build process can quickly become a black box with packages that pull in many dependencies. It does break sometimes, but this is a fault of the target software authors, not pkgsrc. pkgsrc is first and foremost a system for building packages. Linux distributions OTOH tend to be much more focused on binary package managers. Most Linux users do not build packages from source. There is really no comparison.

Your comments do not sound like those of a daily NetBSD user. I have been one for the last 15 years and am therefore all too familiar /usr/share/mk. Most Linux users do not seem very comfortable with BSD makefiles.

> Why are you comparing pkgsrc to pacman.

The original commented suggested that XBPS was similar to pkgsrc. I suggested that XBPS has far more in common with Arch Linux's package manager than pkgsrc or anything else in the BSD world. As you yourself note in your next paragraph...

> Your comments do not sound like those of a daily NetBSD user.

I have commit access. Since three years ago. My name is on the latest release announcement.

You prick.

That's not nice and totally uncalled for. Three years is a very short time.

The original comment did not compare xbps to BSD package managers. What it said was that xbps was written by a former NetBSD developer and thus arguably has different qualities than other Linux package managers. It is a different approach from someone with different sensibilities than a Linux developer. I have read comments from Void Linux mantainers and users that say more or less the same thing.

Can xbps be used on foreign distros like pkgsrc? I'd be down to try it out on Ubuntu and NixOS if it supports that

I've been thinking of delegating mandatory crapware like zoom to my mobile phone. Needing to run stuff like that a few times a year is probably not worth preventing me from using an OS.

pkgsrc does work on Linux, but pkgin does not. I have no idea how to update my mirror and so forth using pkgsrc alone. Do not I need pkgin? I cannot do `pkg_add` at all.

I tried to make pkgin work, too, and I think it is possible to make it work with libbsd but I do not have the time. I wonder if anyone has done it.

I am the pkgin maintainer, and would be interested to hear about any issues you have building pkgin. It should build on any of the 23 platforms that pkgsrc supports, including Linux.

I even offer daily binary package builds for CentOS here: https://pkgsrc.joyent.com/install-on-linux/

Feel free to open up an issue at https://github.com/NetBSDfr/pkgin/issues and we can take a look.

Those binaries even sometimes work on Ubuntu, with a little manual linking and good luck, even though it's a total abuse. :D

I am trying to compile it from source on Linux.

I immediately ran into an annoying issue, which is having the need to specify those directories manually. So I did, and the configure finished successfully when I ran:

  ./configure --prefix=/home/john/.local/pkg/pkgin-20210529 --with-dbdir=/usr/ --with-libarchive=/usr/ --with-libfetch=/home/john/Downloads/libfetch-2.33 --with-openssl=/usr/ --with-sqlite3=/usr/ --with-pkg-install=/home/john/.local/pkg/pkgsrc-2021Q1/sbin/ --with-machine-arch=x86_64

When I ran `make`, I ran into:

  actions.c: In function ‘verb_flag’:
  actions.c:53:3: error: implicit declaration of function ‘strlcat’; did you mean ‘strncat’? [-Werror=implicit-function-declaration]
     53 |   strlcat(pkgtools_flags, "v", 1);
        |   ^~~~~~~
        |   strncat

Which tells me that I could fix it by providing `-I/usr/include/bsd/` and `-lbsd` or the like. I will give it a go. Any ideas anyway?

Edit: I added the above, and it went on to:

  In function ‘strncpy’,
      inlined from ‘read_repos’ at fsops.c:133:6:
  /usr/include/x86_64-linux-gnu/bits/string_fortified.h:106:10: error: ‘__builtin_strncpy’ specified bound 8192 equals destination size [-Werror=stringop-truncation]
    106 |   return __builtin___strncpy_chk (__dest, __src, __len, __bos (__dest));
        |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This does not sound too good, you probably want to fix it (along with the other warnings on Linux). In any case, I got rid of `-Werror` and I have a couple of undefined references. I will most likely get it to work, but you see my issue here, right? Is it the "official" way to compile it from source by the way?

Edit #2: I got a working binary by making those changes[1]. Last few lines:

   CCLD     pkgin
  /usr/bin/ld: external/pkgin-var.o: in function `var_get':
  /tmp/pkgin/external/var.c:99: warning: This function cannot be safely ported, use getline(3) instead, as it is supported by GNU and POSIX.1-2008.
  make[1]: Leaving directory '/tmp/pkgin'

---

You might want to look into compiling it on Linux, because without those changes[1], you cannot do it. I was using commit e963d1e1a2409ed51d191d936b5f75e35bdf9197, version 20.12.1.

[1] You also have to include `<bsd/string.h>` for `strlcpy` on Linux. Only `actions.c` is affected.

---

If you want, I am willing to send you an e-mail with the diff, but I will fix the other warnings, too, then. Oh, and instead of relying on `libbsd`, one could just add `strlcpy` and the like themselves.

---

By the way, from where do I get the `repositories.conf` exactly? The supplied `repositories.conf` seems outdated. I tried using both `ftp://ftp.netbsd.org/pub/pkgsrc/packages/NetBSD/$arch/5.1/All` and `http://mirror-master.dragonflybsd.org/packages/$arch/DragonF...`, but they were not working.

Hmm, I tried to use a working one, but I got this:

  processing remote summary (https://pkgsrc.joyent.com/packages/Linux/el7/trunk/x86_64/All)...
  SSL support disabled
  SSL support disabled
  SSL support disabled
  pkgin: Could not fetch https://pkgsrc.joyent.com/packages/Linux/el7/trunk/x86_64/All/pkg_summary.gz

I am currently using `file://` that points to the manually downloaded `pkg_summary.gz`. Not sure why SSL support is disabled when I configured it, it specifically asked for it, and I provided it through `--with-openssl=/usr/` and it passed because it found it.

  http://ftp.fr.netbsd.org/pub/pkgsrc/packages/NetBSD/amd64/7.1/All/

seems to work!

Although I do get:

  pkg_add: Warning: package `bootstrap-mk-files-20180901' was built for a platform:
  pkg_add: NetBSD/x86_64 7.0 (pkg) vs. Linux/x86_64 5.8.0 (this host)

So how would I remedy this for Linux? I want to use "https://pkgsrc.joyent.com/packages/Linux/el7/trunk/x86_64/Al..." but I get "SSL support disabled". I will debug this I think.

---

I fixed it. I had to compile libfetch with `-DWITH_SSL'.

---

All in all, there are many problems that I ran into, but nothing that much of a big deal.

---

There is one problem I cannot seem to fix:

  pkg_add: unable to verify signature: Signature key id 4735b9a256aaacaf not found

I keep getting this, and I have no clue how to change it or where it came from. Without this, nothing actually gets installed, so I cannot use pkgin or pkg_* until this is resolved. Do you have any clue? I tried to look for anything to change in pkg_install.conf or mk.conf but to no avail. Any ideas?

I'd been wondering how good pgksrc is on macOS for a little while, previously having used Homebrew wherever Nixpkgs was inconvenient or incomplete. Could pkgsrc could be a better choice?

pkgsrc is relatively nice for a cross-platform, source-based package manager in the old style, in that it provides convenient binary caches for most use cases.

I wonder: Has the writer of the post in the OP tried any next-gen source-based package managers, where the kind of isolation he describes as desirable between system and user packages is guaranteed between all packages altogether?

I'm always interested to try different things, but I don't understand why people seem to dislike Homebrew so much. I've been a mostly-happy user on both Mac and Linux for years.

I dislike it because it’s really slow to do everything. Even printing help text is slow.

Contrast with pacman on Arch, which is incredibly fast at nearly every operation.

Unfortunately I’ve found all the package managers for macOS to be lacking.

brew is slow, but has packages for most things and generally up to date.

fink packages seem a bit out dated last I checked. At least the ones I cared about.

MacPorts is also slow and somewhat out of date.

pkgsrc doesn’t have Apple M1 binary packages available yet.

I haven’t tried Nix.

Sometimes I’m tempted to try my hand at writing my own package manager for macOS, but getting a critical mass of packaged software seems quite hard.

Nix is probably the slowest and is missing many packages. It's Kool-Aid for the cool kids. MacPorts was good until everyone started using Homebrew. Homebrew has some weird quirks that drive me crazy, but it mostly works and is up to date. I tried pkgsrc and liked it, but a few common packages were weirdly named and it didn't have some Mac specific stuff.

Some new package manager that's a mix of pkgsrc, brew, and MacPorts would be awesome. Especially if portable and could somehow also replace nvm, rbenv, pythonenv, and things of that nature.

> Nix is probably the slowest

Nix evaluation can be slow sometimes, but installation is generally relatively fast, since everything in Nix is fully parallelizable. It's generally several times faster than apt, for installs that involve multiple large downloads, for example, and historically it's also faster than pacman.[1] (I think this is probably no longer the case with Pacman 6.0, which finally parallelizes downloads. In that case Nix evaluation time probably dominates.)

I imagine what you're identifying as slow is searching through Nix packages. On the old/stable release of Nix, you use something like `nix-env -qaP` for this, and that requires evaluation of every package in Nix, and can take a long time. `nix search`, however, is extremely fast on the unstable releases of Nix.

A Nix search using the old method is extremely slow, > 7 seconds:

  I  ~  time nix-env -f '<nixpkgs>' -qaP libreoffice
  libreoffice                  libreoffice-7.0.4.2
  libreoffice-still            libreoffice-7.0.4.2
  libreoffice-still-unwrapped  libreoffice-7.0.4.2
  libreoffice-fresh            libreoffice-7.1.2.2
  libreoffice-fresh-unwrapped  libreoffice-7.1.2.2
  libreoffice-qt               libreoffice-7.1.2.2

  ________________________________________________________
  Executed in    7.31 secs    fish           external
    usr time    6.79 secs  335.00 micros    6.79 secs
    sys time    0.52 secs   67.00 micros    0.52 secs

A Nix search using the new method, which leverages cached evaluation of Nixpkgs, takes about half a second:

   nix-shell -p nixUnstable --run fish
  
   time nix search --experimental-features 'nix-command flakes' nixpkgs libreoffice
  <snip>

  * legacyPackages.x86_64-linux.libreoffice (7.0.4.2)
    Comprehensive, professional-quality productivity suite, a variant of openoffice.org

  <snip>

  ________________________________________________________
  Executed in  576.30 millis    fish           external
    usr time  561.00 millis  252.00 micros  560.75 millis
    sys time   16.07 millis   55.00 micros   16.01 millis

And installation, in either case, is pretty fast. It takes a few second to install, for example, Libreoffice on my system, with the vast majority of the time taken up being the download:

   time nix-env -f '<nixpkgs>' -iA libreoffice
  installing 'libreoffice-7.0.4.2'
  these paths will be fetched (153.59 MiB download, 574.57 MiB unpacked):
 <snip>
  copying path '/nix/store/8yq9h91bsbgi62dhvh4xfqxvhq58b5y4-libreoffice-7.0.4.2' from 'https://cache.nixos.org'...
  building '/nix/store/nhmf4ak414lrzy3387kbsx8brm8hckds-user-environment.drv'...
  created 68 symlinks in user environment

  ________________________________________________________
  Executed in   21.32 secs    fish           external
    usr time    1.34 secs    0.00 micros    1.34 secs
    sys time    0.43 secs  521.00 micros    0.42 secs

(I tried for a bit to get Linuxbrew installed so I could test the same functionality with it, but Homebrew isn't portable enough to work on NixOS, since it hardcodes a ton of command and interpreter paths to files that don't exist on some distros.)

—

1: https://michael.stapelberg.ch/posts/2019-08-17-linux-package...

tl;dr: Homebrew actually takes several times as long to install packages as Nix does, and uses up way more space despite a less isolated toolchain.

Now that I'm on my Ubuntu system, I can easily test Nix and Homebrew side-by-side.

To make it fair, I've removed almost everything I have installed with Nix, as well as Homebrew. Interestingly, Homebrew is using ~2x as much disk space, and listing my installed packages (none) with Homebrew takes ~30x as long as with Nix:

    ~ 
    du -hsxc /nix/store /home/linuxbrew
    238M /nix/store
    584M /home/linuxbrew
    822M total

    ~ 
    time nix profile list

    ________________________________________________________
    Executed in   19.88 millis    fish           external
        usr time   11.21 millis  174.00 micros   11.04 millis
        sys time   11.18 millis  141.00 micros   11.04 millis

    ~ 
    time sudo -H nix profile list
    0 flake:nixpkgs#legacyPackages.x86_64-linux.nixFlakes github:NixOS/nixpkgs/b2f2d04f01b3de3abad0dd9bc48ad7a3ee32bbc6#legacyPackages.x86_64-linux.nixFlakes /nix/store/g4xva7x46h5bqqh811n12dzjc7325x21-nix-2.4pre20210326_dd77f71
    1 flake:nixpkgs#legacyPackages.x86_64-linux.starship github:NixOS/nixpkgs/b2f2d04f01b3de3abad0dd9bc48ad7a3ee32bbc6#legacyPackages.x86_64-linux.starship /nix/store/5n36vabqg701l9nji30m1r004zcy4q0w-starship-0.51.0

    ________________________________________________________
    Executed in   29.91 millis    fish           external
        usr time   27.30 millis  380.00 micros   26.92 millis
        sys time    4.72 millis  283.00 micros    4.43 millis

    ~ 
    time brew list
    ==> Formulae

    ________________________________________________________
    Executed in  881.81 millis    fish           external
        usr time  527.24 millis  411.00 micros  526.83 millis
        sys time  173.15 millis  298.00 micros  172.85 millis

Homebrew takes about twice as long to search for packages related to Libreoffice (results omitted to HN won't tell me this post is too long):

    ~ 
    time nix search nixpkgs libreoffice >/dev/null

    ________________________________________________________
    Executed in  554.37 millis    fish           external
        usr time  537.39 millis  434.00 micros  536.95 millis
        sys time   16.44 millis  289.00 micros   16.15 millis

    ~ 
    time brew search libreoffice >/dev/null

    ________________________________________________________
    Executed in    1.33 secs      fish           external
        usr time  578.16 millis    0.00 micros  578.16 millis
        sys time  124.57 millis  367.00 micros  124.20 millis

I had some trouble looking for large packages to install with Homebrew and Nix for comparison. It seems that GUI apps (Libreoffice, Firefox, etc.) tend to be packaged exclusively as casks for Homebrew, which means they don't exist on Linux. So instead, I selected a handful of packages from the Linux formulas list[1] that I figured would pull in a substantive dependency closure. Apparently there's no bottle for Crystal, so it took a very long time to install because it had to compile from source. Prior to that, though, installation crashed because of a ulimits issue when ‘pouring the bottle’ for LLVM:

    ~
    time brew install imagemagick bat feh crystal python mruby
    ==> Downloading https://ghcr.io/v2/linuxbrew/core/zlib/manifests/1.2.11
    ######################################################################## 100.0%
    ==> Downloading https://ghcr.io/v2/linuxbrew/core/zlib/blobs/sha256:6f604280b9e056da163b81cef3076b2e39bbd2b8d3cc8c36d27c19c472c760a4
    ==> Downloading from https://pkg-containers.githubusercontent.com/ghcr1/blobs/sha256:6f604280b9e056da163b81cef3076b2e39bbd2b8d3cc8c36d27c19c472c760a4?se=20
    ######################################################################## 100.0%
    ==> Downloading https://ghcr.io/v2/linuxbrew/core/libpng/manifests/1.6.37
    ######################################################################## 100.0%
    ==> Downloading https://ghcr.io/v2/linuxbrew/core/libpng/blobs/sha256:c5d219b27780c730af50693110ed2c79e50200d3595c7e395998fe3950354ec9
    ==> Downloading from https://pkg-containers.githubusercontent.com/ghcr1/blobs/sha256:c5d219b27780c730af50693110ed2c79e50200d3595c7e395998fe3950354ec9?se=20
    ######################################################################## 100.0%
    ==> Downloading https://ghcr.io/v2/linuxbrew/core/bzip2/manifests/1.0.8-1
    ######################################################################## 100.0%
    <snip>
    ==> Installing crystal dependency: llvm@9
    ==> Pouring llvm@9--9.0.1_5.x86_64_linux.bottle.tar.gz
    Error: Too many open files @ rb_sysopen - /home/linuxbrew/.linuxbrew/Cellar/llvm@9/9.0.1_5/bin/llvm-jitlink

    ________________________________________________________
    Executed in  600.02 secs    fish           external
        usr time  203.24 secs  479.00 micros  203.24 secs
        sys time   70.80 secs  313.00 micros   70.80 secs

I decided to re-run the brew install command, but skip Crystal, which only took a few more seconds:

    ~
    time brew install imagemagick bat feh python mruby
            <snip>
    ==> Installing mruby
    ==> Pouring mruby--3.0.0.x86_64_linux.bottle.tar.gz
    /home/linuxbrew/.linuxbrew/Cellar/mruby/3.0.0: 323 files, 14.7MB

    ________________________________________________________
    Executed in   20.18 secs    fish           external
        usr time    9.56 secs    0.00 micros    9.56 secs
        sys time    3.06 secs  761.00 micros    3.06 secs

    I left Crystal in when I tested Nix, since I knew it would be no problem, and Nix was done in ~36 seconds, meaning Homebrew ~17x the time, skipping one package entirely.

    After the two install operations, Homebrew was using several more gigabytes of storage than Nix:

    du -hsxc /nix/store /home/linuxbrew
    2.0G /nix/store
    4.9G /home/linuxbrew
    6.8G total

I'll reply to this comment with a ‘warmed-up’ test of installing a single additional package, as well. But it seems pretty clear already that it's misleading at best to characterize Homebrew as faster than Nix.

—

1: https://formulae.brew.sh/formula-linux/

Now for a single-package ‘warmed up’ test, and some cleanup on both sides to see what the total installed size looks like after autoremoval of orphaned dependencies, build-time dependencies, etc.

Installing one package with Homebrew, where the whole dependency closure is bottled, no building from source:

    ~ 
    time brew install graphicsmagick
    ==> Downloading https://ghcr.io/v2/linuxbrew/core/libxfixes/manifests/5.0.3
    ######################################################################## 100.0%
    ==> Downloading https://ghcr.io/v2/linuxbrew/core/libxfixes/blobs/sha256:3aaba846570380f7dbf5cf7f93280866318a7730c9c5fd38fd1ad275aec037bd
    <snip>
    ==> Pouring freeglut--3.2.1_5.x86_64_linux.bottle.tar.gz
    /home/linuxbrew/.linuxbrew/Cellar/freeglut/3.2.1_5: 20 files, 1.3MB
    ==> Installing graphicsmagick dependency: jasper
    ==> Pouring jasper--2.0.32.x86_64_linux.bottle.tar.gz
    /home/linuxbrew/.linuxbrew/Cellar/jasper/2.0.32: 40 files, 1.1MB
    ==> Installing graphicsmagick
    ==> Pouring graphicsmagick--1.3.36.x86_64_linux.bottle.tar.gz
    /home/linuxbrew/.linuxbrew/Cellar/graphicsmagick/1.3.36: 497 files, 14.7MB

    ________________________________________________________
    Executed in  362.04 secs    fish           external
        usr time   56.32 secs  603.00 micros   56.32 secs
        sys time   20.00 secs  383.00 micros   20.00 secs

Installing the same package with Nix:

    ~ 
    time nix profile install nixpkgs#graphicsmagick

    ________________________________________________________
    Executed in   11.58 secs      fish           external
        usr time  701.80 millis  500.00 micros  701.30 millis
        sys time  310.06 millis  319.00 micros  309.74 millis

So in terms of time, they're not really in the same ballpark. Nix is operating in ‘wait a few seconds’ time, and Homebrew is operating in ‘go make yourself a cup of coffee and come back’ time.

And after running `brew cleanup` and `brew autoremove`, then `nix-collect-garbage` and `nix-store --optimize`:

    ~
    du -hsxc /nix/store /home/linuxbrew
    1.9G /nix/store
    3.9G /home/linuxbrew
    5.8G total

Listing installed packages now:

    ~ 
    time brew list
    ==> Formulae
    aom      fontconfig      imagemagick  libelf libmpc    libx11       libxshmfence  mesa-glu   python@3.9      xinput
    bat      freeglut      imath   libev  libomp    libxau       libxslt      mpdecimal   readline      xmlto
    binutils     freetype      imlib2   libexif libpciaccess   libxcb       libxt      mpfr   rtmpdump      xorgproto
    bison      gcc      isl   libffi libpng    libxdamage   libxv      mruby   ruby       xz
    brotli      gdbm      jasper   libgcrypt libpthread-stubs  libxdmcp     libxvmc      ncurses   shared-mime-info   zlib
    bzip2      gettext      jbig2dec   libgpg-error libsm    libxext      libxxf86vm    nghttp2   sqlite      zstd
    c-ares      ghostscript     jemalloc   libheif libssh2    libxfixes    libyaml      openexr   unzip
    curl      giflib      jpeg   libice libtiff    libxi        little-cms2   openjpeg   util-linux
    docbook      glib      krb5   libidn libtool    libxinerama  llvm      openldap   wayland
    docbook-xsl  gmp      libde265   libidn2 libunistring   libxml2      lm-sensors    openssl@1.1  wayland-protocols
    expat      gnu-getopt      libdrm   liblqr libva    libxrandr    m4      pcre   webp
    feh      graphicsmagick  libedit   libmetalink libvdpau   libxrender   mesa      pkg-config   x265

    ________________________________________________________
    Executed in  580.23 millis    fish           external
        usr time  478.15 millis    0.00 micros  478.15 millis
        sys time  108.59 millis  759.00 micros  107.83 millis

    ~ 
    time nix profile list
    0 flake:nixpkgs#legacyPackages.x86_64-linux.imagemagick github:NixOS/nixpkgs/b2f2d04f01b3de3abad0dd9bc48ad7a3ee32bbc6#legacyPackages.x86_64-linux.imagemagick /nix/store/5ps3bydsjiqgycbzg2cxkwyz50m4rga7-imagemagick-7.0.11-6
    1 flake:nixpkgs#legacyPackages.x86_64-linux.bat github:NixOS/nixpkgs/b2f2d04f01b3de3abad0dd9bc48ad7a3ee32bbc6#legacyPackages.x86_64-linux.bat /nix/store/0lyhwrmf3nbadavaq3ixfndhiaw3b86l-bat-0.18.0
    2 flake:nixpkgs#legacyPackages.x86_64-linux.feh github:NixOS/nixpkgs/b2f2d04f01b3de3abad0dd9bc48ad7a3ee32bbc6#legacyPackages.x86_64-linux.feh /nix/store/sy0xxs6lbvwhidg5dlj6ckwik2mn3b6l-feh-3.6.3
    3 flake:nixpkgs#legacyPackages.x86_64-linux.crystal github:NixOS/nixpkgs/b2f2d04f01b3de3abad0dd9bc48ad7a3ee32bbc6#legacyPackages.x86_64-linux.crystal /nix/store/fbnk8kwbllsn2m60d43y2cfqpwbw5pk4-crystal-1.0.0
    4 flake:nixpkgs#legacyPackages.x86_64-linux.python github:NixOS/nixpkgs/b2f2d04f01b3de3abad0dd9bc48ad7a3ee32bbc6#legacyPackages.x86_64-linux.python /nix/store/pp0fp5zds7wqhyx6zqnnxl2aqzf4a51r-python-2.7.18
    5 flake:nixpkgs#legacyPackages.x86_64-linux.graphicsmagick github:NixOS/nixpkgs/b2f2d04f01b3de3abad0dd9bc48ad7a3ee32bbc6#legacyPackages.x86_64-linux.graphicsmagick /nix/store/fm7fa16blb4xadsxxagxy0vfmhrjqns0-graphicsmagick-1.3.36

    ________________________________________________________
    Executed in   25.18 millis    fish           external
        usr time   18.24 millis  452.00 micros   17.79 millis
        sys time    9.17 millis  279.00 micros    8.89 millis

Nix's output is a mess, visually, and that'll change in the future. But note that it doesn't bother me with the whole dependency closure, and instead tracks what I actually wanted to install.

Finally, a removal test. I'll uninstall everything but graphicsmagick, with both package managers.

    time brew remove imagemagick bat feh python mruby
    Error: Refusing to uninstall /home/linuxbrew/.linuxbrew/Cellar/python@3.9/3.9.5
    because it is required by freeglut, glib, graphicsmagick, jasper, libheif, liblqr, llvm, mesa, mesa-glu and shared-mime-info, which are currently installed.
    You can override this and force removal with:
    brew uninstall --ignore-dependencies imagemagick bat feh python mruby

Oops. A second try, since Homebrew put me in dependency hell because it doesn't know the difference between what's installed as a dependency and what I explicitly asked it to install (I thought this functionality was needed to implement `brew autoremove`??).

    ~ 
    time brew remove --ignore-dependencies imagemagick bat feh python mruby
    Uninstalling /home/linuxbrew/.linuxbrew/Cellar/imagemagick/7.0.11-13_2... (805 files, 31.4MB)
    Uninstalling /home/linuxbrew/.linuxbrew/Cellar/bat/0.18.1... (12 files, 5.5MB)
    Uninstalling /home/linuxbrew/.linuxbrew/Cellar/feh/3.7... (26 files, 716.9KB)
    Uninstalling /home/linuxbrew/.linuxbrew/Cellar/python@3.9/3.9.5... (2,650 files, 76.9MB)
    Uninstalling /home/linuxbrew/.linuxbrew/Cellar/mruby/3.0.0... (323 files, 14.7MB)

    ________________________________________________________
    Executed in    1.11 secs      fish           external
        usr time  874.06 millis  420.00 micros  873.64 millis
        sys time  224.91 millis  261.00 micros  224.65 millis

    ~ 
    time brew autoremove
    <snip>
    Uninstalling /home/linuxbrew/.linuxbrew/Cellar/libgpg-error/1.42... (48 files, 1.8MB)

    ________________________________________________________
    Executed in    3.35 secs    fish           external
        usr time    2.51 secs  424.00 micros    2.51 secs
        sys time    0.82 secs  261.00 micros    0.82 secs

Nice and fast. :)

Now for Nix:

    ~ 
    time begin
    nix profile remove '.\*'
    nix profile install nixpkgs#graphicsmagick
        end

    ________________________________________________________
    Executed in    2.99 secs      fish           external
        usr time  499.04 millis  871.00 micros  498.17 millis
        sys time  126.67 millis  527.00 micros  126.14 millis

Not as fast, because I garbage-collected the evaluation of Nixpkgs itself, so it fetched and re-evaluated that. (Also I did this in a weird way because the UI for the `nix profile` command is incomplete and still kinda clunky.)

I also need to clean up with Nix:

    ~ 
    time nix-collect-garbage -d
    removing old generations of profile /nix/var/nix/profiles/per-user/pxc/profile
    removing generation 49
    <snip>
    deleting '/nix/store/znvl3v02hy8ffkhwc6m9z1q5pj8j6ppj-setup-hook.sh'
    deleting '/nix/store/trash'
    deleting unused links...
    note: currently hard linking saves -1.14 MiB
    188 store paths deleted, 1574.20 MiB freed

    ________________________________________________________
    Executed in    1.23 secs      fish           external
        usr time   14.30 millis  160.00 micros   14.14 millis
        sys time   10.70 millis   96.00 micros   10.60 millis

    And after cleanup, the usage:
    ~ 
    du -hsxc /nix/store /home/linuxbrew
    412M /nix/store
    3.2G /home/linuxbrew
    3.6G total

Is there a cleanup command I've missed for Homebrew?

Performance considerations aren't really among my main concerns when it comes to package management, so I didn't mention them in my original post about what's ‘wrong’ with Homebrew. But I'd have to add it now.

I don't even like Homebrew, but I use it because of the network effect. Several years ago I tried a bunch of package managers in identical VMs on MacOS and noticed Nix was slower than all the others (all operations), missing packages, and some packages were broken. During that experiment I really liked pkgsrc, Homebrew didn't have a few things I needed and outright refused to install TeX Live (it literally said it was too hard to build). I stuck with MacPorts for a few years until MacPorts started having many broken packages from lack of maintainers, then I reluctantly switched to Homebrew.

> pkgsrc doesn’t have Apple M1 binary packages available yet.

basing this reply on my past experience with pkgsrc (in a netbsd or linux environments though) as long as you can point pkgsrc to a decent compiler and it can complete its bootstrap phase, it'll likely work.

That sounds right. But then I'm having to compile all the packages I need, which makes it even much much slower than brew when binaries are available.

Compiling on an M1 is too slow? ;)

+1 for pacman.

Homebrew's biggest issue is that it isn't a package manager for MacOS. You can interpret that however you want, but it's versatility is extremely limited compared to "full-fat" package managers like apt and pacman. At the end of the day, all of the MacOS package managers fit together to perform separate stop-gap solutions, where none of them really fell that great to use in the first place. Hell, pacman is so good that it's one of the core reasons I'll probably never end up fully switching my workflow to MacOS.

Pacman and friends used to work on macOS (source: I compiled and used them) and maybe they still do, but when it moved to bash 4 while apple is shunning newer versions, it became too much work to build all of the prerequisites.

How is macports out of date?

In addition to the valid criticisms in sibling comments:

I don’t like that it changes the rules on file ownership in /opt and /usr/local.

And it’s petty but I really don’t appreciate the beer brewing metaphor. It doesn’t fit and it feels forced. Giving the project a searchable name is fine but cutesy names are irritating when I have real work to do. Just name the tool and especially commands for what they do.

I say this as a person who loves beer.

Yes, the beer brewing stuff, formulae, taps, kegs, casks, and bottles stuff is annoying. I have no idea what "keg only" is supposed to mean or why I see that phrase sometimes.

I still don’t understand why sometimes I must type “cask” and sometimes I must not type “cask”.

No, I didn’t RTFM; I’m sure that would help, but I mostly just avoid homebrew when I can, and that’s worked out OK.

brew cask is a whole separate quasi-package management system built around pre-built macOS GUI applications distributed/installed as .app folders, same as you'd get downloading a .dmg from the web them doing a drag and drop install.

Think of it as an installer fetcher and runner, as opposed to a collection of integrated and uniformly built packages.

It's great, because downloading and running installers manually by clicking around sucks. But it's totally separate from the rest of Homebrew, which is a monorepo of build recipes targeting a separate base system, similar in that basic way to things like pkgsrc and FreeBSD Ports.

I also hate the brewing metaphors. I don't know why MacOS developers feel so compelled to abstract integral parts of your program away into unrecognizable metaphors. It's like asking for ketchup at the counter of a diner, and they insist that they only have "Heinz Tomato Sauce". Nobody in real life does this because nobody likes a smartass, so it completely escapes me why people think it's suddenly okay to do in software, particularly software designed to be used in a professional, power-user workflow.

It's more like using a plumber analogy to make sandwiches. I mean plumbers eat lunch right? So we can refer to condiments as grout and bread and meat as tiles and pipes are the sandwiches because plumbing.

Want a Reuben? That's:

  plumber tile rye beef-corned
  plumber grout saurkraut russian

Also since this is in the context of making a sandwich a bunch of options to 'plumber' mention things like toaster oven wattage and temperature.

  set PLUMBER_PREHEAT=350
  set PLUMBER_BACKEND=toaster

This is all kind of a pain. Let's use pipes. Except for some reason whoever defined this uses thousand island instead of russian dressing and expects the sandwich on a plate.

  plumber pipe reuben --russian --to-go

Thanks for clearing this up. I don't make sandwiches very often, but putting it in plumbing terms made it a lot easier to digest!

I do like the idea of a tool named plumber with leaky abstractions.

Built in telemetry/analytics was enough for me to not touch it with a ten foot pole.

It's got a larger user and contributor base on macOS than competing package managers, and that works in its favor, especially in terms of package selection. I'm not a total hater; there are people I respect who prefer Homebrew over tools that I like better for reasons I understand. And package management is a peculiar interest of mine, so I care a lot about details and there isn't any package management system that totally satisfies me.

That said, every time I've used it, I've been struck by design flaws that make Homebrew feel incomplete or immature compared to most Linux system package managers. Some examples:

• The last time I used it, Homebrew didn't even support recursive uninstallation of orphaned dependencies, guaranteeing the accrual of entire packages of cruft over time. This was apparently finally fixed (in the crummy way that apt-get, the worst of all the Linux system package management tools, handles it) only in 2020 [1].

• A secondary package manager taking over all of /usr/local is quite presumptous and not portable at all

• Homebrew's combination of an assumption of a single-user setup with the choice of installing to global directories in the FHS is awful, and imo it's straight up deceptive to claim that Homebrew's default setup meaningfully supports ‘unprivileged package installation’

• `brew cask` is useful, but the cleavage between it and the rest of Homebrew is awkward, and it's more like an installer-fetcher à la Chocolatey than a real package management system

• Homebrew's packages are built with a relatively low degree of isolation from the base system's libraries and toolchain, so it's not unusual for a macOS upgrade to break installed Homebrew packages, and the build outputs and bottles associated with a package are given per-macOS release. (To be fair, the former is a pretty common problem for all kinds of software on macOS, as well.)

• I really don't like Linuxbrew's abuse of /home, which you have to stick with if you want to use bottles (binary caches)

• Homebrew just doesn't support managing dependencies on libraries in some languages (e.g., Perl) at all [2]— even though it includes packages that depend on them— which fails to meet my bare minimum expectations for a general-purpose package manager. In practice, this means that you have to install additional, language-specific package managers to use Homebrew; you have to manage dependencies in them yourself; and Homebrew packages require you to install these dependencies in system-wide locations, polluting your whole operating system (a potential source of dependency hell, as well as a practical nuisance) for their sake. The latter also, of course, undercuts the unprivileged installation claim that Homebrew touts.

To be fair to Homebrew, it's been a while since I used a Mac on a daily basis, and I expect that Homebrew has improved in some ways in the meantime. I know for certain that it's improved, to reiterate an example given earlier, with respect to some, if not all (i.e., not those managed externally) orphaned dependencies. And Homebrew also has some strengths, namely its very slick and orderly command-line interface.

But the real reasons I try to use Homebrew as little as possible aren't about Homebrew being _bad_ so much as my preference for tools with other strengths (which require a design different from Homebrew's to implement). On macOS, I try to use Nixpkgs and nix-darwin as much as possible because I prefer things like:

• declarative configuration rather than imperative state management

• fully isolated builds, more predictable solutions

• using a single package management tool for libraries and apps of all languages

• building .apps from source like any other package

• atomicity of upgrades

• freedom to roll back configurations/installations at any time

• ability to mix and match packages from different releases, modify dependencies for one project without affecting another

• never installing language interpreters or libraries globally, except for login shells

I don't really want to use anything that doesn't have those features, and for that on macOS, Nix is the only game in town. Homebrew is a pile of state to manage, and I'd rather have intentions to declare than a pile of state to manage. That issue isn't unique to Homebrew, of course.

Pkgsrc doesn't share some of Homebrew's (in my view) other defects, like its abuse of the FHS on Linux, single-user design/bad permissions configuration, building many packages directly on top of macOS system libraries that change from OS release to OS release when it doesn't have to, or the expectation that library dependencies for some languages will be managed manually/externally (and in global directories, to top it off), so that's what makes Pkgsrc interesting to me as an alternative to Homebrew, even though Nix will still always be my first-line choice on macOS.

—

1: https://github.com/Homebrew/brew/commit/0fa417706a2143dbec1e...

2: https://docs.brew.sh/Gems,-Eggs-and-Perl-Modules

How are you handling the read only root folder with nix?

You mean for installation on macOS? For current versions of Nix and macOS, you just invoke the installer with an additional command-line flag at installation time, per the manual[1]:

  sh <(curl -L https://nixos.org/nix/install) --darwin-use-unencrypted-nix-store-volume

I don't remember whether I actually did this on my system. I think I manually created an additional volume for /nix, but encrypted it. That approach is also now described in the manual, should you prefer it.[2]

(Unencrypted should be fine anyway, as the Nix store is designed/assumed to be public (world-readable). Packages and tools in the ecosystem never put secrets there.)

—

1: https://nixos.org/manual/nix/stable/#sect-macos-installation

2: https://nixos.org/manual/nix/stable/#sect-macos-installation...

Thanks. The first and last option seem to be the same, no?

What are the dependencies of these "next-gen" package managers. pkgsrc itself only needs C and a POSIX-like shell.

Fair question!

The two that I know of currently are Nix and Guix. Both target fewer platforms than Pkgsrc, so we know right out of the gate that they're not suitable for all Pkgsrc use cases.

The question of what they depend on needs fleshing out before it can be answered. Both Nix and Guix package software in a way that is, in general, hermetic. This means that they don't assume the installation of anything outside their domain— they don't require you to have anything installed on the base system to get going with them. They don't use your system's shell and they don't use your system's C library. As an exception on macOS, Nixpkgs allows some impurities for some macOS packages, e.g., they are allowed to call xcodebuild. But in general, they include their own toolchains in their entireties. So if what you're getting at is ‘where do they depend on the base system during operation’, the answer is, more or less ‘nowhere’.

If the real thrust of your question is rather ‘what do you need to build them from source’, the answer, for now, in both cases is ‘a bunch of shit that they kindly prepackage for you in a binary tarball, if you want to bootstrap your own Nix or Guix on a given platform’. Guix has been doing a ton of work to reduce the size of their bootstrap seed, and it seems like within a few years they'll have it down to a hex monitor from which they can build a series of Scheme interpreters and C compilers until they have the whole system.

If the main thing you're asking about is ‘what do they use internally to process their package sets’ or something like ‘what's the minimum installed size of these package managers’: for Nix/Nixpkgs you need the Nix language interpreter (written in C++), plus at least what's packaged in the ‘standard environment’ for Nixpkgs (GNU bash and coreutils, plus a libc and a C compiler which vary by platform); and for Guix you need Guile, an implementation (interpreter and runtime) of Scheme. I've never worried about their installed sizes, since they're much smaller than the total size of everything you might want to install with them.

Idk the size of the full dependency closure for Guix and I don't have a great way to look it up at the moment. In the Nix case, the binary installation tarball is ~17 MB compressed and ~72 MB uncompressed, when you install it.[1]

Happily, if Pkgsrc is easier to get started with due to its smaller footprint, it looks like in the future users will be able to use Pkgsrc to install Nix.[2] :)

(For my part, I wouldn't mind if Nixpkgs included Pkgsrc, too. I wish it included Guix.)

—

1: https://releases.nixos.org/?prefix=nix/nix-2.3.9/

2: https://pkgsrc.se/wip/nix

Check this : https://pkgsrc.joyent.com/install-on-osx/

I really like pkgsrc. I'm not really qualified to give a technical justification for it. It just had the right simple, clean feel to it, which NetBSD in general has. I would even go so far as to say that it was fun to use.

I consequently once spent quite some time trying to get it working under Cygwin so I could use it at work, but never got past problems with Cygwin group names that have spaces in them. It was too complex for me to fix....

No mention of DragonflyBSD? They used pkgsrc as their main package manager for 10 years, portable or not, it still required quite a bit of maintenance. Seems like it was too much in the end, second-class citizen rights are not great.

Did they migrate away from pkgsrc? If so, do you know any specifics about why?

https://lists.dragonflybsd.org/pipermail/users/2013-Septembe...

it was a big help in Solaris to VWware migrations I did circa 2008-2012. I had one bash script that would install the whole site on either, bootstrapping almost everything from pkgsrc.

I wonder if anyone is going to create a linux distribution with a base system baked in with pkgsrc managing the applications. It would be very useful for containers and WSL.

There used to be one, which was based on Slackware (of course). If i find the name of that distro i'll either edit this post or reply to myself :)

I tried hand-rolling something similar with slackware 11 or 12 (can't remember now).

I always liked its concept and I am surprised very few projects are using it. On NetBSD it works well, I guess NIH (https://en.wikipedia.org/wiki/Not_invented_here) syndrome :(

Such a thing exists - https://github.com/oasislinux/oasis

I used it with Netbsd 1 and it was the first thing to install in Solaris.

Other question: how does one handle detection whether upgrades exist? Something like renovatebot but using pkgsrc. Or what is the best way in 2021 to trigger fresh builds against the latest packages?

pkg_rolling-replace -u

I wish there was a package manager for kde and konqueror for the mac

There's a Homebrew tap for KDE! I'm not sure what the present state of it is, but it's here if you wanna try it out:

https://invent.kde.org/packaging/homebrew-kde

Thanks. I'll look in to this.

From pkgsrc homepage:

> You can checkout pkgsrc HEAD with CVS:

...really? Not even Subversion, but CVS?

The repository of record is CVS, but you can - like many of us do - use either the hg export (https://anonhg.netbsd.org/pkgsrc) or GitHub (https://github.com/netbsd/pkgsrc) instead.

At some point we will transition the repository of record to something else, but it's being done carefully. We want to do it right, and there aren't many volunteers willing help with the work.

No, I was just surprised - I don't know anything about NetBSD (nor pkgsrc for that matter), but I worked with FreeBSD some 15 years ago and there were plans to move to SVN even back then. A quick search told me that the switch was completed in 2008 (a little bit after I left, begrudgingly, for Linux).

What challenges are there in switching to another VCS? Is the difficulty coming from the need to preserve history, or the need to re-evaluate workflows around the repo, or something else entirely?

Does pkgsrc have "port variants" like macports?

Not for prebuilt binaries (variants are just separate packages) but because the pkgsrc repo is just a collection of Makefiles, you can cd into the directory and run

    bmake show-options

Then set PKG_OPTIONS.somepkg and

    bmake install

Or even just edit the Makefile.

Reply on news.ycombinator.com