Hacker News Reader

Comment by jauer | Hacker News Reader

> Two different communities create two very different kinds of software, which run on the same systems, but are created and distributed in very different ways.

This is where the disconnect is coming from. The distro maintainers are coming from a world of multi-user systems where backwards compatibility and updating deps without disturbing a user's workload / forcing them to recompile is paramount.

Go (and a fair amount of rust/python work) come from the land of CI/CD and, to a lesser extent, monorepos. When you are rebuilding the world above a bare minimum of the OS literally on every commit (or at least several times per day), it's easier to reason about code that is running if you can look at the commit that a bin was built from and know exactly what's inside (including all deps).

View on HN

I agree. I think the difference has been that until recently "the land of CI/CD" and so on has been certain segments of the corporate world, and not how typical open source developers did things*. So when the former developed new technologies and new languages, they created build tools for them that anticipated being used in the ways that they usually produce software.

The "problem", in the sense that it's a problem, is that these languages and related technologies are all pretty good! And so it's understandable that many developers who would traditionally be in the open source ecosystem want to use them. As a result they end up creating software that can't easily be shipped in traditional distributions. Ecosystem fragmentation is the unavoidable result.

* By typical open source developers, I mean the sort of developers (and their development practices) that produced most of the software on my computer. I don't mean Firefox: Mozilla and Google have much more standard corporate development practices despite both producing quite a bit of open source software.

Although continuous integration starts in proprietary software, it's been present in Free Software for at least decades. Netscape may well be the second or third medium-large software outfit to do continuous integration the way it's done today (we know Microsoft had a team doing this by hand every single day for Windows NT but that's completely insane) because some of its team had experienced this approach elsewhere and knew they needed it if they wanted to ship software that actually works. When Mozilla was created, Tinderbox (that system) along with the Mozilla browser (and so today Firefox) and Bugzilla (a bug tracker) were freed.

I know it probably seems like last week, but that was more than twenty years ago.

I like Nixpkgs and NixOS for understanding both worlds. The Nix ecosystem's best path to mainstream success is being that go-between for everyone.

> The "problem", in the sense that it's a problem, is that these languages and related technologies are all pretty good!

Yes; and frankly the development ecosystem for making software for linux & friends on top of apt/etc is terrible - at least from the perspective of a modern professional software engineer. The assumption is C, and C programs have no package manager - so of course dependencies get bundled / vendored sometimes when the alternative is linking to potentially out of date dependencies in apt. Autoconf/automake is awful to learn and understand. CMake is better - but its horrendously complicated because it tries to solve the impossible job of paving over all the junky custom compilation scripts that came before. (And it still has no cargo equivalent for actually fetching your deps.)

And then to work around all of that, each distribution will make weird, custom, maybe buggy patches to your software before adding it to their package managers. (Which has caused some high profile bugs and security issues a number of times.) Now when there's a bug, nobody knows who's fault it is!

This worked in a world when there wasn't much software, when releases were rare and when most programs only had one or two dependencies. None of these properties are true any more.

Rust, go, python and nodejs don't fit well with linux's package managers. The obvious alternative would be putting every crate, gem, pip package and npm package into apt, rpm and all the rest. And keeping them up to date with every version. But lets be real - that would be horrible. Apt et al aren't (currently) up to the task. (Can you imagine every npm package needing a maintainer in apt alone? I can just imagine the github issues: "I'm in debian stable and this transitive dep you're using only has version 0.1 available, from 6 years ago. What do I do?". Yikes.)

I'm sympathetic to the argument that modern million dependency software development has its own problems; but right now it (sadly) has no competition in terms of ergonomics and build reliability.

> This worked in a world when there wasn't much software, when releases were rare and when most programs only had one or two dependencies. None of these properties are true any more.

I don't think this is even the problem.

It's that upstream maintainers have stopped worrying about compatibility.

Once upon a time you would have regular minor releases of some package. 3.0.2, 3.0.3, 3.0.4, but they were all backwards compatible. If you had version 3.0.4 and some software that was built against 3.0.2, it still worked against 3.0.4 because the only difference was that things were added or compatibly improved, not removed or incompatibly changed.

Version 3.0.x wasn't compatible with version 2.9.x, but then the package maintainer for the distribution only has to package versions 3.0.4 and 2.9.16, i.e. suitably recent minor versions of each compatibility revision. Compatibility revisions so old that nobody relevant uses them anymore can be ignored, so they only had to package two or three incompatible versions which together are compatible with everything in active use.

The problem today is that everything is a compatibility-breaking change, so there are dozens of releases from this year alone that are all mutually incompatible and would have to be packaged separately. And that doesn't scale.

Interesting - I didn’t know that. Can you give some examples?

https://wiki.openssl.org/index.php/Versioning

https://developer.gnome.org/gtk3/stable/gtk3-Feature-Test-Ma...

https://everything.curl.dev/libcurl/api

So, you deliberately chose distro intended for users, with low version churn, instead of distro intended for developers, e.g. Fedora, which ships even pre-release versions sometimes, and now you blame ... apt? Just curious, what you are using for coding? MS Word or Excel? For example, Linus uses Fedora/MATE/Emacs.

It's relatively easy to convert between packages between different packagers/distros. Automatic converters are exists for deb/rpm/pip/cpan/ctan/cargo, so it's easy to convert all existing packages into one packaging system and drop all of them into huge mono-repo.

Yes, it's much easier to throw a new version to repo in many non-linux repositories. It's the equivalent of rolling distros, such as Arch, or development version of distro, such as Rawhide in Fedora or Sid in Debian. However, it also much easier to: break the world and make into news, distribute keyloggers and steal passwords and keys, forget to backport security fixes to users of older versions, pivot into completely orthogonal thing, etc. No code reviews means no responsibility.

If you want your software to work everywhere, you either need to take responsibility for making it build everywhere (yes, including old versions of Debian which don’t support the versions of your dependencies you need) or you punt that work to someone else - in which case your software simply won’t work on lots of computers. From the perspective of an upstream maintainer, the status quo is pretty awful.

There’s a reason people are turning to docker - because the “portable executable” idea on Linux is so often broken by weird incompatibilities between libc versions or some important dependency being missing or broken from some users’ systems. Automatic package translation suffers the exact same problem - a dynamically linked binary you build on your computer often won’t run on my computer.

If you want for your software to work everywhere, but you want to avoid compilation step, then use Perl/Python/PHP with bindings to QT/GTK.

Linux has perfect backward compatibility, I still able to compile and run 70 years old app, developed on completely different OS and processor.

IMHO, you think that your app/lib binary will work on all combinations of OS/processor flawlessly without recompilation, which is not true. Nobody promising that. It's by design.

> weird incompatibilities between libc versions

Care to provide some examples?

I disagree that rust, go or python inherently do not work with linux package managers, instead there's a culture within subsections of those communities who do not value stability (I'm not sure about nodejs community though), which cause these disagreements. C isn't immune to this (and never has been)—science codes are infamous around their lack of care around stability (and IMHO one of the reasons for scientific python's success over alternatives has been this stability).

I'd also disagree with the idea that the alternative is reliable—try building and modifing a project that hasn't been touched in 6 months, and see how reliable that is. With your tree of dependencies (absent specific projects which abstract over a set of unstable dependencies and hence implicitly stabilise them, e.g. SDL) the stability of your project (whether it is a library or application or framework) is set by the least stable dependency you have. Increasing you dependencies increases the risk, but if you have an ecosystem which values stability, larger dependency trees should not see a significant increase in risk (personally, I'd love to see the scientific rust ecosystem achieve similar stability to python).

That's not to say linux distros are perfect (change can be slow ;)), but there's lots of little things they do get right (how many projects handle updating configuration files correctly—that kind of thing is built into distro tooling), and they enable ecosystem-wide changes more that the alternative (e.g. https://reproducible-builds.org/).

Additionally, when you have languages with rich library ecosystems, the OS kind of becomes irrelevant, the platform is the language ecosystem.

Just to pick Go as an example (not to be lost discussing VMs and such), it doesn't matter if I am targeting bare metal, Linux, Windows, IBM z/OS, AWS special cloud runtime, whatever.

As long as the Go code is the same, and someone has done the low level runtime support, it is a compile away and done.

Finally by pushing containers no matter what, the Linux community has made this even easier.

What if there's another software in another language you want to interoperate with? What if you want to avoid containers with their complexity and dubious security record?

Some form of IPC, usually OS agnostic ones.

That is what I have been doing the last 20 years, in the context of C++, Java and .NET.

This makes me wonder if there's a Linux base system suitable for servers that embraces the newer approach. That is, a minimal base system that's built for immutable container images, providing just what's needed to bootstrap the current generation of language-specific build and package systems. The Alpine Linux Docker images might be a good choice for now, but IIUC, Alpine Linux itself still embraces the older distro approach.

Maybe the Nix package manager / NixOS is what you're looking for? I think it takes the best features from both worlds.

Every package installed with Nix is isolated into content-addressable* directories, so for example, my install of Firefox is located at /nix/store/c7pmng2x05dkigpbhnjs8fdzd8kk31np-firefox-85.0.2/bin/firefox. This is pretty inconvenient to use directly, so Nix generates a profile that symlinks all your packages into one place (eg. /run/current/system/sw, ~/.nix-profile), and then environment variables like PATH can just include <PROFILE_DIR>/bin.

With this approach, I can have multiple versions of the same package installed simultaneously, without them conflicting with each other. Like in a traditional distro, any dependencies that are shared between packages aren't duplicated, but if a package needs to explicitly depend on a different version, it can.

Also, because Nix is designed as a functional package manager for building packages from source (even though it has a binary cache), you can trace back exactly what sources were used to build your package and its dependencies, all the way back to the bootstrap binaries used to build any self-hosting compilers (gcc, rust, openjdk, ...)

* Most packages use a hash that's generated from the inputs used to build it, rather than the output that's generated.

Sounds like another idea of djb took off (the slashpackage hierarchy).

I'm not sure what the slashpackage hierarchy is, but Nix & NixOS started in 2003.

That's https://cr.yp.to/slashpackage.html which is dated at least 2001. It's not quite the same to the Nix approach though, which also tracks all direct and transitive dependencies.

> That is, a minimal base system that's built for immutable container images, providing just what's needed to bootstrap the current generation of language-specific build and package systems.

At least for the first half, that sounds sort of like Fedora/Red Hat CoreOS[0], the predecessor CoreOS fork Flatcar Container Linux[1], or the Amazon distribution BottleRocket[2].

[0] https://getfedora.org/en/coreos?stream=stable

[1] https://kinvolk.io/flatcar-container-linux/

[2] https://aws.amazon.com/blogs/aws/bottlerocket-open-source-os...

Fun thing about BottleRocket relevant to this thread: it uses Cargo as its build system. It's really wild and very interesting and more people should know about it, IMHO.

Check out Fedora Silverblue for something along these lines.

> it's easier to reason about code that is running if you can look at the commit that a bin was built from and know exactly what's inside (including all deps).

Believe me, it's usually the opposite.

Lack of proper releases, testing, and versioning results in unending checkout-fu to figure out what commits for each of 20 libraries will work for each other.

The idea is plainly stupid, without any redeeming qualities.

The entirety of this cargo cult hinges on the point of "If people calling it a genius invention for last 8-10 years admit it not being it, some major reputation, and cred loss will be incurred"

> Lack of proper releases, testing, and versioning results in unending checkout-fu to figure out what commits for each of 20 libraries will work for each other.

Admittedly I'm basing this on my experience in a very large monorepo environment, but there's no figuring out which commits will work with each other. Every commit with every library will work, otherwise it doesn't get committed. Yes, this involves massive CI infra and tooling to aid in refactoring.

You want to make a breaking change to a lib? Great, it's on you to update every piece of code that calls it.

It's great when you can control every piece of your infra, but I totally get how it's unfeasible (and maintainer hell) for the distro community.

Distros are great for off-the-shelf software especially if you don't care which version you get too much. When versions matter, you quickly get into dependency-hell. So long-lived software tend to stabilize, and then remain unchanged.

K8s, go and even ruby, tend to change and evolve. It's usually a bad idea to pull such software from distros then, even if available.

The means to get software is simply too different, and it's a non-problem for everyone but completist distro maintainers.

Are you actually talking about static linking here, or just venting your spleen about sloppiness in software engineering practice more generally?

Because tracking dependencies in source control (specifically, checking in lock files) is tremendous for reproducibility. It means that when bisecting through the commit history to find when a problem began is not just bisecting through the local source code, but also the specific versions of every dependency.

So regardless of whether the issue you're investigating is in the package, a dependency, or an unexpected interaction between the two, you're able to find the first commit that introduced the issue in O(log(commits)) time, rather than needing O(commits * (num dependencies * dependency versions)) time.

> you're able to find the first commit that introduced the issue in O(log(commits)) time, rather than needing O(commits * (num dependencies * dependency versions)) time.

Try the oldest and newest compatible version of your code and the oldest and newest compatible version of each dependency. This is O(num dependencies), which is generally small N and you can often guess which to try first. Now do each version of the code or dependency that actually made a difference. O(log(N)) on the versions of that code. You're at O(num dependencies) + O(log(N)), not O(commits * (num dependencies * dependency versions)).

Also, doing it the other way can often lead to frustration. The actual bug is introduced in commit #1234 but is timing dependent, then a dependency is upgraded from version 2 to version 3 in commit #5678 which tickles the bug, e.g. the new dependency version moved around some cache lines. Now you're looking in entirely the wrong place at an innocent dependency.

Whereas if you notice that the bug exists with dependency version 3 and the latest commit and then do binary search on all the commits while holding the dependency at version 3, plausibly the bug shows up between commit #1233 and #1234.

While version 3 of the dependency is innocent, commit 5678 is not. Something went wrong in the interaction between the code and its dependencies in that change and discovering that change quickly is valuable.

From there you can start stepping through the code, or simplifying the situation to get a minimal repro of what's going on, or even bisecting with dependency version 3 held constant to see if that's diagnostic.

In my experience, with a bug that's so timing sensitive that some cache lines moving around will trigger it, you're likely to discover pretty quickly that something weird is happening as you try to get a more minimal repro.

Meanwhile, if you're not tracking dependency versions, such a test is likely to appear weirdly flaky. You're trying to track down this failure but something else comes up and you have to set it aside for a few days. When you come back to it, you can no longer get the test to fail because an untracked dependency change has moved cache lines around again. Which change? Which dependency? Since it's not in source control, you've got a painful process of guessing plausible combinations of recent versions of all transitive dependencies.

> While version 3 of the dependency is innocent, commit 5678 is not. Something went wrong in the interaction between the code and its dependencies in that change and discovering that change quickly is valuable.

The trouble is that this will tend to point the finger at large changes that jostle many things around at once and become a rabbit hole rather than the two line commit with a typo that actually caused the problem.

The main advantage you're putting forth is to know the versions of each dependency needed to reproduce the problem. But you can get that from the person reporting the bug. You can add a switch to your software to output the versions of every dependency it's using and then it's there in the bug report. And once you have a combination that can reproduce the bug, the process of experimenting with things to identify the cause is basically the same either way.

The old way of forever support for all versions and never any breaking changes is really beneficial on the consumer side.

However it is incredibly expensive on the producer side.

Package managers have pushed that pain to consumers. It isn't terribly surprising that producers are jumping on board en-masse.

Reply on news.ycombinator.com