Hacker News Reader

> I disagree, it's proven to be inadequate for modern software development

Well, that's exactly why the OP (and my essay) are "anti" modern software development in many ways. The view is that we're moving away from the traditional open source ecosystem and methods of software development with these new technologies, which (to be clear) are good technologies, but were created mostly to solve problems that some large corporations have, not to solve the problems that the open source ecosystem has.

> The least reliable way to package and distribute software is by relying on traditional package managers.

Not sure what you mean by this, but it's entirely untrue in my experience. Anything I install with a package manager just works, 100% of the time. Stuff I try to get any other way is a shitshow, and the lack of "quality control" provided by maintainers speaks for itself. I mean, just look at Android apps or the Chrome extension store. Heaven forbid we go back to the days of curl | bash off someone's website.

> By statically linking everything as much as possible and shipping everything else in a self contained bundle with a launcher that overrides any symbols that might inadvertently be pulled in from the system.

I know you know this, but just to be clear, that's not a solution to the problem of "making things easier for maintainers to distribute", that's cutting maintainers out of the loop. The whole point of my focus on ecosystems is that this is something that I, as a user, don't want to happen.

The open source solutions are primarily for C/C++. A bit for Perl, a bit for Python. But they haven't really moved on. Java has been tacked on since forever. Same for .NET, JavaScript, whatever.

And if you wanted your program propagated to all major distros you'd have to wait a decade.

Nobody has time for that. Not corporations, not mom and pop stores, and I doubt many hobbyists.

Then you always have the option to package your software for the ones you use and not worry about the rest.

If you want to make your program popular then packaging is part of the process needed.

Not anymore

That's not the problem. (*) The actual problem is that the distro maintainers want split packages (for security and so on), not vendored, and this requirement was already burdensome for many languages other than C/C++. If vendored packages were acceptable I believe people would have of course contributed them. Maintainers made this obstacle themselves (for a good cause, arguably) and it seems farfetched for them to then complain other languages are uncooperative.

* Was "isn't that a problem?", which was not I meant to say.

> If you want to make your program popular then packaging is part of the process needed.

That is demonstrably not true. I'm not making a value judgment (maybe it SHOULD be true, I dunno), but it's not true. There's a lot of popular unpackaged software out there.

I mean the whole argument here is that distro packaging is not needed, that there's a better way of doing it.

That actually clarifies some thing about the debate for me. Thanks.

I'm not sure I'm convinced based on the semi-frequent posts from maintainers and security pros about the issues with vendoring dependencies for software that is widely deployed.

This "better way", since we lack a more concrete name. Seems to be really great if you're running a web app, or server software in your own company and can rebuild and run a rolling deploy pretty easily. For someone pushing software to users all over the world, and as one of those users the downside to allow every application to be responsible for updating this stuff seems pretty steep.

I'd disagree that the problem of large organizations are different from the problems of the FOSS ecosystem. Organizations just have a financial incentive to fix them, the FOSS ecosystem does not. If mutually incompatible dependencies and security updates breaking software weren't problems for both corporate and FOSS ecoystems, these new technologies wouldn't have needed to exist. They'd just use the existing platforms.

And mind you, this is not a corporate/open source split. The burgeoning ecosystems are also full of FOSS technologies doing new and exciting things, they just don't break when a dependency updates!

>Anything I install with a package manager just works, 100% of the time

I run into issues with packages weekly. So much so I've spent engineer days purging references to system packages. It's universal too - yum, apt, pacman, brew, macports, I have to make sure nothing tries to reference packages installed outside a local working directory for an application because of mutual incompatibilities. Maybe it's because I'm trying to write software that runs on multiple targets and not use software where someone else has already spent the time and money to resolve these issues.

> I know you know this, but just to be clear, that's not a solution to the problem of "making things easier for maintainers to distribute", that's cutting maintainers out of the loop. The whole point of my focus on ecosystems is that this is something that I, as a user, don't want to happen.

They should be cut out of the loop. Maintainers don't have a right to dictate what design decisions I put into my applications because they don't think it adds value (the value is, it doesn't just run on their distro!). Another comment in this thread put it better, maintainers shouldn't place themselves in the development process.

> maintainers shouldn't place themselves in the development process.

Then perhaps we as developers should spend more time thinking about and participating in the maintenance process?

Which is exactly what developers have done, and what this guy is complaining about: developers have figured out a different approach to maintenance that works better.

I don't pretend to have an answer but I'm trying to listen to both sides so maybe I can formulate "the question" a little bit better and improve the discussion around this.

It seems like have posts every few months where security professionals, maintainers and sysadmins explain that the "developer usability first" approach to maintenance and dependency management is having massive consequences to our ability to keep systems secure, and to even know if an affected version of a library is on a system.

That is a different problem that doesn't seem to be addressed at all in the current iteration of these tools. If you're aware of efforts to solve that problem in the rust and go ecosystems(since those are the ones cited here) I'd love to read about it.

Knowing whether a vulnerable version is somewhere in your dependency tree, and making sure it gets fixed, is absolutely being done and being made part of CI etc. (I don't know about Rust/Go specifically, but the JVM ecosystem is the subject of similar complaints and we're absolutely doing vunerable dependency scanning and also things like "edge builds" where we bump every transitive dependency to the latest version and see if anything breaks). Nowadays Github itself will give you an alert without you even needing to do anything.

Frankly, most distribution maintainers seem to not know or care about how upstream software is built; they have an idea about what's "best practice" in the handful of languages they're using to build their distribution (which is mostly, like, C and Perl) and insist that they know best, without realising the rest of the world has passed them by.

> They should be cut out of the loop.

I disagree. This is how you get the Google Play store or the "freeware" app marketplace. It sucks. As a user, I'm quite happy to continue using a traditional distribution even if it means I don't get to use a handful of flashy programs by developers that disagree with the concept of maintainers. So far that choice has been much more positive than negative for me, and I'm doing my best (by promoting the open source ecosystem) to keep it that way.

> Maintainers don't have a right

If you're creating open source software, they quite literally do! https://www.gnu.org/licenses/gpl-3.0.en.html

If you put in a bunch of crap that doesn't belong in an application (ads, for example), I'm glad I have a maintainer that can either strip this out thanks to the GPL (or BSD / MIT etc), or else choose not to include your app in the distribution at all.

> a handful of flashy programs

This only works if "core" refuses to use Rust, so it's not sustainable. We already can't build Firefox and GNOME without Rust. Maybe Apache and curl next. And then?

We may be forced to work for free to make rusts build less of a pain. Or to abandon those projects which have abandoned us.

> As a user, I'm quite happy to continue using a traditional distribution even if it means I don't get to use a handful of flashy programs by developers that disagree with the concept of maintainers.

As a user, I'm afraid that you don't seem to be a representative of typical users. I would be happy to use a traditional distribution if they don't break on updates, which is still not the case. It's clear that traditional distros were not satisfactory on even keeping their own promises. I know they work hard, but ultimately the visible outcome says everything.

Distributions are not breaking software, they are just distributing broken software. Blame upstream for lack of testing. Or create your own distribution of flawless software and keep it up to date and flawless for the rest of your life.

Oh, sure. Softwares are broken, you can't fix that. Traditional distros just happen to (falsely) believe that they can somehow fix at their level. What we need instead is a distro that is resilient to software breakage, not flawless software. Does my hope look that unreasonable to you?

My distro works just fine, when an X11 app crashes due to a bug in the Nvidia driver, but I cannot.

> If you're creating open source software, they quite literally do! https://www.gnu.org/licenses/gpl-3.0.en.html

Clearly you missed the disclaimer of warranty in the licensing terms.

Packagers are welcome to maintain their own patches, or their own fork, if they like. But they don't have any right to tell upstream what to do or demand particular guarantees from upstream.

It's temporary, I'm sure. When broken software or a weaponized patch will slip through weak fence of maintainers and then break critical infrastructure, Congress will vote for something to stop that.

I’ve been on Debian since Potato, so I totally see what you’re saying. But...

> that's cutting maintainers out of the loop

Is this necessarily a bad thing? The market has seen the need to fill a hole, and it seems to be working.

I first started with Slackware, and dependency nightmares is what got me into Debian in the first place. Although Debian is nice because of its slow and stable base (which makes me happy for production), I’ve recently moved to Arch and have been so happy as it’s brought back Slackware’s idea of getting as close to upstream as possible and it handles dependencies! And to be honest, I’m loving it. And as an even added bonus, I’m getting more and more surprised how a tonne of packages that I’ve installed are Rust apps.

So, coming back to your comment:

> that's cutting maintainers out of the loop

With systems like Arch that get us closer and closer to upstream, are maintainers the unnecessary middlemen? Of course they’re not entirely redundant, but maybe a new model of distros like Arch will be more commonplace in the future

Thoughtful comment, thanks. I'm an Arch user as well and agree broadly with its approach to a desktop operating system. (I.e. stick as closely to upstream as possible.)

That said, I disagree that this means maintainers are unnecessary middlemen, even though their role on a distribution like Debian is obviously more prominent. The essay I linked to in my top level comment is actually by an Arch maintainer, explaining why they still see maintainers as playing an important role. http://kmkeen.com/maintainers-matter/

>With systems like Arch that get us closer and closer to upstream, are maintainers the unnecessary middlemen? Of course they’re not entirely redundant, but maybe a new model of distros like Arch will be more commonplace in the future

Arch is an old distribution, very much in the same class as Fedora, Debian, Gentoo and all the traditional ones.

What makes you think Arch makes maintainers even slightly redundant?

We still deal with security issues. We still need to figure out which Go software utilizes a library with a CVE with no tooling (go list and grep comes a long way). And we still need to deal with pinned dependencies in upstream projects.

Maybe they have heard of the AUR and think it's how the rest of Arch works?

I'd be terrified, but that is how NixOS works.

That is nonsense and you seem to spread this misinformation in a lot of places. You should also add a disclaimer that you are part of the Arch team.

AUR: Anyone can create an account and upload PKGBUILDs. There are no checks at all. AUR users should verify whether PKGBUILDs are not malicious. In practice, a lot of people use things like yaourt to install packages from the AUR without verifying the PKGBUILDs.

nixpkgs: anyone can contribute a PR with a new package, package update, or package modification. However, changes only get added to nixpkgs after someone with commit privileges verifies the PR and merges it. Also, a common misconception is that nixpkgs package maintainers can merge changes. This is false, only a much smaller set of committers can merge changes in the actual nixpkgs repository.

nixpkgs is more like the Arch Community repository, where committers are long-time contributors with a track record of high-quality contributions. Parts of nixpkgs are like Arch Core/Extra, because they are marked using the GitHub codeowners mechanism and changes are generally not merged unless approved through the code owners.

Disclaimer: I am a nixpkgs committer, former Arch user and AUR contributor.

>That is nonsense and you seem to spread this misinformation in a lot of places. You should also add a disclaimer that you are part of the Arch team.

The AUR comment is unfair, uncalled for and adds nothing to the conversation. I apologize and hope our previous conversations have been more productive. It was meant more tongue in cheek then some grand claim about the quality of nixpkgs and stems mostly from the frustration of the entire vendoring issue.

> We still deal with security issues

I don’t see your point. In fact, you’ll most likely find that distros that follow upstream closer than “slow and stable releases” will get their patches as soon as upstream fixes them

This assumes an ideal upstream: This is not always the case.

If someone publishes a CVE for a Go or Rust library it's not always the case the project is well maintained or the dev cares to update the dependency. Even if they did, there are no guarantees the upstream decides to publish a minor release just to update dependencies. Because that is what vendoring dependencies get you.

Instead of applying one patch to a shared library I'd need to hunt down all upstreams utilizing the library and manually patch between 10-140 packages independently and submit them upstream.

If upstream is not keeping their software up to date, why are you using that software? Imagine if Google stopped updating Chrome. Would you keep using Chrome?

Traditionally package maintainers has kept dependencies up to date. This has changed with vendored dependencies where upstreams have to do the work. They are not always up for that work. This isn't strange, nor weird and the comparison to chrome and google doesn't make much sense.

> will get their patches as soon as upstream fixes them

i.e. never. Lot of upstreams are abandoned with tons of PR's waiting years for somebody to merge in.

The problem is that the modern practices are being adopted to meet the changing needs of this code, which weren't an issue in the enthusiast linux and BSD communities, but are an issue when open source code is being put in mission critical professionally managed production environments on AWS, or into cars and appliances.

The userland is much more complex now, so freezing dependencies and bundling them in order to have a smaller number of test cases may not have been necessary when the environment was smaller and simpler. The reliability expectations were lower, and there weren't big money professional support contracts which required you to validate your application against a set of well defined environments. All that has changed.

Why are you comparing distro package managers to the Play store or the Chrome extension store? You should be comparing them to npm, pypi, etc. That's clearly the context of this discussion. No one is saying that the Chrome extension store does a better job than package managers. The language ecosystems do an amazingly better job.

> The universe I'd like to live in is where the only use case for dynamic linking are OS vendor APIs and cryptographically secure functions like TLS. My dream package manager would whitelist those system librarie and forbid distribution for any bundle that does contain the shared objects with the symbols it needs.

This idea seems to be predicated on a belief that OS vendor APIs and cryptographic libraries are the only attack surface for serious user-affecting software exploits.

They're obviously not, but they're the ones that package managers can help mitigate automatically. The rest is going to be up to the developers to patch.

Some of the things you mention are not incompatible with package managers. Have you considered Nix?

I actually agree with the parent post in terms of strongly preferring package managers to other means of distributing software. I've always found Linux much easier to admin than other OSes simply because of package managers.

Was really pumped after playing with Nix for a bit. But got bitten twice in the first day. I tried to run my work project using Nix. It needs Pythons python-prctl library - that doesn't work in Nix. So that's a dud. Next I try to use it as a Nim environment - Nim is unable to compile anything in Nix due to some missing (bundled) Glibc symbol. (Nim works nicely in my standard installation). The cool "export full container" thing mentioned in the tutorial failed for all, even the simplest cases. So I am kinda disillusioned by Nix.

You can always emulate FHS environments for things that are hard to turn into a Nix package without significant effort.

Aside from that, Guix is an alternative implementation that might have what you need.

Are you saying Debian stable isn’t slow and careful enough in your experience? I usually hear just the opposite.

> I do not trust system provided libraries to function with my applications because I've been burned so many times in the past.

I agree. About half the issues I've had with dependencies has been due to distributions fiddling with upstream for some reason or another. Probably the main reason I like Arch (which has a policy of just following vanilla upstream, though they aren't immune: 'python' being python 3 is probably the biggest pain point)

>I disagree, it's proven to be inadequate for modern software development

You're not disgreeing with the post you responded to; you're just stating a different priority.

hctaw said the distro/maintainer/pm approach is extremely reliable at producing trustworthy software. That means trustworthy to the user. It says nothing at all about how hard producing that software is for the developer.

You are saying that the distro/maintainer/pm approach makes it harder for the developer. That's true. But it doesn't contradict the above at all.

And anyway, as a user, I don't care. I want my software to work, to be stable, and to not have security flaws; and if a security flaw is found, I want a fix to be pushed to me ASAP. The distro/maintainer/pm approach does that. If instead I have umpteen zillion different statically linked applications installed, each of which packages all of its own dependencies, then instead of just relying on my distro to push security fixes to shared libraries that everyone uses, I have to rely on every single one of those developers to do it for their own packages. And most of them won't do it, or they'll do it when they get around to it instead of when I, the user, need it.

> The universe I'd like to live in is where the only use case for dynamic linking are OS vendor APIs and cryptographically secure functions like TLS

This won't work either, because those are certainly not the only places where security flaws can happen that I, the user, need a fix for ASAP.

> And anyway, as a user, I don't care. I want my software to work, to be stable, and to not have security flaws; and if a security flaw is found, I want a fix to be pushed to me ASAP. The distro/maintainer/pm approach does that. If instead I have umpteen zillion different statically linked applications installed, each of which packages all of its own dependencies, then instead of just relying on my distro to push security fixes to shared libraries that everyone uses, I have to rely on every single one of those developers to do it for their own packages. And most of them won't do it, or they'll do it when they get around to it instead of when I, the user, need it.

This is my main worry as well. We are currently in the early, easy, period of this new development paradigm, where developers are constantly releasing new code, and fixes are easy to deploy. I worry that in 10 or 15 years, when a security bug is found in a critical imported function, the developers aren't going to be around anymore to fix them, they will have moved on to the next new, hot, language, and will have as much interested in maintaining their old Go/Rust code as developers today do in maintaining their old C code.

As a user, my response is simple: I don't use software that's built that way. Outside of code that I write myself, I simply refuse to use software that's not accompanied by a distribution and maintenance infrastructure that I trust. For most software, that means it's packaged by my distro. Some big players might be able to convince me to take their software from them directly, but they will be very few, because there are very few big players that are as reliable as my distro, and that's the standard they have to meet.

> Outside of code that I write myself, I simply refuse to use software that's not accompanied by a distribution and maintenance infrastructure that I trust. For most software, that means it's packaged by my distro.

Okay, that's a reasonable approach, but with distros essentially saying for new software, they can't continue to maintain that standard, and they're just taking software as it comes, without securing it. I think you will find that there are lots of small utility programs and libraries that you won't have available to you in 10 years with this approach. YMMV.

> I think you will find that there are lots of small utility programs and libraries that you won't have available to you in 10 years with this approach.

Then I'll either find an alternate source that has enough reliability to satisfy me, or write them myself, or do without.

(Or I'll end up building what amounts to my own distro. Which is something I have indeed thought of doing, because my preferences are rather idiosyncratic.)

> And anyway, as a user, I don't care. I want my software to work, to be stable, and to not have security flaws; and if a security flaw is found, I want a fix to be pushed to me ASAP. The distro/maintainer/pm approach does that.

Not my experience at all. The distro maintainer generally takes significantly longer to push out a fix than the upstream developer.

> The distro maintainer generally takes significantly longer to push out a fix than the upstream developer.

Of course this is true in a sense, because the distro maintainer has to wait for upstream to push a fix before they can package it.

However, for the upstream developer, "pushing a fix" means "pushing updated source code". For the distro maintainer, "pushing a fix" means "compiling the updated source code and packaging the resulting binaries for all supported versions".

There are some upstream developers who could probably accomplish the latter at least as fast as distros do, but not many. But the latter is what I, as a user, need.

My release builds makes binary packages (including a .deb) and push them to my artifact repository; isn't that normal?

I agree with your comment, but to be clear,

> hctaw said the distro/maintainer/pm approach is extremely reliable at producing trustworthy software

I think you mean I (bscphil) said that. hctaw was the user disagreeing, saying it makes things harder for developers.

> I think you mean I (bscphil) said that.

Oops, yes, you're right. Sorry for the mixup.

> it's proven to be inadequate for modern software development

Sure, because it's not meant for software development. It's meant for users to run software.

Software development packagaing/distribution is pretty bad. Most languages do it their own way and never seem to learn the lessons of previous ones. And on top of that, they encourage poor habits, like writing your own module that's the same as somebody else's with one new function, rather than contributing to the existing module, or writing an extension for it.

> I do not trust system provided libraries to function with my applications because I've been burned so many times in the past.

Probably this is because both you + the library developer are not coordinating with the distributions on how you release your code. Distros get a lot of flack for breaking changes, but they're working from software released by developers, and rely entirely on their own user base to test changes. If developers cared about their software working they'd be more involved in its packaging & distribution.

> The least reliable way to package and distribute software is by relying on traditional package managers.

Checks and balances produce reliability. Having a packaging process and people distinct from the code author validating and enforcing it, most certainly produces much more reliable, secure and stable packages and distributions.

Sure, it's slower. Slower is actually a feature in this use case.

Having anyone push out their code to the world without any constraints or care, breaking compatibility day to day, that is certainly faster and easier. But reliable, certainly not. This is the culture that produces leftpad debacle and assorted malicious packages regularly.

It really doesn't. I've worked on upstream projects where a significant fraction of all bugs reported were created by distributions screwing up packaging and patching in ways they weren't at all qualified to understand and frequently led to non-obvious failures.

When we tried to work with them to fix this, about half the time they flamed us and quoted distro 'policy' as a reason not to fix their bugs, so we just refused to accept bug reports from anyone using those packages anymore.

The fact is that a lot of old-school Linux distributions are built by people who have only a very vague understanding of the software they're packaging, and frequently are closer to the sysadmin side of things than the large-scale software development side. It makes the relationships very frustrating and that's why proprietary software vendors invariably opt-out of distro packaging. Even with apps statically linked to the max possible level Linux users generate disproportionate levels of support tickets due to the general flakiness of the distros they use, so allowing them to modify tested software even further is a losing proposition.

Basically the whole concept of a Linux distribution is obsolete, fading away and irretrievably broken. Hence the proliferation of containers.

> By statically linking everything as much as possible and shipping everything else in a self contained bundle with a launcher that overrides any symbols that might inadvertently be pulled in from the system.

So, you propose to ship your own OS, as single image, for your application? What stops you from doing this? Drivers? Ship your own hardware then, like smartphone vendors do.