Traditional packaging is not suitable for modern applications?
theevilskeleton.gitlab.io
theevilskeleton.gitlab.io
Containers are golden images. You find combination of stuff that works, carefully package them together and ship this whole package. Gold images run on bare metal, containers run on container runtime enabled kernel.
Sure, containerizing an application is /easier/ than working out the dependency trees and solving dependency conflicts between them. It is by no means fundamentally better approach.
If anything, this argument dismisses most work done by distros - building and solving those dependency trees. Maintainers of free OSes sometimes do not bother to solve all possible conflicts, however instead of breaking the system/packages, they do inform you of possible conflicts that are not solved.
Inability to cherry-pick dependencies is deceptively wrong. Traditional distributions resort to use of so called meta-packages and alternatives to keep the system up to date. Deb/rpm packages are simple archives with additional metadata slapped on. The packaging system is transparent to contents: nothing forbids you to package versions 1.0 and 2.0 and have them happily coexist as long as they do not share common paths. In fact, a lot of packages are distributed in exactly the same way.
It seems that the author has a solution - containers - and tries to find a problem for said solution.
I think your comment is analyzing the problem from a different abstraction level than the author's examples.
In other words, your alternative approach of "solving dependency conflicts" is abstract advice unless you try to make it concrete by focusing who solves those dependency conflicts.
Using the author's example of a VSCode conflict (collision of "/usr/bin/code") with Actor in italics:
- Microsoft owns commercial the build of /packages/visual-studio-code-bin
- Filipe Laíns & Massimiliano Torromeo owns the open-source-license build of /packages/community/x86_64/code/
- Levente Polyak, et al owns the Arch Linux distribution
- end user owns the computer that needs to run conflicting software
Generalize the above matrix to every software package and consider which of those 4 actors can practically solve the conflicts in a scalable way?
Filipe Laíns & Massimiliano Torromeo can't control what Microsoft does and vice-versa. (Ironically, the freedom to not be controlled by the other actor is one reason there's a package conflict!)
And Levente Polyak could "solve" it by blessing Filipe Laíns's package and rejecting Microsoft's but this creates a new "problem" for users who specifically want Microsoft's commercial build.
The fundamental issue is that you have independent actors that can't control what others do. Result is that sometimes software packages clash (namespace collisions, dependent libraries not in sync, incompatible licenses, etc).
The approach of using containers -- however flawed it is -- gets past the stalemate of independent actors in the software landscape being incompatible with each other.
What does that is bundling your dependencies, not containers. You could always bundle everything an app needs into a single tarball or a package. Containers didn't make it possible, they made it easy and popular.
Yes, I agree. Where we are in disagreement is what the role of a distro is here. Distros are there to solve that stalemate in one way or another. In Code's case distro maintainers chose to mark these packages as conflicting. They could have changed namespace of one (or both) package[s].
As I have mentioned in reply to a sibling comment this is exactly the work distros are doing - solving the stalemate between upstreams.
> The approach of using containers -- however flawed it is -- gets past the stalemate of independent actors in the software landscape being incompatible with each other.
Sure, containers do try to get past the stalemate, but they try to solve a different problem - how to make conflicting dependency trees coexist. Containers can also fail to solve the stalemate. Sure, containers can isolate `/usr/bin/code` between containers, but contention on `/etc/code`, `$HOME/.code` or other static user data still has to be solved one way or another.
> It seems that the author has a solution - containers - and tries to find a problem for said solution.
I don't really follow this. Distro maintainers are working in a problem and some people are happy with the results.
But the fact Nix, Guix flatpack and others exist shows other folks are not and are coming up with their solutions.
It seems you're painting the author as some container weirdo just for not liking your favorite distribution method.
Containers solve entirely different problem - how do you make conflicting dependency trees coexist in isolation. Layered containers solve this particular problem quite well. But it's a different problem than distros try to solve.
I'm not even sympathetic to weird cases like OBS needing a forked version of ffmpeg. 1 this sounds like a wacky thing to do, 2 it sounds like they created a new project that is a different dependency. Call it something else (I suggest obsmpeg), package it for distros, pull in updates from ffmpeg whenever you want, reap the benefits of FOSS. Inventing an entirely new software delivery system instead seems like an overreaction.
I am sympathetic to working this out across multiple distros though. Finding the intersection of packages across multiple versions of Ubuntu, Debian and Fedora isn't super easy, and it can really hamstring you when building an app (you want to use libcool-1.8.x, Ubuntu LTS has libcool-1.2.25, :sad:). The ecosystem isn't set up for this and it's a pain to deal with in your app (you more or less #ifdef functionality out, which product managers will freak out about).
They're clearly doing a huge amount of work editing everything, but it also doesn't seem like they have the resources to properly test their edits. And while their edits are necessary to make everything work with their system, from a high level perspective they don't seem to be making the software better at what I want it to do. Given that, their model looks like dangerous busywork that at best delays updates and at worst introduces new bugs.
Distros use the word "maintain" to describe what they're doing, but they don't seem to be doing that. To maintain a piece of software I'd expect the maintainer to need a deep understanding of the source code and to continually monitor changes. The complaints maintainers post that software changed seem to prove they aren't doing that.
When I first saw a maintainer complain a library only posted about a breaking build change in their release notes and mailing list I was shocked. I had been assuming they were reading diffs, when they weren't even reading the changelog.
That said the track record speaks for itself. I can only remember one time a maintainer introduced a vuln in Debian. The system works even though you’ll find cracks if you look.
So much of this stuff is like a default unwillingness to work with humans over the internet, but that's fundamentally how FOSS works. A couple years ago there was a kerfluffle between the scientific community and SciPy--the community wanted SciPy to maintain compatibility literally forever and also continue adding new features. SciPy was like, "that's impossible, and even if we wanted to it would take the resources of Microsoft to do". But they never talked about it! Instead there were open letters and forum threads.
The quickest route to fixing social problems like this--and they are social--is to deal with people. I'm pretty misanthropic, but even I have to admit this is true.
> Distros use the word "maintain" to describe what they're doing, but they don't seem to be doing that. To maintain a piece of software I'd expect the maintainer to need a deep understanding of the source code and to continually monitor changes. The complaints maintainers post that software changed seem to prove they aren't doing that.
I just want to point out that this is asking a lot. You're essentially asking Debian to find volunteers to become as knowledgeable about, say, Libreoffice, as the developers themselves. That's probably not gonna happen. I think a more reasonable definition of "maintainer" is "ensures the package continues to work on ${DISTRO}, reaches out to the developers if a new update breaks that". I'm not defending the case you brought up, that sounds pretty lazy to me, but in the vast majority of cases it works great.
Additionally, essentially vendoring full dependency trees for each application separately would massively increase the maintenance burden because instead of just maintaining one version of a library the distro will need to maintain n versions
> Modern applications are inherently fragile.
There is nothing inherent in the fragility, it is self-inflicted problem of some applications.
It's not the distro's job to ensure the application is secure and up to date, its up to the application packager (i.e. docker container maintainer) to do that. i.e. to rev new releases of the application image even if nothing has changed in the core app, just to take care of security updates in distro provided packages). In pr
I would prefer a distro that ensures applications in its repository are up to date.
Consider a log4j-type issue. In one case, the distro quickly fixes the vulnerability in a library package, the user installs the update and that is that.
In the other case, the user has to track down which installed applications use the affected library, update them if there is an update available, and figure out what to do if there is no update available.
These problems are all about dependencies, which fall to upstream to make managable. If your favorite app breaks all the time it's probably because the code base is a giant mess. Oh, and some of the mess comes from too many different licenses in use, which prevents tighter integration of source code.
If I'm just going to completely rebuild the app and all of it's dependencies I may as well statically link them and get the unique optimizations we can do after link time by analyzing the resulting executable.
1) fix the bug in the vulnerable package (cut a new version)
2) make sure the new, bug-free version is actually in use
1 is the responsibility of upstream (though sometimes distros will do this too, and release it themselves while they try to upstream the patch).
2 has to be done by somebody for every application using the vulnerable package, either a) automatically (for example, by Debian releasing the new log4j version and all dependent applications picking it up at runtime) or b) individually (by each dependent project's upstream updating the dependency, cutting a new Flatpak or container image, and getting users to update it).
a is much more pleasant than b, at least in the universe as currently constituted. We can imagine a world where b is mostly automated so it is about the same or less work, but we don't live in that world yet.
I hear what you’re saying. But - I’ve been writing a lot of rust lately. For better and worse, most rust programs pull in dozens of crates as direct or transitive dependencies. Apt isn’t up to the task of replacing cargo:
- Apt’s dynamic library mechanism is designed around C’s dynamic library support. This doesn’t work with many other languages. It certainly doesn’t work with rust, which doesn’t have an ABI.
- There are too many rust packages, and they update too frequently for apt to keep up. Apt could be a bad, out of date, partial clone of cargo. But that sounds worse than cargo in every way.
- Shipping deps with cargo (or whatever) is preferred by app developers because cargo is better integrated with the language. And it works reliably on every platform. A dpkg wouldn’t even work on every version of Debian.
We have a similar problem with ruby gems, nodejs packages (in npm), etc etc. I could imagine apt wrapping cargo, npm, and friends and providing equivalent functionality. But the authors of apt seem to think it’s already good enough. It’s not. Cargo, npm, and friends run rings around apt from the point of view of developer mindshare, ease of use, platform independent packaging, and so on.
Apt and friends don’t have the features we want, and all the browbeating in the world won’t change that.
That sounds like a pretty large downside to using Rust.
https://wiki.debian.org/StaticLinking#Rust https://github.com/rust-lang/rfcs/pull/2603 https://github.com/rust-lang/rust/issues/60705
Rust libraries can be dynamically linked if they look like a C library, or get wrapped by abi_stable[1] or something. But this isn't the standard. Probably less than 5% of the crate ecosystem is packaged like this.
And a 5% solution is nowhere near good enough. Right now, I can build any rust package on any system, no matter what dependencies it has by just running cargo build. For apt to be competitive, it'd need to pull the ~100k cargo packages into apt's repository, wrap all of them in abi_stable or some equivalent and parse Cargo.toml files into the equivalent dpkg config.
Ideally it should also work across every OS (though starting with ubuntu, debian and mint would be a start).
This is possible. But for all the hand wringing about how modern packages should use apt, nobody seems to actually care enough to make any of this happen.
https://github.com/rust-lang/rust/issues/60705 https://github.com/rust-lang/rfcs/pull/2603
It might be cool if we "united" all the build apps (npm, pip, cargo, go, etc) under a similar interface, but that also sounds like a lot of work, which is why I think these package managers stick with binaries.
Nothing's stopping them from shipping them as static libraries - with some simple tooling it's easy enough to figure out what needs rebuilds and automate it.
> There are too many rust packages, and they update too frequently for apt to keep up.
And what do you think cargo does to keep up? It uses a metadata system and a common build system across packages to make automatic discovery of dependency trees seamless. There's nothing stopping a debian maintainer from writing automatic tooling around this, in fact, it already exists[1].
> A dpkg wouldn’t even work on every version of Debian.
And neither does cargo as soon as there's a crate that requires a too new version of openssl or whatever other C library. The reality of the matter is that cargo can't express the full dependency tree including any cross-language or system-level dependencies.
> We have a similar problem with ruby gems, nodejs packages (in npm), etc etc.
I can name a few problems that make nodejs slightly different, and it's the fact that npm doesn't deal with source packages the way cargo does, it deals with binary packages that can include whatever flavor of bundled shared libraries du jour, probably only built for x86_64 and if you're lucky arm64, and with unclear source and/or build instructions or OS compatibility. Its build system also has support for defining custom build procedures and calling random binaries during build like the npm package for 7zip, which is also prebuilt. This, and the tendency to make microdependencies just doesn't mesh well with distributions that need to make sure everything has corresponding source and can be built reproducibly.
Ruby gems have been solved since they've been around for longer, as have python packages, even though the latter keeps introducing wrenches to make packaging harder.
> Cargo, npm, and friends run rings around apt from the point of view of developer mindshare, ease of use, platform independent packaging, and so on.
Maybe, but they introduce their fair share of problems as well. I've already mentioned the piss poor support for cross-language dependencies, but let's mention that the "stores" the packages for these are published to are only moderated after the fact - it's really easy to introduce malware into these stores without anyone noticing, and it's happened a bunch of times. Distributions make sure the binaries correspond to the sources[2], and the maintainers are the first to spot any obvious malware or breakage in software as a result of a change. Sometimes it goes under their noses as well[3], but these cases are much more rare than npm breakage. In general, the distribution model allows for better trust and security, and I think this is something that should be valued more than it currently is.
[1]: https://salsa.debian.org/rust-team/debcargo/
[2]: https://reproducible-builds.org/
[3]: https://www.theregister.com/2015/06/17/debian_chromium_hubbu...
On the contrary, it's the most important feature of distributions.
I'm not talking applications provided by the distribution, those aren't "applications" (i.e. in my usage, something installed onto an existent operating system), but part of the provided operating system itself, and yes, I'd agree it's their job to keep their provided operating system secure. It's not their job to keep external apps secure (but to provide the means for whoever is maintaining those apps to keep them secure).
...
It sure seems distro inflicted to me. The architecture induces fragility. Apps needing to very carefully step to avoid breaking things is the definition of fragile.
I think there lies deeper, cultural division between developers, who think the world (at least the computer) revolves around their app and their downstream, who desperately try to avoid more and more code shipped unto them for the upstream's convenience. It's not really a difference for upstream if they bundle a dependency or not (or 10k+, in the case of npm), but it quickly becomes a big support burden if you have multiple such "apps" installed.
But only because distributions want to take control of everything. If a distribution just provided the necessary infrastructure to empower developers to maintain their own dependencies (like eg. Flatpak does), we wouldn’t have this issue.
I'm not sure where that's coming from. Distros/packagers (should) rightly feel a responsibility for a packaged application's quality, security, interoperability.
> If a distribution just provided the necessary infrastructure to empower developers to maintain their own dependencies
To stick with the Arch Linux example in the article, is the flatpak you get from `# pacman -S flatpak` not suitable for some reason?
This seems like false advertising. Ensuring an application is secure would require reading it's source. Based on maintainer complaints (like when that python package added a rust dependency) they don't seem to have the resources to even read all the changelogs.
Moreover, why shouldn't we empower libraries' developers to push their versions as they want? They also have limited manpower and support capabilities.
Empowering users is also making it easy to install what they want or need to install, not just what a distribution decides to offer. To give an example, many people need Slack for work, but distributions don't offer it, because it is non-free.
This quickly becomes important when interests misalign, for example when developers want to push incompatible update to users and stop supporting previous version,
Why should a distributor decide what my interests are?
Also, this argument is kind of strange, if a developer is pushing an incompatible update or stops supporting a previous version, will the distribution continue to maintain the application? Not really. Usually, it just means that the application is frozen in time and you can only hope that security updates are applied. If the application happens to be (e.g.) in Ubuntu Universe, you are usually out of luck and you just get to keep a bunch of applications with known vulnerabilities.
https://downloads.slack-edge.com/releases/linux/4.28.171/pro...
Not only that, but it contains ffmpeg and other LGPL stuff without distributing source code, which seems like an LGPL license violation to me.
This misses entirety of what distributions are. Distribution is a collection of software packages that in tandem create uniform coherent unit. They only want to control what they distribute.
As I have mentioned in another comment, one of the core problems is that we do not understand our dependency trees. If you use version "latest" (or even do not audit your dependency trees not to contain such meta-versions) you consciously relinquish control of dependencies. At this point there is no real difference in power and control between trusting apt and npm not to break anything. Trusting npm (or pip, or whatever) is actually even worse, because they explicitly are just registries without any supervision or soft guarantees.
It is choice between a system with bleeding edge / ancient(or sometimes insecure) program mixed together or a system with well audited recent enough programs.
Windows/Snaps go for the first, and apt/yum..etc chose the later. It's just all about it.
The architecture is fragile due to too many layers. Anything is "fragile" in the sense that you can't really design a really complex, generic system with all possible state pre-computed and handled up front, the point is how to handle the fragility.
Classic systems was "the OS is a single application where anything is ready available source code", this was utterly fragile in potential faults BUT far easier to correct any error so in the end far more robust. Think about some CL systems: you'll easily get far more issues in normal usage, but essentially any issue is easy to handle.
The real underneath issue is business: software can't be a commercial product, must be a common open knowledge, like a kitchen recipe, where any party involved can see and act and anything is designed ONLY for the human. We need desktops in a network not dummy terminals of a central service designed to serve their owner.
Kitchen recipes rarely need features added or even bugs fixed. But when this happens, it's usually done by a cook on the spot.
This is only possible due to utter simplicity of a typical recipe, compared to any substantial piece of software. Also, small variations in cooking usually result in pretty edible dishes, while small variations in a program can have drastic results.
So no, sadly, software already cannot be like a cookbook.
Let's say we have a widespread classic desktop system, we start learning it when we start using a desktop computer in the childhood, when we arrive at the high school most already know it a bit, not differently by those who start driving cars: they do not really start from sketch. After university some will know some aspects, some others some other aspects but in general the current common social brain is known enough by most to be mastered enough by most.
At that point a CAD user do know how to fix certain CAD issues and who to ask if he/she can't because at that point IT is a common knowledge enough.
Instead if we keep up the refrain "just learn to click around and there you go, no need to study more" we end up in present impasse where in most professions, not just IT, there is a so deep lack of competence that anything fall apart because even those who know a bit can't act lacking tools to act.
We can't design of course a really perfect system, there always be issues, but we can and should try to tend to. Aiming high to still collect something good.
The maintenance burden for who? Using isolated apps (Assuming that no-shared-libraries is what we want, which is probably true for desktop systems like and less true for security critical systems or "traditional" linux systems) just places the maintenance burden on the apps where it belongs. Yes, unmaintained apps should run out-of-date libraries.
That's certainly not what I want on any system, desktop or otherwise.
Devs want to ship updates whenever they want and force them into their users' machines as soon as possible, and forget about updating dependencies and pin them to ten versions ago so they never have to eat the cost of upgrading them. Devs don't want to make sure their application plays well with others, they just want to package it up in its own universe, the user's hard drive space and bandwidth be damned. Devs never want to finish their software or declare it stable, they just want to ship their machine on Monday morning so users have it installed by Monday afternoon, just to be updated again on Wednesday. Devs want to ship spyware and adware and malware without any oversight. All of this is cheap and profitable for devs but highly antisocial.
Everything is better about the distro model: security, efficiency, and stability, and they represent the users interests even when they conflict with the dev's interests.
"Modern applications" are developed by children and distros are the teacher slapping them on the wrist for misbehaving. Don't bitch about it -- just grow up.
In this scenario (for my personal desktop OS) I don't care about bandwidth, disk space or whether an app uses an out of date version of a library. Having a release per week isn't uncommon and isn't somehow a sign that an app has poor quality.
I'm not going to say a generalization is dead because of one counterexample, but It's perhaps not as clear cut as you make it out to be.
A release per week is a sign that an app has poor quality.
Worse than what? A program with a million features is going to have a hundred known bugs at any point. Getting ten fixed every week is better than getting 100 fixed 3 months from now.
That's the thing though: as a user (not as a developer, I assure you) I want the million features, and accept the thousand bugs in order to get them. But that also means I want the weekly patches. A desktop app with 1000 features and 1 bug is good quality. As is a desktop app with a million features and the same number of bugs-to-features. You might argue that one should count bugs in the absolute, but that's just moving goalposts. With that bar there simply are no complex high-quality apps! Any useful definition of complex high quality app must be relative.
> I suspect you are a developer, not just a user
Indeed (this is HN after all).
Ideally.
In real world though it's more like fix 10, introduce 5 more. So modern software is always in constant beta.
I'd rather have them release every 3 months with proper QA process in place so it'd be fix 100, introduce 20 new or something.
Maybe your sweeping generalization is just wrong. Users, developers, and distro maintainers are not uniform groups with rigid pre-determined opinions and desires.
I have different thoughts in different use cases (for example, I have servers farms and other specific systems where I more strictly batch updates and value a slower cadence), but for my desktop, my mobile devices, and essentially all of my personal systems, I want constant rolling forward progress.
...and with zero security. No 3rd party reviewing it for vulnerable dependencies, homecalling and telemetries or worse, or breach of licenses.
No guarantee of receiving security updates. Especially given that most upstream do not maintain stable release train and do not backport security fixes.
And keep in mind that flatpak sandboxing is only partially effective and it also does not protect the data you put in the sandboxed app in any way.
(Not to mention that more than 4 billion people on this planet have very slow, expensive and unreliable Internet access, and old devices with limited storage space.)
No thanks, I'll stick with distributions.
I thought one of the selling points of Electron et al. was building apps on web standards. What is changing so fundamentally between versions of Electron that prevents apps from running on the latest release? I wouldn't expect Element to not work on Firefox 432[0] after using it on Firefox 431...
[0] I lost track of Firefox version numbers somewhere between 3 and 50
I feel that there was no thought here at all regarding the reverse perspective:
Current application development and deployment practices introduce several issues, such as dependency and package conflicts, which usually require workarounds that differ from one application to another.
I suggest that vendoring, just like forking, should be avoided, for largely the same reasons.
I notice the vscode example involves the AUR, which is not the official repositories. I consider that a broken package, and it should be fixed. One of the packages should rename its main executable or whatever.
Chrome, for example, has various versions, but they ship binaries with different names, e.g. /usr/bin/google-chrome-stable.
The great thing about Arch is that there is one canonical user repository, the AUR. So the official repos should not have conflicts unless absolutely necessary, and the AUR should endeavour not to have conflicts, though it's more of a wild-west so they're somewhat more likely.
However, you generally don't have to think about how to not conflict with whatever random packages may be out there - you only need to consider the repos and the AUR. Anyone making their own packages that are not in the official repos or the AUR, and yet conflict with them, well, that's their problem.
Arch being super up to date and the existence of the AUR really does obviate the need for containerising apps to a large degree, at least from the perspective of ensuring dependencies are satisfied (there are still security reasons to want containerisation). I don't believe I have any containerised apps, and yet I'm running loads of modern common programs like zoom, chrome, discord, spotify, the list goes on.
It's possible to have as many Python versions as you like installed at the same time. All of the libraries and headers include the major/minor version, packages are installed in `lib/pythonX.Y` and the Python binary itself is available as `pythonX.Y`. Few distributions support this well for anything other than a single 2.x and 3.x but it's nothing to do with how Python itself works.
Arch actually supports multiple Pythons just fine, I currently have a bunch of 3.x Python interpreters I installed from the AUR. But installing other Python packages with these interpreters indeed needs to be done outside the system's package manager, e.g. with pip -U or a venv (usually a venv for me).
Well, there's your problem.
Traditional packaging is perfectly suitable for modern applications.
What author is arguing against is the distributions' packaging policies. Distribution policies are what prevents shipping multiple versions of the same library or program, not package managers. These policies are in place because they're necessary for releasing and maintaining a 50000 package distribution.
But for shipping your own software you're free to ignore distribution policies. Package managers have always provided the necessary mechanism. Nothing is preventing you from shoving in all library, binary or data dependencies your program needs into a deb package. You're also free to embed version number in package name and installation paths so different versions could be installed in parallel. The package manager will obey dutifully. Build systems have always been capable of installing libraries into arbitrary paths, and deb/rpm packages are just dumb archive files with metadata slapped on top. If you shove everything needed into it and declare no dependencies, it'll be no different than a dumb tar file and it'll work just fine if you did it right.
You can even imitate flatpak runtimes if you want. Put same things a flatpak runtime has into /usr/lib/freedesktop-sdk-21.08 into a package like freedesktop-sdk-2108.deb, and make your application deb depend on freedesktop-sdk-2108 and you're good to go.
I'm really tired hearing of these baseless arguments over and over over the years. I just wish everyone could be honest and just say "We don't like deb/rpm, they're not cool, they're not hip, we just want to NIH them" so we could be spared of these made up justifications.
P.S. I have nothing against nix. Nix is legit innovation.
One problem with having different applications use different versions of libraries is that you can have multiple versions of a library installed.
Having multiple versions of a library installed undermines some of the reasons for using shared libraries in the first place.
A lot of (most?) software was never meant to be used this way. It's uncommon to see an application that separates its data/config with unique filenames per version, for example. And there's not a global answer to whether or not that data should be shared between versions anyway. [0]
Container-style software distribution largely avoids these issues by not having shared libraries[1] for the most part. This has some benefits like the sandboxing and security features and making consistent behaviour across environments more achievable[2]. But you miss out on benefits from using shared libraries. Surprise -- a software development decision is a trade-off!
[0] Blender stands out for separating config files per version and also letting you easily import an existing config from an earlier version after an update.
[1] or not sharing shared libraries
[2] consistent behaviour across environments is more achievable as the dependencies are shipped with the application (you can ship dependencies without containers as well, e.g. static linking)
see - https://www.usenix.org/legacy/event/lisa11/tech/full_papers/...
It seems that for many devs guix is less of a technical achievemt,and more a way of preventing people from being able to run what they call "non free software". BTW, "non free software" also includes open source software that was released under certain non-compliant licenses.
The biggest issue I have with guix is that any effort of packaging "non-free" software (such as cuda, intel MKL, nvidia drivers etc). Is heavily discouraged, and if it it does happen its discussion is banned from official channels and all authors which make such software available in their personal repos seem to meekly include various pledges in the READMEs "to not advertise this horrible evil thing, or otherwise let people know of its existence". To me this behaviour is very far away from encouraging "free software", but my and their definition of free differ a lot.
My second issue with guix is that the language used to write everything(guile) still lacks many crucial debugging and ease-of-use features such as: being able to display more meanigful error messages, being able to show on which line of compiled code the error/exception happens at, being able to easily interrogate variable value types (there seem to be lots of is_this_value_a_string?, or is_this_value_an_integer? functions, but no what_type_is_this?), no standard way to convert nested objects to string, or to pretty-print deeply nested objects. To this one has to add pretty substantial, but really poorely laid out documentation of the language (from the very first chapter the authors in their examples use all language features that may not be explained until 300 pages in - so the reader basically has to read the whole thing multiple times to "get it"). And so on... All those things matter a lot to beginners and they matter less to advanced users. If this language and guix in general is to catch on, it has to be more beginner friendly.
However as a package manager it really is pretty good. It delivers everything nix does, but uses a more standardised and powerfull general purpose language.
I can't wait until someone forks guix without the whole ideology thing.
What's wrong with the ideology behind it?
It single-handedly made open source ubiquitous, normalized open source licensing and changed the how software is built.
I'm a strong proponent of the philosophy GNU is putting forward (the reality is GNU is just something which aligns my ideology, I'm not a convert), but I understand that both things are not that easy and non-GNU world will always exist in some form.
On the other hand, regardless of the software I use, all software I develop is strictly GNU/GPLv3. I also contribute to projects with more permissive licenses, but my software is always GPL.
GNU may not be able to convert the whole world to Free software, but it's a necessary force to protect our current environment and evolve it for the better.
Moreover, while not pointed at you directly, GNU and GPL is vastly misunderstood concepts. They are neither evil, nor unreasonable. It just aims to move the freedoms developers enjoy with MIT/Expat licenses to users by making them irrevocable.
It's also considerably less popular than Nix, which is already niche.
I have to admit I'm not 100% convinced by the argument. I've been learning guile for a couple of weeks and I just start to "get it". Compared with other languages I learned the difficulty with guile was many many times more (for various reasons). I now see it as a pretty powerfull language, but not without many faults.
The above notwithstanding I think something like Guix/Nix will become really popular when a truly easy for beginners to learn and powerfull language is used (such as python for example). Perhaps it would be an interesting project to attempt to auto-translate guix from guile into python. I can definitely see it takeoff then.
There's no reason to fork Guix since ideology doesn't affect the software itself at all. Rather what you want is make an alternative definitions repo. Pretty sure I've seen such one that includes software not accepted on official channels.
What does it matter, you may say, the software is available! Yes, it is, but because of the whole "please don't advertise non-guix non-free(by their definition) software" it is hard to find to new users. Additionally pretty much all support there is for guix is provided by maybe 5 people on the irc channel. If you go to nonguix that number shrinks to perhaps 3 or 2. It is not easy to get support when the whole community is few people.
By "forking without the ideology" I would like to see guix under a new name, the name new users would search and find an installer that used normal Linux Kernel that supports a lot of software,after installing they would find a (hopefully) large community that is helpful in how to package any kind of software. One where a user doesn't have to hide the fact their trying top package software with proprietary components to get help etc. The "pure" guix has a different mission and it can continue with its libre-kernel . As of now, the software things I mention (guix with normal Kernel) exist, but they are not popular, as a result they are not as up to date as they could be. Also the whole project popularity suffers as a result.
Using shared libraries but then shipping the .so's in a container defeats the point of a .so in the first place. If the reason containers are okay now since disk space is cheap, why not return to static linking of executables?
In the old world, disk space was limited and download speeds were slow. So it made sense for every app to use the system-provided version of Library X.
Additionally, stability and backwards compatibility was assumed. So if the distro updated to Library X v1.2, and your app relied on v1.1, everything would still work.
Today's world is very different (for many users). Disk space is cheap and plentiful and downloading a couple of GB takes an instant. So why not bundle everything together?
Additionally, developers can't be sure of stability in their dependencies. And the app developer is the person who receives the complaints from angry users, not the library developer.
So it makes perfect sense to use a self-contained set of libraries and dependencies.
The pendulum will swing the other way in time. We'll all get annoyed at having a dozen different Electron libraries taking up space, and we'll go back to one-library-to-rule-them all.
But - as someone who used to hate battling dependency hell - I love Flatpack.
True in a data center. Not true for many people.
> So why not bundle everything together?
Security updates, RAM usage.
Even the cheapest laptop you can buy today has literally 1,000x the disk space of a 1990's tower PC. The packaging format APT was developed in the late 1990s when the average user's modem speed was unlikely to hit 56kbps.
Times have changed. Time to update the way we package software.
> Security updates, RAM usage.
True in a data centre. Not true for many people.
That seems like the main issue to me. Why has it become acceptable for libraries to break backwards compatibility left and right?
No, the problem is that production systems need to receive fine-grained and timely security updates.
You can't rebuild, test and deploy a whole OS every time a vulnerability in OpenSSL is fixed. It would be extremely risky for any non-trivial production workload.
Disk space is not the issue. Distributions do the crucial and painstaking work of backporting security fixes.
This cannot be done when applications are shipped with huge blobs of dependencies, like with flatpack or containers, it's simply too much work for any organization to do that.
I thought all the cool kids had adopted `semver` these days, so this shouldn't be a problem? Except for crappy libraries/dependencies, in which case surely the answer is: Don't use crappy dependencies; find an alternative that does the same thing, without sucking.
> Additionally, developers can't be sure of stability in their dependencies.
Again, be more judicious in which dependencies you use. Stay away from the crap ones, find ones that do the right thing.
> Disk space is cheap and plentiful and downloading a couple of GB takes an instant.
Maybe for you. But not for everyone on the planet. Or, maybe people who live in places with intermittent internet connectivity don't deserve new software, or for their needs to be considered?
> So why not bundle everything together?
Security. If a serious vulnerability is found in Library X v1.1, I want to be able to upgrade it to Library X v1.1.1 and know that every single app that uses it is safe, without even needing to know which apps use it. I don't want to have to figure out which apps use it (which, OK, might not be too hard if the tools are written to find that out), and then have to download an update for every single one. Even if I do have unlimited gigabit broadband, that could still be awkward if the library is sufficiently popular and widely used.
Can you access files outside of a container from inside a container?
Can you access devices like an eGPU from inside a container?
Every application should ship every dependency it requires. Attempting to use random shit that may or may not be installed on the system is a disaster.
Containers that include all dependencies achieves this. Infact I like to argue that the only reason containers exist is because reliably launching software is so outrageously complex the only way to do is create a full system imagine. It’s a modern day tragedy.
Disk space is cheap. If every application shipped duplicate dependencies it’d be… totally fine. The downside is inability to update security vulnerabilities. But if everyone is shipping containers that seems kinda moot.
iOS and Android don’t have this issue. When you install an app it just works. Windows has the issue somewhat, but it’s mostly a problem with dev tools that are trying to pretend their in Linux. If you ship your fucking dependencies it’s a non-issue.
Even IF my bandwidth and time were free (neither are) that is still a ton of duplicated work and wear on non-volatile memory life.
Replacing all instances of libFFMoeg.so on a drive is not particularly harder than replacing one.
> that is still a ton of duplicated work and wear on non-volatile memory life.
Nonsense. We’re talking about replacing libraries on the order tens to hundreds of megabytes during a monthly update. It’s peanuts.
Time I’ll give you. But the current system of system-wide libraries is objectively broken. As evidenced by the increasing usage of containers to work around the brokenness. I’d wager far more time is wasted on docker orchestration.
It’s probably worth differentiating between servers and personal machines. Lord knows how many vulnerabilities are in your Steam library. It’s a lot.
but how would you know the new version works with the app?
And how do you even know the app needed this library? Are you going to now investigate every app, and check that the library needed updating?
It's work, even if each individual app has only a small amount, it will add up. And not to mention _every_ user would need to do it for themselves.
How is this any different than replacing one instance of a library that is then used by all applications?
We really need to get this mentality sorted, as we are quite literally blocking everyone else from better technology.
Instead, ensuring that your applications do not break is extremely hard.
Instead of unpackaging a package into a file systme, the package is just a layer that can be unioned in on demand (for live updates, though in the age of containizerization, this hasn't aged as well, so in practice it be running a new container).
link to paper is somewhere in the threads here
And then you get Electron, and people start complaining about the fact that you ship all your dependencies instead of using the dependencies that are already there.
I mean there’s always someone who complains about something. But if Electron apps had a large folder of binaries but it was super fast and responsive that would eliminate the vast majority of complaints.
electron isn't laggy or slow, the app might be because it's poorly written. There are fast electron apps!
The complaint about electron i hear the most are 1) the memory footprint - each instance of electron takes up at least a gig of ram. 2) the browser is duplicated, and cannot be upgraded independently (but i argue that is the point of electron). The third, but irrelevant complaint is from people on high horses talking about wanting native apps, and look down on html based apps.
For a smartphone, the same.
For a server or security sensitive system? Perhaps a different scenario. You wouldn't update a dozen large complex apps every week on such a system.
I think the issue is that the distros with their application trees are trying to solve a handful of completely different problems. Me updating a 5GB IDE and a Game once a week is not the same problem to solve as running a web server.
The AppStore or using a library like Sparkle will update when the developer finds an issue or security issue with a dependency.
The advantage here is that the OS knows about bundles and so gives an organisation above the file. Anything in that bundle should not be used by anything else. I suspect this is what using containers gives Linux
Alternative do as work did 30 years ago - explicitly link to a versioned number of a library. OK then you need to identify which apps use a particular library that needs to be updated for security reasons.
- packages/package management is intimately tied to filesystems and so installers. Actual package models are based on '80s Unix model that do not scale anymore since decades. Such model was designed decoupling "system code" to "user code" (script, mostly) while all along the IT history, both before and after unix we see a better model (see below);
- packages in FLOSS are NOT meant do be made by upstream developers, that's a need of commercial only software. In FLOSS systems packagers of any distro package upstream code.
Now a step at a time: classic systems where designed as a single application, indefinitively extensible by many, and users included. At a certain point in time AT&T/IBM and others say "ah, no that's bad" IMVHO just to deprive users of much power and gain the ability to sell limited, limiting and poorly designed crap. For more than a decade such decoupling take shape. After decades of big mess frameworks spring to life as a kind of middle-ground and they prove to be no better than the decoupled crap they try to solve without coming back to the classic OS-as-a-single-application model. Now essentially ALL non trivial applications being unable to effectively integrate each others try to be "the only one application" and finding no way to do so some pull out of the hat an even worse cage: hiding the OS on someone else computer, used as a bootloader of a modern more limited and limiting Java VM: the WebVM improperly named browser for legacy reasons. Pushing "the network is the computer" to "the cloud is the system".
To solve this mess what we need is the original solution: the OS as a single application where source code is in the runtime, no "packaging" just SCM. Oh, surely that does not solve version conflict between source, these actually are ALREADY solved in architectural terms by the FLOSS distribution model: packagers assemble code. But working directly on source makes that far easier with simple sharable patches who happen to be also a bit distro-neutral. Nix/Guix offer the actual middle ground.
To solve the storage mess it's far more complex. Today most solutions simply use a backend storage service, a kind of DB, no matter if SQL-alike or key-value or something else. SUN with zfs realize a very first limited evolution in that sense, but far from current needs. BeOS with if raw "query-able file system" have done another very small part before. https://twizzler.io seems to offer something more modern, but that's still an open issue.