But if you want a package in the official repos you have to play by the rules they set, the primary one being that official packages ought to only depend on official packages. It'd simply be impossible to have any guarantees about the quality of a package otherwise.
If someone doesn't like apt & the main repository they shouldn't use Debian because that is pretty much the only big thing separating Debian from the tens to hundreds of other distros out there. Take apt and the main repo out of Debian and it is basically Linux From Scratch with a bigger team of bugtesters. Distributions aren't 'jealous' about packet management, they are the packet management. It is right there in the name "Debian distribution". The Debian brand, and all linux distro brands, are deeply linked to how they execute packet management to distribute software.
But that's a policy decision - the dependencies should themselves be packages, because that's the only way to guarantee reliability.
The inability to install multiple versions is probably the only serious problem with the dpkg model. Even then we might argue that the dependent software should be fixed so that it's not so fussy about specific dependency versions.
Of course this is work for application as well as library developers, designing and keeping compatibility is hard. Most would rather build the new shiny, consequences be damned.
When I left that position, we were in the middle of an OpenShift (k8s) deployment, and one of the things motivating the change was the ability to more easily run separate versions of services for different customers if the need arose. Yes, this would put a lot more burden on us as the service provider, but it would also allow us to iterate faster while maintaining stability.
Sure, but this should be considered either a defect or something highlighted by a major version number jump.
Or maybe he agrees and fixes it and it's included in distributions 7 years later.
I've changed 5 jobs in 7 years (because of life circumstances) and I'm not even a job hopper... How is any commercial shop going to plan around 7 year time frames?
It's effectively part of insisting that it's properly Free software - if you can't maintain your own bugfixed fork, but have to keep going to an external organisation for their version, is it really Free?
Also can't install to separate disks.
But that raises the question: why are your file systems sensitive to disk drives? Use a layer (or more) of indirection through LVM and RAID so to make the drive placement opaque and robust.
A couple of systems I have at home have a 16GB primary disk, so it is useful to be able to put things on an external disk. I have yet to see any package manager that can actually handle that.
> Use a layer (or more) of indirection through LVM and RAID...
Yeah, that's a great idea, just introduce more complication and abstraction to make up for a shortcoming in how Linux developers think about applications.
Or, just have your programs be self contained in a single file or folder and put them wherever it makes sense.
This baseless assertion is simply out right wrong. See for example Debian's docs on Private Package Archive:
Maven, on the other hand, is plain scary.
Not criticisms that apply to maven. Binary dependencies on released versions are the norm, and all packages in the central repository are signed with GPG, just as with apt.
> The computing world would be a better place if the people forcing their language-specific idioms on the rest of the world had instead spent their time improving the platform package managers that already existed at the time.
If those platform package managers had been open to being improved, which I don't think they were (and conversely, deb/apt originated because Debian insisted on doing things their own way rather than improving RPM). Apt's closed-world assumptions seem like a policy decision. Apt not being cross-platform is definitely policy. And surely it's occurred to people in Debian that the ability to have per-user installs, and more importantly non-shared/non-overlapping installs of libraries that are depended on, would be useful; the insistence on single system-wide installs of libraries can only be a policy decision, and one that's turned out to result in a less usable system.
choco install s3cmd
brew install s3cmd
https://formulae.brew.sh/formula/s3cmdLet's ignore for a moment all your baseless assertions and focus on the following question: how do you ensure that a build is reproducible if you have no control on which required dependencies are downloaded by a build system which requires root access?
Is jealousy the only explanation that comes up to you?
I use it for most of my dependencies, and software in OS X/Windows WSL/Linux
It is a lovely environment to work with. Has completely changed how I feel about package management and the like.
I know you were asking figuratively but something does happen to slot in that space.
Nix is a non-sequitur. Nix requires specific packages and package versions to be specified, and Nix requires the build process to be "free from side effects". That fails to address both problems stated by the OP, and actually try to reimplement what Debian's package building process already does. Other than trying to publicize Nix, your comment adds nothing to the discussion.
By having control over which required dependencies are downloaded. As Nix does. Why would you settle for not solving that problem?
Sorry to hear you don't think my comment adds to the discussion, but I'd disagree
Again, Nix adds nothing to the discussion. The problem stated by the OP was caused by the (broken) way that the custom build system of a specific package was designed to work, which failed to adhere to the standard practices enforced by Debian's packaging process.
If the packagers of said software project followed Debian's practices then the problem wouldn't exist.
You're parroting that Nix also enables users to specify dependencies. Ok, so it works just like any other package system. What does that have to do with the problem being discussed? Nothing.
I don't think it's fair to say that Nix is 'just another package manager'. It solves many of the stated problems with distro package managers (overlapping versions, user-specific packages, strict build environment rules), and provides many of the benefits of Docker and it's ilk (perfectly reproduceable environments for packages to run, including dependencies that don't fit well into traditional package managers, like JARs). Because it doesn't rely on the standard POSIX filesystem layout, it runs happily on any Linux or OSX system, alongside whatever package manager your system uses.
If people started using Nix recipies instead of Docker or janky bash scripts for deployment of Hadoop and other complex software, most of this article's complaints would disappear. And Nix is, if anything, better from a sysadmin point of view than apt-style packaging systems. It makes a fine distro package manager (see: NixOS).
You're right, sorry for the tone. The thing is, it sounded an awful lot like a blatant attempt at derailing the discussion by shoehorning span to promote a build tool. Nix does not solve anything, particularly as it was being proposed as a solution to a problem that plain old Debian packages do not have. So if Nix solves nothing and Nix adds nothing to the discussion then why waste everyone's time by adding noise by selling a tool that does and solves nothing wrt plain old Debian packaging?
> If people started using Nix recipies instead of Docker or janky bash scripts for deployment of Hadoop and other complex software, most of this article's complaints would disappear.
...or simply build a plain old Debian package?
Is it that hard to simply follow the happy path of packaging for Debian?
Why is suddenly Nix the only option on the table, specially as it brings absolutely nothing to it wrt what plain old Debian packages already provide for decades now?
First, of course, it requires an apt-based distro, so software distributors need to have apt alongside all the other packaging alternatives. Or they just provide a bash script. Ew.
Second, apt doesn't elegantly handle different versions of the same package. That's rarely an issue for well-established C libraries, but it's a big issue for Java and most of the dynamic languages. So you end up with a host of language-specific package managers.
Third, there's stuff beyond simple files that falls outside the wheelhouse of apt. Networking, configuration, whatever. The paradigm for apt is very much to have a small number of systems, manually curated by dedicated sysadmins. When you start scaling up to tens, hundreds, and thousands of hosts, you end up writing and maintaining long scripts to initialize a freshly-installed system and put it in the right state. And those script will break as packages evolve. Getting a system into a known-working state is difficult.
A specific example: I installed and set up GitLab on a Debian system a while back, and it was a huge pain. It's not just a package, after all, it's web code, a couple daemons, a sql db, a redis db, git repos on the filesystem, and more. The install guide was pages and pages long. I never quite got it working right (something about SSL certs, IIRC one of the daemons wasn't using the system CAs?).
So I tried docker for the first time, and had GitLab up and running in about 10 minutes.
And if I ever wanted to migrate to a different host, spin up another node for load balancing, or do backups and restores, you bet your ass I'd use docker.
Apt is great for carefully curated, individual systems. It was perfect for the world circa, say, 2005, and the world would be better of if we'd all standardized on it then. But even if we had, somebody would've invented something like docker in the meantime, for managing complex software (like GitLab) on tens, hundreds, or thousands of hosts.
But docker has all the issues pointed out in the article above, and more besides (every image is hundreds of megs, because it contains a full, running Linux system...that's just crazy).
Nix can do the package management thing that apt does so well, and it can also do the reproduceable, holistic system build thing that Docker does. It can also make management of language dependencies (i.e. Java JARs) much more clean and elegant.
It's seriously worth checking out.
This is wrong and unfair. One of the major problems the OP has with Docker is that many software builds are unreproducible, encouraging many people to deploy binaries of dubious quality. Nix tries to solve precisely this problem through better tooling which makes it easy to ensure that its packages are reproducible and its dependencies easily verifiable. This is also what makes Nix distinct from Debian, which tries to improve package quality through policies and community collaboration.
> Nix requires the build process to be "free from side effects"
I disagree that this makes Nix irrelevant to the discussion at hand. Another gripe the OP had with Docker is that it sandboxes entire apps, making it a blackbox. Nix does sandboxing on a more granular level, which provides more transparency into individual packages.
Your comment sounds very disingenuous, as Debian was first released in 1993 (about 26 years ago) and has been pushing for reproducible builds since 2000.
The .sdeb format packages up sources and build scripts into one atomic, build-able unit.
Well, programming languages clearly should not be in business of software distribution. That creates unnecessary tight coupling between language, build system and distribution, causes proliferation of language-specific package managers incompatible with a platform way of doing things.
[0] We should have also been following 11 but just didn't have the resources to even think about doing it.
I suspect that a big reason was due to using Scala, which is notorious for not maintaining binary backward compatibility across minor releases.
In fact, this is so common that some package managers don't even bother distributing binaries: go modules, cargo, conan, etc.
If I can not install on Debian in a clean way, I will just try to avoid using the software (and everything that comes on the ecosystem). It may be unavoidable, but if there is an alternative, it will be preferred. If one alternative appears after I am dealing with it for some time, it is still preferred (because the work with out of distro software never ends).
You are complaining that you can't automatically install broken packages out of the box through the official repo.
And packaging an application is the responsibility of the people working on that application, not the OS.
What's your point?
That assertion is quite wrong. Just because the OS has its official package repository, and just because OS maintainers volunteer their time to package some software projects, obviously that does not mean that maintainers are responsible for anything. You are confusing offering a convenience service with being responsible for packaging each and every software under the sun.
In fact, your baseless assertion ignores two basic facts: packages are proposed and adopted by volunteers, thus what you've described as "OS maintainers" is pretty much any random person who simply wants a software to be available for download in the distro's official repositories, and packaging systems such as Debian's apt supports private package repositories, where anyone can make available their packages to the world.
Another fact that you missed is that build systems such as Maven or msbuild or cmake or Gradle or whatever do support download packages only for a reason: convenience. It has absolutely nothing to do with where lies the responsibility of providing packages. It's convenient that a build fulfills all build dependencies. However, the responsibility of packaging and distributing packages of a software product lies exactly where it always was: those working on the software product.
The entirety of the Debian project are volunteers, random people who simply wanted xyz. And Debian's raison d'etre is as a "distribution" of existing software.
More to the point, Debian packagers explicitly overrule "upstream" application developers; see the history of cdrecord or the ssh key generation bug for particularly spectacular examples. How can you say it's the responsibility of those working on the software project when they don't get to make the final decisions?
> packaging systems such as Debian's apt supports private package repositories, where anyone can make available their packages to the world.
Not really, because apt's approach to dependencies requires a central naming authority. In practice any private repository is necessarily a dead end: you can package additional software that depends on software from the central system, but not vice versa.
That said, a well-written app should not present any difficulty for the package maintainer.
No it is not. Although distros might pick and adopt some packages and include them in their official repository, it's obvious to anyone that as a distro maintainer you are not responsible for packaging each and every software package under the sun.
Moreso, linux distros such as Debian rely on volunteers to propose and adopt packages, which obviously also includes people affiliated with each software project.
Additionally, it's patently obvious that the responsibility of packaging and making a software available to the public lies on the people involved in developing the software project.
> That said, a well-written app should not present any difficulty for the package maintainer.
That is true, particularly as the primary package maintainers are in fact those actually developing the software project.
(I'm including volunteer maintainers in the latter category, unless they submit the package to the original author rather than directly to Debian.)
Once you get used to this method of packaging it’s actually really empowering for the end user. You can download the source for anything in the system, modify it, recompile, and install it all with a standard set of commands (no one-off custom build steps for every package).
I feel it’s the closest thing to the spirit of “free software” we have today: software designed to empower the end user to read, understand, and modify their system without a huge amount of obfuscation.
It increases the burden on the packagers to reduce the burden on the users. I would be interested to see how often these features actually get used though. I wish distributions would put this front and center.
System package management systems do things just fine. Ask yourself, “when did I last properly package a Debian library? Do I even know how to do it properly?” Most people won’t be able to answer that because they didn’t.
But truth be told, having each programming language a unique package manager sucks. I’d rather have a consistent way of managing packages on a given system so that I can use various programming languages.
The three cardinal virtues of a programmer are laziness, impatience and hubris. I've never seen a decent argument against using ever-smaller libraries (as long as our dependency management is good enough to track them); all of the usual arguments for reusing libraries still apply, even if we take them further than their originators imagined.
> System package management systems do things just fine. Ask yourself, “when did I last properly package a Debian library? Do I even know how to do it properly?” Most people won’t be able to answer that because they didn’t.
Well, as we're seeing, Debian is struggling to keep up with packaging of everything that users want. I have actually created some (unofficial) debian packages; it's fine, but it's not a particularly inspiring experience either. Creating a maven/pip/... package is generally a nicer experience.
> But truth be told, having each programming language a unique package manager sucks. I’d rather have a consistent way of managing packages on a given system so that I can use various programming languages.
In principle I agree. But I'd definitely need to be able to install independent copies of the same library, different packages for different users, and fundamentally just have a nice development experience when working on that package manager. And that's not something I see the sysadmin tradition being able to come up with.
All true. Did we learn nothing from the left-pad incident?
Platform managers have been successful because a) they have the same interface across operating systems, b) they're the only game in town if you want to work in that language, and c) everybody loves to re-write existing software in their favorite new language.
Not really. Language-specific package managers are usually capable of integrating with "system" dependencies from outside the language (certainly maven has decent support for this) - they kind of have to be after all.
> 3) fundamentally, not very good, particularly with regard to multiple versions of the same package (technically, OSGi could do that...), repeatable builds, and the desire for do-what-I-mean behavior.
How so?
I ultimately agree that the system package managers do a poor job of wrapping language-specific package managers. But you lost me when you suggested Maven has a good user experience compared to most OS package managers. If that was a sly joke, then it was a good one.
Not at all. The large majority of packaging systems have reasonable and similar requirements for upstreams, like not bundling dependencies or not hardcoding paths.
> fundamentally not very good, having very limited ability to do things like install packages for a single user or install multiple versions of the same package
That's completely by design, and for good reasons.
> install packages for a single user
because of the security angle, Not allowing this:
> install multiple versions of the same package
is completely indefensible.
Who are you, package manager, to decide that I shouldn't be able to use different versions of the same package? Do you know more about my context than I do?
That's one of main point of a distribution, rather that simply throwing software on a hard drive.
The maintainers have to work hard to guarantee that a specific set of packages and version work well together. Well enough for most users to deploy 99% of the packages without surprises.
And then backport security updates (often faster than upstream), for 3 years, often 5.
And various companies provide longer term updates and maintenance.
Oh, and provide license checking and vetting.
Turns out that it's a lot of work and no serious distribution would consider supporting multiple version of packages while guaranteeing the same level of quality.
> Do you know more about my context than I do?
Given your statements about distributions... most likely yes.
Hello there, ad hominem.
Linux distributions are overreaching. They should offer those services for core software. I should be able to install Inkscape 0.91 and 0.92 at the same time, easily.
* compartmentalized side-effects (not per-user even -
users might decide they want multiple compartments for
different apps) and
* a shared storage area for all the packages and their
versions (i.e. Nix)I'm sad that this is not happening since in the process we're also losing the good bits of system package managers as per this article.
This is just wrong. Package managements support simple models by design so you don't end up with a mess of conflicting and potentially insecure dependencies in deployment environments.
Use maven in your build pipeline, fine, but distribute a minimal package using tarball/RPM/APT/etc. so that only the minimal runtime dependencies install on the deployment target. In fact, I built my first RPM using the maven rpm plugin. It takes just minutes to set up and use.