At the same time, it is sad to see how much work you guys need to do for things that should have been automated or done by the original developers... Let's hope AppImage changes that
Edit: Changed flatpak to appimage
https://www.debian.org/doc/debian-policy/
Sections 6 thru 12 are what it means to be "a Debian" rather than say freebsd packaged into .deb files.
For some very end user applications that don't interact in any way with anything else, games perhaps, that works pretty well, until something is run into that does need to interact with other components also operating under the same anarchy.
The problem with not having a closed system or standard or method of operating a system aka an operating system, is you end up with the deployed machines having an Apocalypse Now quote conversation "Are my methods unsound?" "I don't see any method at all, sir." If you're a system Administrator what does it mean to Administrate mere anarchy?
PS: Thank you, Riku. I never understood the sheer amount of effort involved in being a packager / maintainer until I started doing the same task for NixOS. I regularly depend on Debian's excellent patches and CVE details to do my work. May you not be bogged down by nirvana fallacies :)
I think the Solus guys would have a word with you. Even with no packaging experience it is pretty simple to create a package for Solus Project.
Here a short intro:
You can't just upload it and forget, you have to sign up and become associated with that work, you have distribute your keys and get other people to trust you, you have to make a bug and file the package against that bug, etc. There is this whole social dynamic within Debian that I just don't understand at all.
If it was more like software development; make the package, git commit, push, post it to software like reddit and have people approve (upvote) it, then I'd be a maintainer already. Instead there's this whole process behind becoming a maintainer, finding a "mentor" etc, that for years I've just found to be a complete roadblock.
Leaving aside the specifics of Debian's tooling and social process (undeniably, these are arcane and complex), I suspect this has a lot to do with why Debian is such a reliable environment over time in a way that few software collections manage.
It would almost certainly be better if it were easier than it is to participate in Debian. I've thought about contributing for years, but I certainly haven't had the energy to clear those hurdles. On the other hand, there's something really important in the distance between typical modern software publication and packaging for an ecosystem like Debian.
I think we'd be a lot better off if more people were committed to the hard, tedious process stuff that renders software accessible to users, a good neighbor to other projects, and maintainable over the long term.
That said, the Debian processes are over 20 years old. I find contributing to the FreeBSD ports and MacOS X homebrew package collections simpler, and without the same level of jumping through hoops. Homebrew's git-based submission, review and CI testing is simple to use. Likewise submitting a patch for the FreeBSD ports. Debian could do something similar, but its workflows predate this significantly. Were Debian to adopt a similar process, I think it would make the process significantly more transparent. The existing practice is still oriented around single individuals maintaining and uploading single packages (though it can also be done by groups with their own private version control for the package/group). The newer methods are significantly more open with much lower barrier to entry.
Upload and forget leads to dead packages that get removed next time there is a library transition. Debian is about long-term maintenance.
The social stuff (as well as the social contract and DFSG) is also what creates trust within and towards the Debian community and holds it together, which is the main reason it has lasted so long.
Already often a big part of Debian packaging work is making the software compile outside the developers personal enviroment without manual steps - or in the other end decoupling from the developers CI loop.
The point was that a lot of packaging would be done by the devs, allowing minutes maintainers to so more important stuff.
For interested readers, here's some best practices for being a good upstream:
- Just use the GNU autotools. Users expect `./configure && make && make install` to work. Too often people roll their own configure scripts and Makefiles and they always miss something important. Distros expect there to be certain knobs to tweak, and configure scripts and Makefiles generated by the autotools have all of them.
- Don't bundle third-party dependencies. For security (and for better documentation of the true dependency graph) distros often must go through extra trouble to unbundle third-party libraries when present. Some project even add their own custom patches to their bundled source. Resist the urge to do this.
- Include accurate copyright information. Put a license header at the top of every source file. Any serious distro will need to do at least a cursory inspection of licensing info to make sure it meets requirements.
- Make proper source release tarballs. Do not depend on your version control tool being available at build-time. Do not depend on the autotools being available build time. Use 'make distcheck' to make a fully bootstrapped tarball to distribute.
- Do not make any use of the Internet during a build. That means no downloading third-party libraries, pre-built binaries, etc. It's imperative that a build can succeed without network access, and some distributions isolate builds from the network to ensure they don't misbehave.
- Do not hardcode absolute paths to binaries. No /usr/bin/bash or etc. Your assumption will surely fail on a non-trivial number of systems. Find the location of a binary at configure time by inspecting $PATH. GNU autoconf can do that and substitute the absolute file name where it's needed, such as in a script's shebang. The same advice can be applied for anything else you need an absolute file name for.
- Do not assume /usr exists. The Filesystem Hierarchy Standard is not as popular as it used to be. This is a more generalized form of the previous point. Again, if you use the Autotools you will be doing the right thing by default.
There's surely more, but that's what I can think of right now. Surely a Debian developer or someone else has compiled a more thorough list. Anyone know of one?
I think that today's software being so difficult to build is making practices that are frowned upon by distributions (for very good reason) seem like acceptable solutions, which leads us to the growing popularity of Docker, Snappy, and Flatpak. "This software is nearly impossible to build, so just use my {Docker,Snappy,Flatpak} bundle!"
tl;dr - Make your software easy to build, don't just package up a mess.
I'm not saying packaging is easy, and I do try to make it friendly for distributions, but don't pretend that doesn't make it worse for general users in the process.
And I'm not saying that you should never provide some prebuilt binary to your users if their distro is lagging behind. And if you really feel the need to bundle third-party libs then just make sure there are configure switches that can be flipped so that system libs are used instead. The best thing for users is for them to be able to get all of their software from their distro, and that requires distros and upstreams to each do their part.
But please, if you decide to bundle third party libraries make sure you can build without them and use ones provided by the system instead. It's a political nightmare to include packages with bundled libraries because it makes security updates a huge headache since we can't simply rely on Anitya (https://release-monitoring.org/distro/Fedora/) to send us notifications that a new release of the library is available, not to mention the extra work of actually updating the bundled library once we do find out an update has been published.
You've pretty much hit the nail on the head. You forgot one additional bit though, please for the love of god don't have a crazy web of dependencies.
I see a lot of Node and Ruby apps online that I think would be incredibly useful in the Fedora package collection and have considered contributing them on more than once occasion. What always stops me is the 50+ dependent NPM packages or Gems they require that aren't already packaged by someone else.
The incredibly annoying part is most of these packages provide minimal functionality that you could have just implemented yourself, or that shouldn't in turn need another 3-10 transitive dependencies of their own. I get that not re-inventing the wheel is generally a good idea, but please try to pick your dependencies wisely if you want to see a distribution include your package - because a volunteer maintainer likely doesn't want to be responsible for your package + a dozen or more dependencies if they can avoid it.
I see more and more programming languages trying to bundle their own dependency management tools 10 years ago I thought CPAN was great nowadays I'm not as sold it's basically reinventing distro style package management but in a way unique to each programming language.
[1]: https://www.digitalocean.com/community/tutorials/how-to-use-...
The real problem comes when developers start using these package managers with reckless abandon and letting their dependency tree grow out of control. I don't mind packaging an extra library or two, but a dozen or more is pushing my patience.
I packaged a Github clone called Gogs (written in Go) for Debian/Ubuntu, complete with Lintian support. But I had to compromise on the 'rules' file and add a "get-orig-source" target that uses Go's package manager to grab all of the dependencies. I used that rule to grab all of the source files required to create the source package (which can then be built in isolation).
But if I understand Debian's official packaging rules, this is verboten because it winds up including a bunch of interconnected third-party libraries. Since I didn't write Gogs or any of its dependencies, I can't exactly go through and eliminate all external dependencies. And even if I could, much of Go's standard library exists only in ecosystem form.
How should a prospective package maintainer handle these kinds of ecosystems? Trying to distro-package every library (Perl-style) would be a Herculean effort, and could conceivably be met with hostility by the upstreams.
There is so much software being written in Go/Rust/Ruby/Node/etc. How can we go about packaging it?
In the meantime, we can use the information available in these language package managers to help bring that software to the systems package managers. How easy it is all depends on the language. If the language/package manager is sane and the build system isn't conflated with the package manager, we can make quick progress by writing importer scripts that automate most, but not all, of the work. Node, Go, and Java are utter nightmares for various reasons. Python is pretty good. Ruby is somewhat annoying but doable. It seems that Rust is decent but I haven't used it. All I know is that someone recently wrote a Crate package import for GNU Guix (the package manager I contribute to and recommend highly) that seems to work. [0]
[0] https://www.gnu.org/software/guix/manual/html_node/Invoking-...
Would you happen to know a good video or writeup on why language-specific package managers are a bad idea? I mean, the situation with C and C++ libraries seems significantly worse to me, and I personally really enjoy having the fully Crates.io index at my disposal on any box that runs Cargo.
The two styles of managers just have different goals. That is, a package manager such as apt has the goal of creating a single, harmonious system from stuff written in many languages. But a language-specific package manager like Cargo has the opposite goal: to provide a good way of writing software written in one language across multiple systems. This is where most of the tension comes from. The rest of it is from the same general structure, but with different specifics: the goals of these kinds of systems are very different, and conflicting.
Software is hard.
I think language-specific package managers are fine for easily sharing source code amongst developers using the same language, but they shouldn't be used in a production system.
Nix and Guix don't work on Windows, right? They're still not close to a solution until they do.
The third sentence is not a contradiction. I'm just saying that I can live with people using language-specific package managers, but really they would be better off with a general-purpose one.
https://en.wikipedia.org/wiki/List_of_software_package_manag...
If any of the dependencies aren't currently packaged in Debian, how would one follow this guideline?
If I wanted to package my third-party dependency, the first thing I would do is "learn about personal interests of sponsors" and see if my third-party dependency and a sponsor's interests intersect. There's a link to a page describing the sponsoring process, where apparently I'd file a bug against a "sponsorship-requests" pseudo-package and then, I guess, wait.
Next (or perhaps concurrently) I'd file a separate "Intent to package" bug against the "Work-Needing and Prospective Packages" pseudo-package. There's a whole page about WNPP and format guidelines for submitting said bug using the "reportbug" tool. Those format guidelines are longer than the JSON spec.
Then I'd still need to make the package, after all. That link you gave lists five important reference materials, one of which is said to be "must read" and has 12 chapters and 7 appendices. There's also a "New Maintainer's Guide".
Then I need to publish my package. There's an account to sign up for. Plus I'll need to create, keep up with and sign stuff with a GPG key because uploads are http/ftp only.
Once that is finished I apparently get an email response. Finally... I am done!
Now it's time to find a sponsor.
There's a whole section on what to do if you can't find a sponsor. The first is to follow up on the WNPP request I was supposed to make six paragraphs ago. The other is apparently to look up sponsors in a sponsor search-engine on the Debian website and bother them.
Then there's another section on actually getting the package into debian through an ftpmaster. (Both the sponsor and the non-Debian Debian-package maintainer are ominously reminded here that the ftpmaster's _opinion_ on inclusion is binding.)
And then maintaining it.
I would be, for the life-time of my application, maintaining the Debian package of one of my third-party dependencies. This, in response to my query about how to be a good upstream citizen in the hopes that downstream maintainers can more easily package my application! :)
It is in many scenarios. If I have a little app I want to package then it becomes my responsibility. For commercial software I make it always is and this is part of the reason that linux sucks for commercial software.
And then there's issues like security patches. Developers need to know what branches are used downstream.
You are talking about proprietary software, where developers have unjust power over users. If you want to distribute such software then yes, you have to do the work of making binaries for each distro you want to support by yourself. I would argue that it's not GNU/Linux that sucks here. If instead you gave your users freedom by using a free software license on the source code, then others may package the software for use on the system of their choosing.
In the cases (which happens more than you might think) where an upstream developer has actually got really good packaging, I've usually taken the approach of tidying up the last few details and committing back and then mentoring them for a while with them doing the majority of the work and I just double check and upload it.
The level to which people are willing to work on it varies from accepting patches that affect the distribution package version (e.g. hardcoded paths or porting issues) to actually doing the packaging.
If you're interested in helping out with Debian: https://www.debian.org/intro/help
If you're an upstream developer and you'd like your package to be in Debian: https://wiki.debian.org/UpstreamGuide
I am upstream for some of my packages and I don't provide the same debian/ directory upstream as I use for Debian.
I've heard time and time again that there are issues with Debian's version being out of date, but this doesn't have to be this way, at least not by policy. If there is a backport maintainer (doesn't even have to be the same person working on unstable) then the latest version can be installable within a stable system giving you the latest and greatest of the applications that need it with a stable base underneath.
Thats very nice. Although I wonder what Debian's 'official' container format will be.. almost everyone else seems to be standardizing around Flatpak so it'd be nice if Debian did as well, especially since that will mean Ubuntu (hopefully) will switch to it in the future as well instead of Snappy, same as the whole Upstart -> SystemD story.