PackagingCon – A conference only for software package management
packaging-con.org
packaging-con.org
Attending in person won't be possible, but I'll have to keep an eye on this conference and see if they provide a way to post/share such opportunities.
Also, you could sprinkle in some 'optional but desired' skills and experience relating to packaging into your advertisements, anything from CMake and GNU Autotools to Nix and Podman. That would be enough to pique a packager's interest and differentiate it from the 'write-only' coding jobs!
I’ve not created a Python or Rust package or something like that. If you’re creating that type of package, my guess is that it may help a lot to hire a developer who loves to think about the most “idiomatic” way to package your code.
Edit: if your package will be published and used by customers, then you might look for DevRel skill as a bonus.
They’re not going to grasp everything as well as somebody with a development background, like learning NEW programming languages on the fly and upstreaming changes, but I don’t think it’s totally outside of what they can learn and hiring a developer would present just as many challenges.
I might look more for somebody who bills themselves as “devops” or “sre” than sysadmin though since those job descriptions tend to effectively filter out those who can’t program at all. If anything I see those people as being software developers that specialize in systems administration.
I don't want to talk about ageism, or hiring biases, but I'd imagine if you found anyone who had an SRE-role and a long history they would be extremely likely to have had serious coding experience.
I've run into a few "young" devops engineers, and they tend to be more familiar with k8s, CI, and similar, rather than actually developing. But I guess that could well come down to the random people I've met, I wouldn't like to generalize.
Usually they don't retitle themselves, companies do.
It seems the whole IT industry just collectively decided to rename the sysadmin role as devops when using cloud and containers technologies regardless if they actually use DEVOPS methodology or not.
That is my history as an engineer at least.
But these days my titles are Devops-This, and SRE-That. I find it hard to take too seriously, but I guess it is what it is.
What one needs to understand will vary so much depending on the upstream package and how it's normally built (what it expects) that I think even someone with experience in packaging will sometimes get stuck or need to seek help or advice. The more substantive the porting work is, the more that will be true.
That said, this kind of thing is pretty common for Linux distro packagers, and the weirder the Linux distro is, the more true that is. It's not at all unusual for Nix folks to add special build params, patch upstream software, or mangle binaries with sed in a post-build step to get them to understand NixOS' unusual filesystem structure, or to build offline in the restricted sandbox.
So I'd reach out to contributors to BSD ports systems, especially if they added ports for Linux-centric software, and contributors to really weird Linux distros (NixOS, GuixSD, GoboLinux, Void Linux, etc.).
I've never been in charge of hiring decisions anywhere so idk what the overall landscape of DevOps job candidates looks like. But I've done DevOps and I definitely do, and my friends in roles associated with that title can definitely do at least some coding and scripting in a way that I think suits most packaging tasks well. If a DevOps person has a CS degree of previously worked as a developer, they can probably figure out what they need to when it comes to patching upstream software to get it to build. Even if they don't, if they do some recreational programming, they can probably push through.
Honestly, sometimes I wonder if this is perhaps an under-considered area for Generative AI applications given how much drudgery is involved yet precision tends to matter.
There's nothing sustainable as an industry about something like software dependency management and versioning being so critical for things "just working" while people are sticking their heads in the sand saying "not my problem" as a rule. I'm so tired of solving the same problems that were already solved in the past because of developers wanting to work on "more interesting things" and doing the bare minimum to get stuff compiling and running again regardless of the package format. In fact, Docker may have made this situation even worse with so many Dockerfiles that will basically be unable to work. This is part of why I'm interested in the Nix ecosystem because there's so much work being done to make software installations repeatable and easier to manage. Unfortunately, it's squarely aimed at developers and is way, way too complex for most sysadmins to put the time into understanding, especially when they're so overwhelmed like everyone else with so many tasks.
As to docker. Yes I also think it made stuff worse in the majority of cases. What people forget to grasp is that Linux be it a vm or a container is just a the kernel and that we have dependencies for some programs and libraries to the kernel. Reason why we have distros in the first place. So not all containers might run the same on each host because it docker runs on the same kernel as the host machine or the VM which installed (macOS) Issues are rare but can happen. Also the number of base images. „Let’s use alpine because that means I build a small container image“ … No it means you build on top of an alpine release with all the installed system libs etc. And that may or may not work with your software. And stuff like that.
How long is acceptable for your packagers to struggle with learning a build toolchain and its quirks? How many language ecosystems are you working with at your company?
> And don‘t get me started with CI etc.
Respectfully, I'd like to get you started on CI. I kinda feel like the landscape of CI platforms right now sucks, and a lot of my daily work is integrating with them. I'm curious what your gripes are!
I also don’t like the current CI landscape. I still run a Jenkins at work cause it gives me the most freedom and I know it best (it’s sadly what keeps me off other systems, at least I know what parts not to touch). We use gradle which I really liked 5 years ago. We also use it for other non Java systems since it is pretty adoptable. But the recent version changes bring more and more weird changes which make it harder and harder to maintain custom plugins for it. The rules to code build plugins are more stricter enforced by the compiler which yes is good for gradle but hard to maintain as one needs to read up and understand why one can no longer access a file input outsite of a build execution phase for example even though it worked fine for the last 4 versions. And then there is more.
OP: post here: https://lists.debian.org/debian-jobs/
Actual package specs require effort for good reason. It's not just an archive... but interdependence, steps to perform on (un)installation, changelogs, and so on.
There are some helpers already provided, ie: RPM macros.
Sure, they're esoteric, but show me a specialization that isn't. Refer to the Fedora packaging guidelines and enjoy life.
Especially with DEB and RPM, where the packaging format supports arbitrary hooks that run as root, this is a big deal. High quality packages that meet distro standards will inspire confidence in customers' sysadmins. Substandard packaging may do disservice to your core software's brand, if your actual software is more solid and thoughtful than the packaging.
I question how one gets here, nay, the 'problem' we're solving.
Presumably there's config management involved already capable of shipping bits. ie: the repo definitions to find these hackjob packages, or deliver them directly.
If this is what you're willing to invest, go Slackware at it - use a tarball. You aren't gaining anything notable by throwing an archive at a packaging format.
The meta that FPM ignores is what provides packages their value! If changelogs were at least a part of it, I'd be a bit more accepting.
Otherwise, I see it mainly as misappropriation. Best case, naive and well served. Worst case, giving the impression of better distribution than actually exists
An organization that does packages, but leaves this as the answer, fails itself and the members. Over 90% of the purpose is dutifully ignored/not standardized
You want people who, when tasked with creating a package for a distribution format that is new to them, look not to tools like this, but to the conventions and standards of successful distros which use that format as examples.
You want 'native' engagement with those packaging formats, not hacks like FPM. The parent commenter's suggestion is an excellent one.
It's for sure a ton harder to use (IMHO) because it mandates the creation of a manifest yaml, versus "fpm -s dir -t deb my-directory && echo tada" but not having to deal with ruby (or docker) can make it a better fit for several circumstances
The good news is that I learned a tremendous amount about where programs and files should actually be installed through packaging. It was surprising to me how many conventions package managers enforce. I was also surprised how poorly software followed those conventions when installed without package managers. Packaging improves user experience, developer experience, and quality. Can’t recommend it enough!
At its toughest, packaging is a long series of exchanges: one build error for another. It is a stubborn, patient, rabbithole diving, well-I'll-be-damned, yak-be-shorn kind of work.
> The good news is that I learned a tremendous amount[.] [...] Can’t recommend it enough!
And it's totally worth it, both for the packager and their users. :D
As others have said, what you’re hiring for is more of a specialised sysadmin role. I agree with leaving the software engineering label behind.
Good ones are a bit rare... as this tends to go
They tend to write automation/utilities that help keep the lights on - packages are an important part in distributing that.
I feel like I missed something obvious here. Doesn't this answer your own question? I've worked for companies before that found great talent for very unique skills by contributing to similar projects and reaching out to frequent commit authors.
I told you us Nix weirdos enjoy packaging, GP!
Why is it so hard to accept that the best candidate you will ever get may be someone with relevant but not _exactly_ the experience that you want?
I do packaging work at my current job, and it's one of my favorite tasks there. I currently maintain a small collection of private Nix packages for macOS and Linux, and last week spun up a private Homebrew tap (for Casks only!) on a lark. I have some experience packaging things in other formats, too, and I'd be happy to pick up a new one if I had a use case.
But it has literally never occurred to me to look for a packaging job specifically! I didn't know roles dedicated to packaging were available anywhere.
> How would someone/a company go about finding people that are actually interested in this kind of work?
Even when I'm happy at work, I'm generally interested in keeping an eye on jobs where Nix skills are desired. 'Nix' itself is hard to search for (because it's ambiguous with *nix as an abbreviation for Unix-likes), so I also search for 'NixOS' and 'Nixpkgs' on job search sites sometimes. I don't think I've ever once seen a result for 'Nixpkgs', so any listing that mentioned that would stand out, and a job explicitly titled something like 'software packager', 'software distribution engineer', 'package management engineer', etc., would definitely catch my eye.
Job boards and development mailing lists for large package collections (Linux distros, BSD ports collections, and other collections like Conda) would also likely have some interested people! (In the case of the latter, just make sure that job posts are in line with the norms of the list.)
Communities surrounding advanced build systems (Bazel, Buck, Pants, etc.) would also probably be good places to look.
The reason most find it boring, uninteresting, and drudgerous (I would also add "frustrating" to this list) is because one has to build on top of decades of bad decisions and quick hacks, over and over again. Which also suggests a way to make it interesting: make it about fixing the underlying problem (even if in a limited context) instead of just going along with whatever is available. For example, you can attempt to automate producing various packages from some "sane" common metadata.
You didn't provide much context on what exactly you are trying to package, but here is one example of what it might look like: https://build2.org/bpkg/doc/bpkg-pkg-bindist.xhtml
I also helped build a web based system to track approvals and schedules for all the changes.
I think Dev Ops or Systems Engineers would be the most used title for this type of work.
Depending on your needs I'd first try to not need the expertise in the first place (a generic public package manager is much more complex than say, a tightly controlled plugin system), or try to leverage existing package managers/tooks.
If that fails/not possible, and you cannot find a proper fulltime dev, many open source devs working on these topics might be up for consulting, so I'd reach out to them to try to set the general guideliness/direction and then use in-house talent to fill in the gaps.
[1] https://umbrellajs.com/ as a clone/alternative to jQuery
https://www.debian.org/consultants/ https://lists.debian.org/debian-jobs/
I wouldn't know how to find those kind of people today.
Just something to think about when interviewing those sysadmin types (recommended by others).
If your a developer working full time in only one or two languages you may never experience just how good/bad you have it.
When you do, its really eye opening.
Every time I transition to a new language professionally it can be like opening a bag of Bertie Bott's Every Flavor Beans when you look into the packaging story.
* Go binary release story is great but the gopath method for dependencies is annoying
* Elixir has lockfiles and built-in package docs but the release story deviates too much
* Javascript now that everything has settled into npm is a delight but the lack of stdlib, painful local aliasing and extremely heavy node_modules folder can be offputting
* Python just sucks (lets hope poetry can bring the promised land of deterministic builds)
PS. If you find yourself in the same situation I do, Repology is your friend: https://repology.org/
I'm not aware of any language ecosystem package managers taking similar measures to ensure that dependency declarations in their packages are complete.
No, because the OP was comparing them to language package managers, which are widely supported on Windows
That is one reason I didn't specify the solution.
Trying to release libraries and language upgrades when you have to worry about different versions.
One of the reasons Rust has been able to move so rapidly is that it is not dependent on system packagers.
* Rust is awesome, and nothing can beat it. Cargo (the package manager that ships with Rust) is the best thing since sliced bread! Every other language can suck a bag of burritos.
I'm not meaning to criticize Rust or Cargo here; they're both doing their jobs just fine. But I do find myself craving a different compromise for much of what I/we use Rust for today. And I'm really hoping that Wasm Components (and WASI) will be that different compromise — e.g.:
- Don't rebuild an entire HTTP stack from scratch for every tool that happens to use HTTP. - Therefore most projects won't have enormous `target/` dirs. - Reuse components built in CI for Linux directly on a Mac dev machine (cross platform). - Mix and match new processor architectures available in AWS without building two versions of everything. - Also reuse some components in the web browser (yes, we actually have real, boring use cases for this). - Wait, I'll be able to do all of this and still be writing Rust? Shut up and take my money.
Some clever garbage collection would help, too, but I imagine different people would have different and very strong opinions about how that should work.
The fact that two projects that use the same library may enable different features also complicates things. Again, a of this can be mitigated.
I'm looking for a step change for application development, though — i.e. not 5 minute build reduced to 2 minutes, but 5 minutes reduced to 5 seconds (and a correspondingly tiny target directory). That's what excites me about WASI in this context at least.
Leaving WASI aside for a minute, I do wonder how much more could be saved in local disk space and compilation time across projects (and hosts, a la sccache) if this was a high priority goal for the Rust project. E.g. even if the MIR for a crate with two different sets of feature flags enabled ends up substantially different, would they still compress well against each other if a lot remained common?
It sounds like what you might want is a shared global dir that uses hashes of feature flags to separate crate installs in the same target directory, then some after-the-fact GC to hardlink matching files across different builds of the same crate. Then you can symlink from there into your local project target.
With GOPATH, I can package a single library and use that package to fulfil the dependency on the library of any other Go application offline. Conversely, with the vendoring mechanism, the Go tooling will need to download (or copy from cache) the library when you originally create the vendor directory or when you add a new dependency to the root project.
Like if my codebase has webapp A, library A and library B rather than separately defining that they all use third party library foo v3.1.4, it would be really nice to have a single source of truth.
We had two large dependencies: the JDK, and the app server (our container, like Tomcat). Maven, that we’ve had forever, for library dependency. The resulting War files were effectively self contained.
JDKs were trivially installed, explode it somewhere (anywhere) and set JAVA_HOME. App servers were the same. Self contained directory trees. Install as many as you like, just change the port they used. Or just add another War to the one you have already. Pros and cons.
War files were essentially static linked, carrying all of their libraries. Self contained blob you could drag and drop into a directory and watch the server shutdown your old version and fire up the new one.
Sure, we had our bit of DLL Hell, rarely when building the app. I’m certainly not going to suggest we never had class loader issues.
And enterprise Java has a notoriety all its own, but packaging was pretty low on the list. But we didn’t need Docker, or dedicated VMs or anything like that. Our OS dependencies were all handled by the JDK. Java didn’t have any shared library concerns. I honestly never questioned it, either it was statically linked, or just entirely self contained outside of the C library. Everything else was in Java. OpenSSL, for example, was never an issue.
I’m on board though, I loathe packaging. I’m not a big fan of arcane parameters passed on over the council fire and tea. Just never been my drive.
It definitely does have such concerns, but the community has just accepted that the JDK will never help with this and so every JAR that uses a native library hacks around the lack of proper support in its own unique and special way. Usually some project-specific ad-hoc code that extracts native libraries into some directory in the user's home directory.
Is the "cache" versioned properly? Maybe.
Can you control where it is? Maybe.
Code signing? Probably not.
Can you ship only native code for the machines you actually care about? Maybe.
Does it work when you run as a dedicated server UNIX users that doesn't have a home directory? Nope.
Hydraulic Conveyor (https://hydraulic.dev/ - disclosure, my company) fixes a lot of these problems automatically. But it's not like there are no problems to fix, and of course writing a library that uses native code is still a big pain. It's a major blind spot of the JDK unfortunately, and the community has never risen to the challenge of fixing it.
https://conveyor.hydraulic.dev/10.1/configs/jvm/#native-code
So it's really easy. Just run System.loadLibrary("foo") before doing anything else. On developer setups that line will throw because the JVM won't find the shared library, so then you can go down the road of extracting things to the homedir or whatever else you want to do (or do it manually and check in the results). Deployed installs will find the library in the right platform specific directories inside the app package and pass.
what about global folder with most of the packages zipped with yarn?
Software is so complex, dependencies so deep that we have to be experts in both minimising our dependancies and in recreating them from the ground up.
In every team I join my first thing to hang on about is recreating the same builds time after time.
Now there's for sure a spectrum there, since "can't" could mean a lot of different things to different people, but a good straight-face test is whether they have a CI build specification in the public repo since building in CI and a newcomer trying to build the repo often have very overlapping concerns
Being a Librarian is a skillset that takes years of study. Library Science. https://www.bestmastersdegrees.com/best-masters-degrees-faq/....
Local dev, cloud dev, CI, production – all with the same config file. Fingers crossed my talk submission for PackagingCon gets accepted. It'd be awesome to share this new way of working with a wider audience.
With regard to your "CI" use case, this is part of why I said I must have improper expectations because
$ docker run --name dbox ubuntu:22.04 bash -c 'curl -fsSL https://get.jetpack.io/devbox | FORCE=1 bash; devbox init; devbox add awscli2; devbox add zulu@11'
$ docker exec dbox du -hs /nix
4.1G /nix
When trying to use it locally on macOS, this here is just some "you wanna do _what_?!": https://github.com/DeterminateSystems/nix-installer/tree/v0.... (not to pick on determinate.systems, the upstream is similarly facepalm: https://nixos.org/manual/nix/stable/installation/installing-... )
Just goes to show how we are (or at least, I am) used to $PROG_LANG conferences, or Agile/SysAdmin/SRE/Mobile confs. Could be cool to have linting conf, code editor summit, etc.
2021: https://2021.splashcon.org/home/conflang-2021
2023 (upcoming): https://2023.splashcon.org/home/conflang-2023
* A new programming language and ecosystem could try to solve package management from day 0. ScrapScript is an example of this. I've heard good things about Go and Rust.
* You can make package management fun by thinking of it as a data flow factory and like Factorio (which I've not played but I do get the feeling of that game) As it stands it's just lots of tedious boring busy work.
* If only dependency usage was as enjoyable and straightforward as shopping and arranging bought things in a room.
* I am investigating the modelling of packages as bundles of types foremost and state machines that can be traversed by the package manager to determine state interactions and compatibility automatically.
* Changes to packages break everything. You could diff ASTs to see what's different.
* I don't enjoy breaking changes. I have some old projects where I never pinned versions that cannot be built because I don't know what versions they work against.
having played the game, Factorio is way more fun than package management
Even JS/NPM, if it has this, would be a lot nicer to use. I find Python nicer than NPM just for this reason even though people grumble. And .NET even better. Elm is perhaps the best package manager because of the focus on developer experience there.
Another good tip: have just one package manager for your ecosystem!
This is something that Linux distros had to solve right from the start, so it's built into the concept, but I only very rarely see it done at all, let alone done well, in language package managers.
That was mostly a joke.
Excited to attend-- this is a topic that's becoming extremely important especially in the ML world where dealing with dependencies is a total nightmare and most of the solutions we've seen don't scale well to large orgs.
i really miss the conary packaging and build system. it was not perfect, but it essentially put packages into a revision control system so that for one version numbers of packages didn't matter any more. the whole set of packages for a release was locked into place so you could have a new release of the distribution with an older version of a package. and you could switch distribution versions like you can switch branches in git.
at one point my system was so messed up that it wasn't really usable any more. even installing or removing packages didn't work. but i was able to run a command that would switch to the latest stable release version. conary then shuffled around downgrading several packages that i had installed to the right release version and getting me to a clean release state, so my system was workable again. neither rpm nor deb systems are capable of doing that, and i am not aware of any others either.
Give me a Makefile any day. Yes, that's a specific choice of top-level tool, but it's a better choice than most for that specific top-level job because you just shell out to whatever else you need. Not having to rebuild dependency management is a win, too, where you can take advantage of it.
Lorem ipseum -- whoops!
Bindle seems interesting but I'm not quite sure how to use it or whether it will take off anytime soon. Maybe cargo has some of its interesting features.
I am a fan of building a traditional native package in a multi-step Docker build, and the final container artifact can be a simple `RUN deb -i my-pkg.deb`
I find that targeting traditional system packages has benefits. 1) it's not really that hard, and 2) it forces you to lay things out consitently, or, at the very least, the distros conventions are helpful.
Docker (or container images in general) are great but they solve a limited set of problems well and tend to hide others.
I like container images and runtimes. Docker is awful, and its "inside-out" approach encourages and enforces bad practices.
Docker "solves" package management in the same way Bash scripts "solve" package management: you can use them to run actual package managers; but also, you probably shouldn't (e.g. Nix is better at creating Docker images than Docker is, for example).
oof