Hacker News Reader

Comment by 0x0203 | Hacker News Reader

I'm not going to say where and I'm not asking for interested parties (we're not currently hiring anyway), but where I work (as a kernel/OS developer), package maintenance and management has been an ever-increasing burden to the point that we've tried hiring somebody specifically for the role. Unfortunately, it seems like this is a very specialized skill set that the vast majority of developers seem to find boring, uninteresting, and drudgerous, and finding someone qualified and interested has been exceedingly difficult. How would someone/a company go about finding people that are actually interested in this kind of work? There are clearly people who take on this task (some may even enjoy it!), and many who even do it for free for various open source projects, but I've no clue how to recruit or connect with any that would be willing to do this for a job. Any tips for finding or connecting with such folks would be much appreciated.

Attending in person won't be possible, but I'll have to keep an eye on this conference and see if they provide a way to post/share such opportunities.

View on HN

I think you might have more success advertising to sysadmins or devops engineer rather than developers. It's precisely the negative attitude, or at least apathy, towards packaging that makes it difficult to find employment where packaging is valued. On the other hand, a lot of the packaging related work has traditionally been the remit of the UNIX/Linux system administrator, and that as a profession is slowly moving towards automation as 'devops' - but it's still fundamentally the same concepts.

Also, you could sprinkle in some 'optional but desired' skills and experience relating to packaging into your advertisements, anything from CMake and GNU Autotools to Nix and Podman. That would be enough to pique a packager's interest and differentiate it from the 'write-only' coding jobs!

I will second this advice; both of the engineers I have worked with who actively enjoyed being tasked with packaging software identified as systems operations / devops / site reliability engineering.

These are both very useful comments; thank you! In your experience, do the devops/sys ops engineers typically have useful coding experience? In our environment, porting a package to our OS may require a little, or sometimes a fair bit of modification to make it work and we often need to be able to read/understand the upstream code in whatever language it's in and change/add to it. Hence our (or at least my) assumption that we'd need a "developer."

A good sysadmin may not be able to do graphics programming, but they have plenty of practice in the sort of programming used in package management, as well as a fairly good understanding of operating systems.

They’re not going to grasp everything as well as somebody with a development background, like learning NEW programming languages on the fly and upstreaming changes, but I don’t think it’s totally outside of what they can learn and hiring a developer would present just as many challenges.

I might look more for somebody who bills themselves as “devops” or “sre” than sysadmin though since those job descriptions tend to effectively filter out those who can’t program at all. If anything I see those people as being software developers that specialize in systems administration.

In my experience a lot of people who started as developers 20+ years ago move on to system administration, and retitled themselves SRE/Devops over recent years.

I don't want to talk about ageism, or hiring biases, but I'd imagine if you found anyone who had an SRE-role and a long history they would be extremely likely to have had serious coding experience.

I've run into a few "young" devops engineers, and they tend to be more familiar with k8s, CI, and similar, rather than actually developing. But I guess that could well come down to the random people I've met, I wouldn't like to generalize.

> In my experience a lot of people who started as developers 20+ years ago move on to system administration, and retitled themselves SRE/Devops over recent years.

Usually they don't retitle themselves, companies do.

It seems the whole IT industry just collectively decided to rename the sysadmin role as devops when using cloud and containers technologies regardless if they actually use DEVOPS methodology or not.

That is my history as an engineer at least.

Agreed. I'm an ex-developer, turned sysadmin.

But these days my titles are Devops-This, and SRE-That. I find it hard to take too seriously, but I guess it is what it is.

In my experience creating packages for an operating system package manager or Docker images, a DevOps or sysadmin mindset is more beneficial than a developer mindset. For those packages, the goal is to follow operating system or deployment best practices.

I’ve not created a Python or Rust package or something like that. If you’re creating that type of package, my guess is that it may help a lot to hire a developer who loves to think about the most “idiomatic” way to package your code.

Edit: if your package will be published and used by customers, then you might look for DevRel skill as a bonus.

> In our environment, porting a package to our OS may require a little, or sometimes a fair bit of modification to make it work and we often need to be able to read/understand the upstream code in whatever language it's in and change/add to it.

What one needs to understand will vary so much depending on the upstream package and how it's normally built (what it expects) that I think even someone with experience in packaging will sometimes get stuck or need to seek help or advice. The more substantive the porting work is, the more that will be true.

That said, this kind of thing is pretty common for Linux distro packagers, and the weirder the Linux distro is, the more true that is. It's not at all unusual for Nix folks to add special build params, patch upstream software, or mangle binaries with sed in a post-build step to get them to understand NixOS' unusual filesystem structure, or to build offline in the restricted sandbox.

So I'd reach out to contributors to BSD ports systems, especially if they added ports for Linux-centric software, and contributors to really weird Linux distros (NixOS, GuixSD, GoboLinux, Void Linux, etc.).

I've never been in charge of hiring decisions anywhere so idk what the overall landscape of DevOps job candidates looks like. But I've done DevOps and I definitely do, and my friends in roles associated with that title can definitely do at least some coding and scripting in a way that I think suits most packaging tasks well. If a DevOps person has a CS degree of previously worked as a developer, they can probably figure out what they need to when it comes to patching upstream software to get it to build. Even if they don't, if they do some recreational programming, they can probably push through.

Some do, maybe more with SRE title's. Better to put that in ajob description and filter for those with development backgrounds.

As a sysadmin I used to do package management for different OSes but this is not necessarily the most rewarding job. Atmo, most engineers doing that are juniors or started doing that as junior and never cared exploring anything else / moving company.

I not 100% convinced that profile works either. I tried multiple times to hire for my team which works with a combination of library maintenances build tool development and package management. I even tried a junior developer with an open mind. The skills needed for this job is sadly to broad. One needs to know the platform/language and tools to package software. And don‘t get me started with CI etc. Sysadmins and what most understand of DevOps are not the right kinda people. Not to say there are not people among this group.

The most appropriate group of folks may be those that already maintain software for the OSS community. The skillset more resembles trades where time and experience matters substantially more than any intrinsic talent or overall intellect to be effective at the job.

Honestly, sometimes I wonder if this is perhaps an under-considered area for Generative AI applications given how much drudgery is involved yet precision tends to matter.

Maybe… I mean yes at the bottom most package system are a zip with some kind of manifest or a manifest which describes where and how the lib can be build from which sources. If an AI could deal with the fine details of these systems… I mean Debian packages are harder than Arch packages, or at least that is my experience. There packages which are super easy to work with like native python, ruby and Java. But mix it with c extensions and boom. The CI can get complicated.

If there's a lot of thankless, mundane work for the majority of projects that seems rather routine from what I've seen described thus far across multiple decades, so this seems quite ripe for some form of automation sans projects that seem to keep requiring changes constantly. C extensions, interop across languages, and ABI considerations are certainly much more complicated and require a lot of complicated work but given that 90% of the issues even power users run into are things like "the lxml library I installed won't get detected by package X when I run autoconf, please help."

There's nothing sustainable as an industry about something like software dependency management and versioning being so critical for things "just working" while people are sticking their heads in the sand saying "not my problem" as a rule. I'm so tired of solving the same problems that were already solved in the past because of developers wanting to work on "more interesting things" and doing the bare minimum to get stuff compiling and running again regardless of the package format. In fact, Docker may have made this situation even worse with so many Dockerfiles that will basically be unable to work. This is part of why I'm interested in the Nix ecosystem because there's so much work being done to make software installations repeatable and easier to manage. Unfortunately, it's squarely aimed at developers and is way, way too complex for most sysadmins to put the time into understanding, especially when they're so overwhelmed like everyone else with so many tasks.

I‘m 100% with you there. I also think that the problems described are solvable on some levels if not all. I remember the time when brew took of on macOS. I used macports at the time. The reason why people preferred brew was the fact that installs where way faster. The reason was simple: brew based everything on system libs where macports similar to nix maintained packages for all libraries etc. It took way longer to install a package with macports. I‘m running Nixos on my private machine (I need to work with macOS at work as the only other option would be windows) and it is sadly way too complicated. Nix flakes make this harder and easier at the same time since nix channels had a way of breaking as well. I switched to nixos because I had issues with running nix packagemanager on other distros (including macOS). It either behaved wrong when installing gui packages or in some cases with a rust package I maintain had issues with glib-c which nearly drove me nuts.

As to docker. Yes I also think it made stuff worse in the majority of cases. What people forget to grasp is that Linux be it a vm or a container is just a the kernel and that we have dependencies for some programs and libraries to the kernel. Reason why we have distros in the first place. So not all containers might run the same on each host because it docker runs on the same kernel as the host machine or the VM which installed (macOS) Issues are rare but can happen. Also the number of base images. „Let’s use alpine because that means I build a small container image“ … No it means you build on top of an alpine release with all the installed system libs etc. And that may or may not work with your software. And stuff like that.

> One needs to know the platform/language and tools to package software

How long is acceptable for your packagers to struggle with learning a build toolchain and its quirks? How many language ecosystems are you working with at your company?

> And don‘t get me started with CI etc.

Respectfully, I'd like to get you started on CI. I kinda feel like the landscape of CI platforms right now sucks, and a lot of my daily work is integrating with them. I'm curious what your gripes are!

I meant to not starting with the issues of CI but the complexity of it and how many solve it. There is the simple shell script idea which mostly runs on this one CI system and nowhere else. Or the we use a build system which abstracts all kinds of things away but need a good chunk of knowledge to understand why it’s there in the first place. Developers from camp shellscript with the desire to keep it easy ask: „why do this etc etc“. It’s not that their wrong it’s mainly the ignorance of other things like build reproducible builds and platform and system tools abstractions that make these more complex. To just expect that some developer who never dealed with a multi arch/platform product to just willingly jump into the complexity is unfair.

I also don’t like the current CI landscape. I still run a Jenkins at work cause it gives me the most freedom and I know it best (it’s sadly what keeps me off other systems, at least I know what parts not to touch). We use gradle which I really liked 5 years ago. We also use it for other non Java systems since it is pretty adoptable. But the recent version changes bring more and more weird changes which make it harder and harder to maintain custom plugins for it. The rules to code build plugins are more stricter enforced by the compiler which yes is good for gradle but hard to maintain as one needs to read up and understand why one can no longer access a file input outsite of a build execution phase for example even though it worked fine for the last 4 versions. And then there is more.

> Unfortunately, it seems like this is a very specialized skill set that the vast majority of developers seem to find boring, uninteresting, and drudgerous, and finding someone qualified and interested has been exceedingly difficult.

The reason most find it boring, uninteresting, and drudgerous (I would also add "frustrating" to this list) is because one has to build on top of decades of bad decisions and quick hacks, over and over again. Which also suggests a way to make it interesting: make it about fixing the underlying problem (even if in a limited context) instead of just going along with whatever is available. For example, you can attempt to automate producing various packages from some "sane" common metadata.

You didn't provide much context on what exactly you are trying to package, but here is one example of what it might look like: https://build2.org/bpkg/doc/bpkg-pkg-bindist.xhtml

Or post it as a job to use AI to automate packaging.

I'd be interested in the other direction: I accidentally stumbled into a position where I get to do packaging work, which is one of my favorite parts of the job. But like... that was an accident. As much as I would love to do more work in the same vein, I have no clue how I would intentionally seek it out if I were job hunting.

The large open source companies do a lot of packaging, probably some of the smaller open source consultancies too, maybe try some of the employers listed here:

https://github.com/fossjobs/fossjobs/wiki/resources

To corroborate my claims from another comment, I'm calling you out as a recognizable Nix fan on HN. ;)

I told you us Nix weirdos enjoy packaging, GP!

Lol:) I mean, I was into packaging before I started using nix... it just happens to be an often nice option:P

I think that's how it usually goes! That's how it was for me, too, at any rate. :D

My packaging experience is limited, but I found the docs were quite poor. It took perseverance & researching install error messages to create an RPM that worked. So you might consider hiring for people willing to do that kind of work.

The good news is that I learned a tremendous amount about where programs and files should actually be installed through packaging. It was surprising to me how many conventions package managers enforce. I was also surprised how poorly software followed those conventions when installed without package managers. Packaging improves user experience, developer experience, and quality. Can’t recommend it enough!

> It took perseverance & researching install error messages to create an RPM that worked. So you might consider hiring for people willing to do that kind of work.

At its toughest, packaging is a long series of exchanges: one build error for another. It is a stubborn, patient, rabbithole diving, well-I'll-be-damned, yak-be-shorn kind of work.

> The good news is that I learned a tremendous amount[.] [...] Can’t recommend it enough!

And it's totally worth it, both for the packager and their users. :D

At my last employer, we were called Systems Engineers. I created tooling to switch us off of a custom packaging solution to using Debian packages. The other part of my project was tooling around Ansible for configuration management across a fleet of over 48,000 vms deployed at over 6,000 locations, with different locations requiring slightly different configurations.

I also helped build a web based system to track approvals and schedules for all the changes.

I think Dev Ops or Systems Engineers would be the most used title for this type of work.

Go to where the packagers are; the open source distros and related companies. Some of the distros like Debian have lists of consultants, or a jobs list:

https://www.debian.org/consultants/ https://lists.debian.org/debian-jobs/

> package maintenance and management has been an ever-increasing burden to the point that we've tried hiring somebody specifically for the role

I do packaging work at my current job, and it's one of my favorite tasks there. I currently maintain a small collection of private Nix packages for macOS and Linux, and last week spun up a private Homebrew tap (for Casks only!) on a lark. I have some experience packaging things in other formats, too, and I'd be happy to pick up a new one if I had a use case.

But it has literally never occurred to me to look for a packaging job specifically! I didn't know roles dedicated to packaging were available anywhere.

> How would someone/a company go about finding people that are actually interested in this kind of work?

Even when I'm happy at work, I'm generally interested in keeping an eye on jobs where Nix skills are desired. 'Nix' itself is hard to search for (because it's ambiguous with *nix as an abbreviation for Unix-likes), so I also search for 'NixOS' and 'Nixpkgs' on job search sites sometimes. I don't think I've ever once seen a result for 'Nixpkgs', so any listing that mentioned that would stand out, and a job explicitly titled something like 'software packager', 'software distribution engineer', 'package management engineer', etc., would definitely catch my eye.

Job boards and development mailing lists for large package collections (Linux distros, BSD ports collections, and other collections like Conda) would also likely have some interested people! (In the case of the latter, just make sure that job posts are in line with the norms of the list.)

Communities surrounding advanced build systems (Bazel, Buck, Pants, etc.) would also probably be good places to look.

Find a sysadmin and point them at FPM: https://github.com/jordansissel/fpm

I'd not seen that project; definitely something I'll be looking at, thanks. Unfortunately, for what we're doing, it doesn't look like this will work as is... certainly worth some investigation though.

Good lord. I'll never understand why people promote this project, since it's main feature is that he's doing it wrong. The complexity he's trying to avoid exists for a reason, not because people like complexity.

OP: post here: https://lists.debian.org/debian-jobs/

It depends on what you want as a result. Do you want the shortest path to "dpkg -i the-result.deb" working on your server, or do you want a package complying with debian standards around behaviour and file locations, with packaging/source changelog split, potentially upstreamable? Because the second one is a lot more work, but few people actually need that.

The second one is more work and it's what you want. Because it gets stuff into the main archive that everybody benefits from in the future. Otherwise your project is a series of hacks that create an unmaintainable mess.

No, sometimes you really don't care about that. Some software will always stay internal. Some is so specific that it will never get to a distro. Some is just experimental and providing standards-compliant packaging for it is a waste of time. And that's when FPM can be useful if you want to go that way.

Yeah, FPM is great not for official/public packages (or libraries) but for "for some reason our $org installs internal (or patched, or just self-built OSS) stuff via package manager" and most often even all the versions, like /opt/foo-1.0, /opt/foo-1.1 and so on. It solves this problem very well.

I'd say it solves it passably, in the simplest and most minimalistic way possible to still call them packages, technically.

I question how one gets here, nay, the 'problem' we're solving.

Presumably there's config management involved already capable of shipping bits. ie: the repo definitions to find these hackjob packages, or deliver them directly.

If this is what you're willing to invest, go Slackware at it - use a tarball. You aren't gaining anything notable by throwing an archive at a packaging format.

The meta that FPM ignores is what provides packages their value! If changelogs were at least a part of it, I'd be a bit more accepting.

Otherwise, I see it mainly as misappropriation. Best case, naive and well served. Worst case, giving the impression of better distribution than actually exists

An organization that does packages, but leaves this as the answer, fails itself and the members. Over 90% of the purpose is dutifully ignored/not standardized

It is also a branding problem, imo. Part of the reason I prefer software from distro repositories is because those conventions the distro maintainers enforce help ensure that packages won't do stupid things and break my system.

Especially with DEB and RPM, where the packaging format supports arbitrary hooks that run as root, this is a big deal. High quality packages that meet distro standards will inspire confidence in customers' sysadmins. Substandard packaging may do disservice to your core software's brand, if your actual software is more solid and thoughtful than the packaging.

A thousand times this. As someone who has been fascinated with package management for a long time, my first reaction to FPM was horror. FPM is by and for people who resent package management and want to avoid actual packaging work, not people who are passionate about doing it well.

You want people who, when tasked with creating a package for a distribution format that is new to them, look not to tools like this, but to the conventions and standards of successful distros which use that format as examples.

You want 'native' engagement with those packaging formats, not hacks like FPM. The parent commenter's suggestion is an excellent one.

Please heed this - FPM is basically versioned tarballs. It's only useful for the most simple/contrived scenarios

Actual package specs require effort for good reason. It's not just an archive... but interdependence, steps to perform on (un)installation, changelogs, and so on.

There are some helpers already provided, ie: RPM macros.

Sure, they're esoteric, but show me a specialization that isn't. Refer to the Fedora packaging guidelines and enjoy life.

I love me some fpm, but I recently discovered it has a golang pseudo-replacement: https://github.com/goreleaser/nfpm#readme

It's for sure a ton harder to use (IMHO) because it mandates the creation of a manifest yaml, versus "fpm -s dir -t deb my-directory && echo tada" but not having to deal with ruby (or docker) can make it a better fit for several circumstances

It might be helpful to know that this niche is mostly organizing itself under the "release engineering" term. So explicitly putting up job offers for a release engineer might help.

Like you said there are many people do it for free for open source projects. Why not try to figure out the reason for them to do so and do so for free? With that reasoning, I believe you maybe can find out some changes to the "job" or the "company culture", or the "expectation from the management", or something else that could help you attract more candidates.

I've tried the basics of building a package manager before for fun, and found it a much harder problem than I anticipated (as most things! that's why I experiment by trying to build them, to learn! And sometimes actually build them successfully [1]). So I do agree with the sentiment of finding someone qualified to actually build it and not just a generic dev.

Depending on your needs I'd first try to not need the expertise in the first place (a generic public package manager is much more complex than say, a tightly controlled plugin system), or try to leverage existing package managers/tooks.

If that fails/not possible, and you cannot find a proper fulltime dev, many open source devs working on these topics might be up for consulting, so I'd reach out to them to try to set the general guideliness/direction and then use in-house talent to fill in the gaps.

[1] https://umbrellajs.com/ as a clone/alternative to jQuery

I find it's a mildly common skill among system administrators/SREs. I personally picked this up, and developed it, in these circles across many companies

Good ones are a bit rare... as this tends to go

They tend to write automation/utilities that help keep the lights on - packages are an important part in distributing that.

Literally go to nixpkgs/nixos areas and ask for who's looking for a job? Well, those folks are going to want to do Nix packaging, so... ymmv. Also, my other comment about Nix in this thread has gone through a rollercoaster so uh, that's sure interesting. Hard to not raise an eyebrow at.

I’m a maintainer at Arch Linux and have been in FOSS for long enough to know the landscape and who’s skilled looking for work. Send me an email with your needs and we can talk.

As others have said, what you’re hiring for is more of a specialised sysadmin role. I agree with leaving the software engineering label behind.

If it requires skill but it’s boring, pay them enough to care.

Some large tech companies dedicate multiple teams to package management and installation. It's a unique enough problem that you may not want developers to focus on it, better to roll it up to a dedicated team. They usually work closely with release operations.

> There are clearly people who take on this task (some may even enjoy it!), and many who even do it for free for various open source projects,

I feel like I missed something obvious here. Doesn't this answer your own question? I've worked for companies before that found great talent for very unique skills by contributing to similar projects and reaching out to frequent commit authors.

This attitude is exactly why hiring sucks. You and everyone else want to find your purple squirrel, your Python developer with more years of experience in Python than Guido Van Rossum.

Why is it so hard to accept that the best candidate you will ever get may be someone with relevant but not _exactly_ the experience that you want?

Having been on both sides of this equation, I don't think I'm being unreasonable, and am well aware of the fact that good candidates and the "right" hire will rarely be the ones that check off a list of random attributes plucked from the thin air between the pointy-haired bosses ears. I'm not looking for a purple squirrel, and I'm not looking for someone with 25 years experience in rust or the like. I'm looking for someone who both has an interest in, and a capacity to do a somewhat unique job we need done. We've found plenty of qualified people, and hired some, many of whom tend to be fresh graduates with zero non-academic experience. What I've had a difficult time finding, even figuring out how to look for, are the people that have the interest or desire to work on this particular type of niche problem. Few developers out there seem to, and we've found even less, so the package porting/maintenance/management part of our job continues to be a burden for the whole development team. Thus my question. And thankfully, some insightful replies suggest that I'm mainly looking in the wrong category of candidates. All of which has nothing to do with matching an overly strict list of experience requirements.

Given an upstream project with good practices, packaging is fairly easy these days, since most of it is automated. Most of the work is around reviewing upstream changes and fixing any issues found.

Ages ago, my QA/Test people reveled in managing releases.

I wouldn't know how to find those kind of people today.

Just something to think about when interviewing those sysadmin types (recommended by others).

if it's for some kind of internal systems or shippable hardware/appliances, best approach is to rethink approach to packaging

I can recommend a few people who are passionate about packaging. My email is in my profile.

Reply on news.ycombinator.com