We repurposed NPM to publish and distribute Go binaries for our internal CLI | Hacker News Reader

We repurposed NPM to publish and distribute Go binaries for our internal CLI

medium.com

62 pointsboxbag59 comments

At CERN we have an internal system to distribute software. It is mostly used to distribute software to production systems in the datacenters or to distribute nightlies and toolchains to developers. It replace standard package managers on most production systems.

It is a FUSE filesystem, where we can install software once and have it distributed to a practically unlimited number of clients thanks to a simple and performant cache system.

The filesystem is called CernVM-FileSystem, CVMFS for short.[1]

I maintain a public instance of it under https://packages.redbeardlab.com

It contains "common" software, like standard unix tools, compilers, webservers, etc...

I wrote a bit about it:

https://redbeardlab.com/2020/12/20/announcing-packages-redbe...

https://redbeardlab.com/2020/12/21/bootstrapping-packages-re...

https://redbeardlab.com/2020/12/21/packages-redbeardlab-com-...

https://redbeardlab.com/2020/12/23/packages-redbeardlab-com-...

[1]: https://cernvm.cern.ch/fs/

One of the best things about CVMFS is the idea of LCG views which are coherent installations of a large number of packages. This more or less eliminates the need for package management and guarantees that someone else can reproduce your environment.

It's like Stackage LTS releases, but everything's installed already.

Unfortunately I am not familiar enough with LCG and LCGmake to actually build an outside-CERN release.

Fortunately Nix helped in creating the same coherent installation of packages.

Then `source /cvmfs/packages.redbeardbal.com/setup.sh` give you the same feel of an LCG view, but with more widespread packages.

I was very tempted to reuse a Maven repository for publishing the handful of small Python utilities we had. It's a standardised format, we already had the server set up, we already had different kinds of artifacts in there and multiple languages, and frankly Python-on-Maven dependency management can't be worse than what standard Python dependency management looks like.

I kinda do that today. We’re primarily a Java shop, but we’re using Python for FaaS. So we package up our code and dependencies into a zip and use Maven to push the resulting archive into our common repo.

I wish commentors would read the article.

Why didn't they use the os package manager? Which OS? Their build clearly builds for osx, windows, and Linux.

NPM could handle different OS, I would guess they run more than 1 os. (It is also a tool which the team is used to handle, as they have been using node + npm)

Or since it is an engineering CLI tool, why not have engineers:

1. clone cli repo 2. run goreleaser 3. copy binary to desired path

sure, it's not 1 step (can be condensed with a build script), but it does teach the other engineers how to roll their own cli tool and contribute back if they fix a bug in it.

Or if the blog indicated that not all engineers understand golang and build tools, and there's too many teams - that'd be good for blog context. Hopefully the author didn't spend much time in figuring out or being too clever in making this "hack".

As the number of tools grow and the number of teams using/developing them grow this becomes more and more untenable. You not only have to maintain the tool itself, but also the build stacks associated with them and it becomes the task of the user to make sure they are running the most recent version. This leads to more complex support situations.

Basically, you need a deployment methodology that allows for: * Ease of use by "novice" developers * Only introduces minimal overhead for those developers * Makes it easy to upgrade or remove a tool you installed 6 months ago * Doesn't require a tool development team to come up with their own solution

This is a surprisingly tricky to do on a diverse environment with 1000+ internal users.

This just saddens me deeply. Of all the ways this could have been done and all the package managers they could have used they picked npm.

I bet some day we'll have a Linux distro with npm as the system package manager here on HN.

It saddens you that someone used a tool they already use instead of adding another thing to distribute a small binary?

I guess my comment appears rude, which wasn't intended, I just find npm a troublesome piece of software that is passable as a package manager for the javascript ecosystem, but not a tool one solves problems with if there are viable alternatives.

When you have one thing to nail and already own a hammer, why go and buy another one? Even if it's not a good one, it's gonna do the job, and be the least trouble.

The choice of package manage is much stickier than the choice of hammer. I can trivially change hammers between projects or even between nails. Changing package managers is a lot more involved, and gets more involved as it's used more---and they've just established a precedent for using npm to package go binaries.

Meh, depends. For their case, for an internal CLI tool that will get used interactively on development environments, it's fine, the cost of switching is not very heavy. At worst they're postponing the work of deploying on other package managers, and devs will have to run 2-3 commands to remove the npm package, configure whatever next place it gets deployed to and install the new executable.

Not a big deal at all.

Sure, should have s/much stickier/sticker/. I'm not going to comment on whether the tradeoffs are worth it, I just thought the hammer analogy doesn't work due to material differences in how you switch between hammers vs package managers.

They established the choice of npm when they choose to be a JS shop, that's exactly their argument.

Sorry I wasn't clear, I'm not making an argument for or against npm, only responding to the analogy.

But I wouldn't consider "we're already using X" to be sufficient reason to use X for something it wasn't designed around. Again, these choices are sticky and tend to be vulnerable to scope creep. I'd call it a textbook case of technical debt, and it's ok to point that out. They might have forestalled the criticism by acknowledging that, but they didn't.

I don't know why but I never encountered issues using npm.

I've had far more problems with yarn, which sometimes decides it and it alone not longer has internet access and none of the "fixes" published over the years have worked. Instead, you have to just set your timeout really, really high. That's been the only fix for a good while, and it frequently does not work.

Older NPM could be a pain but modern NPM is pretty smooth and works just fine.

why is it trouble? why can't they improve it even if its trouble?

deeply

I thought the same thing (Go binaries are so easy to distribute), but they do explain that their devs already have credentials for an internal NPM server, and they wanted to do this "without the need of distributing a new set of tokens or making the binary public". I don't think the trade-off is worth it, but at least it explains their motivation.

In their particular case, what's the trade-off that's so terrible though? They already all have npm, they had a binary to distribute, and they distributed it.

https://node-os.com/

Dang, I wish I could say I'm surprised, but I guess it was to be expected.

Your scientists were so preoccupied with whether or not they could, they didn’t stop to think if they should.

There is obviously a need here that is not met by current tools.

I believe that Conda is the closest thing to an answer to this need.

It works on Windows, Mac and Linux. It supports packaging arbitrary binaries. It lets you specify dependencies that can be arbitrary libraries for arbitrary languages. And it has a relatively simple package format, build process, and repository format.

It's not the best piece of software when it comes to user experience, and its niche is still very much data science. But it is a glimpse of what should be possible, but is not currently available.

If all you have is a hammer, everything looks like a nail. [0]

[0] Abraham H. Maslow (1966). The Psychology of Science.

I need a medium account to read this for some reason.

> "This is your last free member-only story this month. Sign up for Medium and get an extra one"

Well, that's about the end of medium I think.

Not logging in gives you a better experience than being logged in, especially when you can use incognito mode.

Only website that punishes being logged in.

I wonder how “publishers” feel about that. I would feel rather angry if Medium dictated that people couldn’t access my blog posts.

But that's exactly what to expect when people donate their writings to a platform they have no control over. There are always those who need to learn the hard way.

Turn off Javascript. They don't (yet) have an SPA and it degrades appropriately for disabled users.

I'm not going to expend any effort to circumvent medium's nonsense. The content there isn't of consistently high enough quality to justify jumping through any hoops. Any barriers they put up will just result in me not consuming content hosted there.

Open a private tab (or in Chrome "incognito") and it will work fine.

here's the summary: it opens with "We recently migrated our internal CLI tool from Node JS to Golang". It's just a way to keep using the hammer they already had, no matter if the problem is a nail or not.

could be titled "how to hack your private npm server to be a OS package manager, because we don't want to learn our OS packaging"

Which raises the question: package management is just a bunch of nails; why do I need a specific hammer for each type of nail?

> why do I need a specific hammer

i call it "lazy opinionated adoption".

Why do you use apt in debian and yum on redhat?

Same reason as you write javascript for the browser. Just because someone put it there at some point. Would it be better to use python or anything else in the browser? sure! why don't you do it? because many people have done it before and there's no adoption!

package manager and languages are just like curl or wget. try one, command not found? try the other. meh. ...until you find people that assume other people can't learn. Then they will create a third option because other-people-are-dumb and it will be worse than both options you had before.

Because there are great tools like artifactory that can handle all the arrifact types.

For future reference: block JS or open in incognito to bypass medium popups.

>We recently migrated our internal CLI tool from Node JS to Golang

Why? Out of things to do? Wanted to look productive?

Go supports native packages natively via go get, why the hell bring npm into the picture?!?

> As the project completed we ran into the issue of distributing the CLI tool to our team of engineers without the need of distributing a new set of tokens or making the binary public.

From the article itself, it seems like they wanted to re-use their existing process without disruptions.

You'd use go get as a developer, not to update workstations and production machines.

Same applies to using npm on workstations and production machines.

What, did curl not work?

"we ran into the issue of distributing the CLI tool to our team of engineers without the need of distributing a new set of tokens or making the binary public"

Seems a neat enough hack to make their newer shit available internally using existing tools/security.

I haven't read the article yet, but I would assume they want versioning, history, checksumming and verification. All that stuff is worthwhile, it's just unfortunate npm is such a heavy dependency to achieve it.

Debian apt gives all these things and would be less of a heavy dependency.

The problem with .deb and .rpm is that their spec file format is horrendous. I've packaged multiple RPMs and every time I have to do it is a complete PITA, lots of quirks, macros, flags, etc.

The official docs are like a deep man page and do not have simple use cases, like, create a pkg for a binary

or create a pkg for a binary with systemd

or how to distribute a pkg of .so

IMO if the .deb and .rpm maintainers invested some time in modernizing their official pages with lots of quick starts and guides it would bring a lot of developers to their side.

(And in this specific case they needed more than Linux support)

If they only needed linux support, that would have worked too

...for a single OS that none of the devs might be using.

Not that heavy if they're already using it though

It also only gets really bad with deep dependency trees. If it's just your stuff, you probably avoid most of the issues.

Next: Docker images!

Given NPM registries are presumably optimized to store text, I wonder how this scales and if other repositories that do expect binaries (eg. Maven, NuGet) compare.

NPM packages are uploaded / stored as `.tar.gz` files, so they can contain anything (alongside a package.json metadata file) - some public NPM packages contain binary files like images or compiled assets too. The registry doesn't really care about much beyond the package.json.

I've done something similar to the author at a previous company, distributing an internal development CLI written in Go, though in our case we published to the GitHub Packages NPM registry rather than private npmjs.com, so we could authenticate our developers via their GitHub accounts.

Including multiple binaries inside the published package, one for macOS and one for Linux (we had no Windows devs), did increase the package size, but the whole gzipped .tar.gz was around 20-30mb, so not terrible from a registry storage standpoint.

Publishing to an NPM registry, vs just hosting a binary for our devs to download, was convenient since we could use the registry API to check for updates and notify the user that an update was available. Users could then update via a subcommand, which just pulled the new package from the registry.

Reply on news.ycombinator.com