Truly immutable builds
blog.frankel.ch
blog.frankel.ch
Unfortunately its quite difficult to get a fully reproducible build with Maven as the post points out. Even if you manage to replicate the build environment and settings, Maven produced binaries will be slightly different on each run. File timestamps are saved in jar files so they change on every build.
Several build systems stress this aspect, include Google's Bazel, in beta.
A Vagrantfile or a Dockerfile are not enough. You need the actual images.
It's versions all the way down
Not sure I am a fan of having every git repo having a copy of everything over and over and over, for "ecological" reasons... But it's probably the best way to endure that your builds will work.
So now we don’t have to worry about them disappearing AND we don’t have to leave the network to get them... assuming we don’t screw up our server.
What we don't do yet is storing a pristine a VirtualBox image for another project. We should start doing that because sometimes building those images requires a surprising amount of work. Bitrot, etc.
You then get to decide to what extent you want to freeze your exact dependencies or leave them abstract to some degree.
I tend not to but generally consider myself on the fence about that one.
This way, I can override things if needed; and in particular I have continuous integration (Hydra) building both a "stable" (all fixed versions) and "unstable" (all latest versions).
This way I always have a known-good build I can use, but I can still spot impending breakages and bit-rot in CI.
In the case of nixpkgs I don't actually track the very latest commit, since it updates so often. Instead I just `git pull` into a fork every now and again.
They were so hard core about immutability: they had an old code base (Z800? I can no longer remember) that they didn't want to migrate away from even though the hardware was no longer available, so they build an emulator and ran the old code on that on their newer, PPC-based hardware. Newer code ran natively on the PPC).
Their concern: they had SLAs with their customers (not SAAS customers, customers who bought the hardware) of no more than two minutes of downtime per decade.
Never heard of DSC? Yes, famously their hardware brought part of AT&T's network down for 10 hours or so in the late 90s or early 00s. That was the end of DSC.
Another example is someone depends on a library, then that library's author deletes it.
This is why it may be worth committing your dependency's sources, despite "best practices" dictating otherwise. When someone needs to add a feature 10yrs from now and they only have an incomplete package.json & a copy of minified code, they're in for a world of hell.
I even had to recover a project which built non deterministically. Sometimes it would build, sometimes it would error. Running the build several times, it would eventually succeed. The project's author had written their build system against an unstable version of Webpack & Typescript, before they were popular (both of which are now perfectly stable).
Maybe I'm not a hip young web developer but I've always considered "best practice" to be keeping the currently-in-use version of all of your product's dependencies in a repo (either your main one or a separate 'dependencies' repo). If you update the dependency, you test against the new version and check it in if successful.
The idea of letting your build process depend on a third-party library that's downloaded ad-hoc (and may or may not be trustworthy) is terrifying.
I think you may have misunderstood the best practice here. That best practice is you shouldn't just copy the dep code into your repo. The maintenance/release best practice however is that you should always have a copy of all your dependencies available locally and permanently store them. They're two completely separate rules.
The reason stuffing the dependencies with the normal source is bad is you have a huge repo after a while, where most of the size comes from old dep versions you're unlikely to ever use. (But have to clone anyway)
Ask your nearby Linux sysadmins about binary packages and source packages.
Let's take an example project with 500 sub-folders under node_modules, are you suggesting I bundle the whole project into an rpm including its dependencies? Because I already have server backups & such, which accomplishes the same thing. Before I spent time making an RPM to backup the project, I could just take container backups or something.
My actual issue is when you have only minified code, or compiled executable -- and your boss wants you to add new features, but you do not have the sources, only a package.json that references projects that have been deleted. Or worse, no package.json -- not much can be done at this point, the onus was on the original author.
Or are you suggesting each dependency gets wrapped into an rpm? Because that sounds time consuming & doesn't seem like it provides any benefit [it'd be just swapping npm for rpm, it sounds like?]. Plus in my parent comment, the example I used, I inherited a project where 10% of the node_modules were simply missing, the project's author built the code, then deleted his node_modules & left the company. I had to derive what the dependencies were by weeding through minified code. Not sure how RPMs help. I'm also not sure how using RPMs helps when you have developers who may be on different distros or mac, now they have to run docker or vagrant to get the frontend deps? ew.
But perhaps I'm not understanding what you're suggesting to do with RPMs, you're not communicating very clearly or nicely IMO. Perhaps linking to an example repo employing your technique or an article that explains it in more detail would be a better way to get your point across. AFAIK, RPM & NPM were made to solve totally different problems. Trying to use NPM to manage linux packages or using RPM to manage javascript packages honestly sounds like a nightmare, going against the grain. RPM was made for linux packages, not frontend web development. I'm trying to keep an open mind but you're not explaining yourself.
Yes, basically. By the same way of thinking npm is essentially a variant of wget and shell scripts interpreter for running build rules. It's the details of the implementation and what use techniques are most obvious and convenient that make the tool.
> Eg. how does each project declare which RPMs? How are these RPMs getting created, by hand and one per package?
For daemons, command line tools, and libraries -- usually one package per repository (sometimes several packages built from a single build rules set, if it makes sense to install part of the project as a standalone thing).
Websites, since it's a custom to put a shitload of mostly trivial dependencies that form a fractal, get two packages: the actual code and all dependencies bundled together.
Note that all of the packages are not built from source code (and `npm install' or `pip install') directly, but they go through a source package (e.g. SRPM), which is especially important for dependencies. This way you always have a way to rebuild the dependencies in exactly the same versions (e.g. after patching one of them), you have an obvious place to store the dependencies (i.e. along with binary packages; package repositories support serving both) without growing your source repository unnecessarily, and you have an obvious place to apply patches to external dependencies.
The only downside is that you'll need to write a downloader script, because language-specific module managers (pip, npm, gem) usually assume their user is too stupid to run as two commands two logically separate steps of fetching and building the packages.
> I've literally never heard of anyone using rpms for nodeJS dependencies, and I am intrigued.
Well, yes, that's because Node.JS crowd doesn't have the habit of learning what other IT fields were using successfully for decades.
> Or are you suggesting each dependency gets wrapped into an rpm? Because that sounds time consuming & doesn't seem like it provides any benefit [it'd be just swapping npm for rpm, it sounds like?].
If the repository was handled the same way as the rest of the code, then yes, there's little benefit. If it was handled the way as the rest of the binary packages are in the distribution, you get for each binary package a source package and uniform build process, so you can rebuild each package every time you need to, and there is good tooling for automating building the packages.
> AFAIK, RPM & NPM were made to solve totally different problems.
One was created to make installing and deploying code easy, while the other was created to make installing and deploying code easy. Yes, totally different problems.
> Trying to use NPM to manage linux packages or using RPM to manage javascript packages honestly sounds like a nightmare, going against the grain.
Actually only the former of these is going against the grain. The latter, on the other hand, requires learning tooling that hasn't substantially changed in over a decade and comes from different field, so it's unsexy and uninteresting for most of the programmers.
> RPM was made for linux packages, not frontend web development.
And how exactly differs deploying some code to a directory and running some daemons between software that comes from a distribution and software that you have written yourself for running a website?
I agree that rpm does a better job at installing linux packages than npm does at installing javascript packages, but hey, I didn't write the tools. Nor do I want to write my own package manager... As a JS dev I want something that works out of the box, not something I have to write custom scripts for. Maybe someone will make it usable for the masses & npm will be a thing of the past someday, I look forward to that day.
I think one of the biggest differences that affects reliability is npm uses github repos, which each author can change, or delete without moderation. AFAIK redhat moderates their repository & does extra leg work to make sure things work together. Its probably also easier to keep things stable when you only target 1 platform.
The thing is, it's not novel by any stretch. Sysadmins use this method for almost twenty years already. About the only novel part (i.e. I didn't read it anywhere but thought up myself) is the binary package that contains all the modules required by web application, and even this I don't believe I was the first person to do.
And it's not untenable. It's not a method for distributing code to other programmers, it's a deployment method. Everybody can use it on their own method, without looking at Node.JS community.
> It also doesn't account for different team members running different OSs.
It's a deployment method. Everybody in the team is free to make as big mess with any other tool as they like, as long as the end thing builds properly for production system, and you can easily provide everybody a virtual machine for checking if build process still works.
> npm/yarn are platform independent, rpm is not.
Not quite. RPM does run on different platforms (OpenPKG proved practicality of this long time ago), and even then, you don't want a build from macOS or from desktop Fedora running on Debian in the production. Binary packages show up in the picture not in the process of writing the code, but in deployment.
> like I said, npm & rpm solve totally different problems. One is made for interactively installing linux binary packages on a specific distro, and requires config management like Ansible to orchestrate in an automated fashion.
Not quite, for both of the claims. DPKG and RPM started as such, but they are designed to allow to anchor the installation root in an arbitrary directory. And managing configuration? That's not DPKG or RPM thing, and you get exactly the same with any other deployment method (unless you do the brain-dead thing and ship the configuration hardcoded in the deployed artifact).
> Whereas npm/yarn is made for platform independence & copying javascript files into a directory, not system binaries, and has a manifest which makes automated builds easier.
This doesn't differ even a bit from how DPKG and RPM operate. The only difference is that npm and Yarn are written in JavaScript, while DPKG and RPM are in C/C++.
> I think one of the biggest differences that affects reliability is npm uses github repos, which each author can change, or delete without moderation.
And this is where building binary packages from source packages comes into the picture. Debian, Ubuntu, Fedora, or CentOS can rebuild their whole repositories even without the original source tarballs available in their official place (it could have changed, it could have expired, project could have been shut down, or for any number of other reasons). Full source is kept in the very same package repository as binary packages, along with build scripts, patches, and all the other things necessary for build.
> AFAIK redhat moderates their repository & does extra leg work to make sure things work together.
Curation of the upstream OS has nothing to do with using your own RPM or DEB repository for deploying code to your own infrastructure.
> Its probably also easier to keep things stable when you only target 1 platform.
Oh, but for production you are targeting just one platform. I'm yet to hear about anybody who deploys their own software to servers running more than one OS flavour. Usually production for in-house developed web software is quite homogenic, all runs under Ubuntu or all runs under Debian or all runs under CentOS.