Ask HN: Should you check in your dependencies in source control? | Hacker News Reader

Ask HN: Should you check in your dependencies in source control?

12 pointsnasso8 comments

Almost every piece of documentation I can find recommends not checking in your dependencies in your repository. This is the way I'm used to doing it myself too. I haven't really reflected on why.

I can see some benefits to checking in your dependencies, for non shared private projects. You will be 100% free from external dependencies in your build process. Builds will be faster, easier and more robust. There is a guarantee that the project will be buildable in the future.

Imagine that you have a small service/application that is running internally on a company infrastructure. You don’t touch it for years, it just keeps running. Five years after the application was built the server breaks and you need to redeploy the application to a new server. There is a risk that some dependencies have changed. Maybe someone raged and removed a GitHub repository, maybe GitHub made some changes or doesn't exist anymore or is just down. There are a bunch of things that could become issues. Checking in your dependencies solves all that.

The negative effect is that your git repository grows in size. You can have separate commits for the updated dependencies but it will still be in the history.

Is that not a small price to pay to be free from external dependencies though?

Thoughts?

You should ensure that they are somewhere under your organization's direct control and you can do a build without going out to the Internet. It's incredibly dangerous to have your build process depend upon random code pulled from the Internet, as left-pad showed.

Whether that means checking it into your VCS depends on which VCS you use. In general, Git does not handle large binaries well. Check out git-annex though, or use a package manager that lets you setup a local mirror.

For those curious, yarn supports a local offline mirror:

https://yarnpkg.com/blog/2016/11/24/offline-mirror/

Yes since dependencies should be immutable given a version number, a backed up file share is suffcient for storage. Nuget allows this. I'm sure maven is similar.

I do both.

On my more serious projects, I check in dependencies. My mantra here is "after a git clone and with the compiler, it must build without internet connection".

The size of my repos ranges from 5 MiB to 1800 MiB (though the later also stores a shitload of archival data for reasons I won't detail), the later one is still very manageable.

On my more "fun" repos I don't commit deps since I think them less important until it turns serious.

I sidestep this issue on my personal projects by using Docker w/ Python (although this is probably applicable to other languages). I use this at work in production as well although to a lesser extent as we aren't containerized for most services.

I create a base image that has all of my deps installed in a virtual env. Use that base image as the `FROM` clause in the next image. Additionally if you were to rebuild this image and a dependency went missing or broke it's trivial to go back to using a previous working image.

So, I guess technically they are checked in since a copy of the system and thus the libraries/binaries on it is stored in a Docker repo but those are separate from your code and well abstracted away. You can also easily run your own PyPi server on your network to avoid going to the web at all (this is probably also possible with other languages). You can also host your own docker registry so you can totally remove dependence on external versions of your deps if you want to.

The problem comes when packages have architecture-dependent build steps at the time of running, e.g. `npm install`. For example many packages depend on binaries that are compiled at build time. If you check dependencies into version control, then you’re checking in scripts that depend on binaries compiled for the architecture of your machine. This can easily cause problems when deploying to a server.

Here‘s what I do for a native app:

Small dependencies are checked in the repository in source form, and compiled with the app.

For big dependencies (eg. OpenSSL), I have a script that downloads and builds them, and then I check in the built object files into version control.

This means that as soon as you check out the repo, you can immediately build my app, and build times are really short since big dependencies are already compiled.

The downside is that I need to update dependencies manually, which is one of the reasons why I keep dependencies to an absolute minimum.

If the dependency is small, not a binary blob (e.g. pure Python) and the distribution license is compatible, I think directly embedding for one less dependency is wise. Just label it clearly and separate it, e.g. isolated in an “abc-1.2” subdirectory that isn’t mixed with other things.

I’d draw the line at large libraries or binary blobs that are highly likely to not work out of the box for somebody.

Reply on news.ycombinator.com