What's the advantage over a build system that downloads the dependencies, but which gives you the ability to prefetch all the dependencies (so you can reliably do work without connectivity after prefetching)?
This separation allows to build the project off-line, which is not an uncommon scenario. There are environments that have the direct internet access prohibited e.g. by a company policy. There are environments that have it difficult (e.g. are behind a proxy). There are distribution package builders (RPM, DEB), which have a policy of only working on local sources.
And then there is build reproducibility. If external network is involved, the whole reproducibility idea goes out of the window. Remember the left-pad farce? Part of the cause was idiotic split to microdependencies, but part was that everybody used external network for their build process.
Fair enough, but if you have outdated dependency artifacts, it doesn't make sense to compile without fetching them, and so I think it's reasonable to make the 2nd step depend on the 1st one (and thus make it execute automatically)
I understand that keeping the 2 of them too-close might inadvertently conflate the 2 concerns (the build step becomes impossible to run without the download step) even if the design goals explictly thought of them as being able to be run independently, so this might be an argument for "builds should only build", but I'm not sold on it yet
> And then there is build reproducibility. If external network is involved, the whole reproducibility idea goes out of the window. Remember the left-pad farce? Part of the cause was idiotic split to microdependencies, but part was that everybody used external network for their build process.
Well, the build systems that I have in mind are Stack ( http://haskellstack.org/ ) and Nix ( http://nixos.org/nix/ )
Both of them have an heavy emphasis on reproducibility, and yet both of them automatically download dependencies.
(the Hydra Nix continuous integration systems OTOH run build and test steps with limited/no network connectivity, to enforce that separation of concerns)
(the microdependencies btw wasn't a problem that caused build non-reproducibility, microdependencies only made it worse when the shit hit the fan)
Also, there are 2 different reproducible build failures: it doesn't build (bad) or it builds but the artifact is not identical (worse)... downloading dependencies from the internet can only cause problems of the first type, if you have a sane way of fetching them
sane means: either you have the hashes of the dependencies to check, or you can trust the archive to always give you the same files when you ask it for a snapshot... and for security reasons you'd still check the crypto signature on the hashes to verify that nothing has been tampered in the mirror from which you downloaded
(Unfortunately, not only Npm but also Hackage afaik have that anti-feature that allows you to reupload a "bugfix" with the same version number... so we cannot always trust that the systems that we're using are sane, which OTOH is impossible also due to DMCA or other law/court shenanigans: if your dependency violates someone's copyright, it might end up disappearing from the snapshots)
I am sorry, I'd contest that. There are often cases where one might not want to use the most current version of some library (incompatibility, undesired features, etc.).
This entire forced update approach is already unpleasant enough in the context of applications (most notably on Android and IOS) but becomes unbearable for software development. While one might argue it could still make some sense for the former, as average users might stay forever with old versions and potential security issues, that argument should not be valid in a "professional" context - where the "users" know about the implications - as it should be given with software engineering.
I think we're misunderstanding each other.
What I meant is that if you have foo-1.2 in your dependencies.conf, and you have previously downloaded foo-1.1 it doesn't make sense to compile, because the result of the compilation will most likely not be what you want
You can just have the build say "foo-1.2 not found. Latest version is foo-1.1", and the developer can decide how to respond.
- "usual" case (if you agree): dependencies will be downloaded from the Internet, single command download+build is useful
- in a bank/SC environment: the developer configured a proxy/local mirror of the package repositories, and the dependencies will be downloaded automatically, single command download+build is useful
- in a bank/SC environment: manual process to obtain and add locally the dependencies, the build/dev system has limited networking: automatically download fails, single command download+build is not useful but not harmful either
Having a "prefetch" and "offline-build" commands are perfectly fine, but I don't see the reason why the default shouldn't be a "build" command that does prefetch+offline-build
Another part was using location-addressable storage for dependencies, rather than content-addressable. If dependencies used something like magnet links, IPFS, etc. then nobody would really care if some particular machine stopped serving some particular file.
Yes they would. Because such systems can only serve files that exist on at least one node, people who want to rely on such a system to serve a given file must run their own node to host every file they care about the availability of.
The topological properties of distribution are different, but the existence of the local cache to provide dependability is not avoided.
mvn dependency:copy-dependencies
Normally you do want to simply have your build download (and cache locally) a new version of a dependency, but an internet connection is not required if you have the dependencies available locally. Maven will always look in your local repository first, and will not try to download anything, except if you are depending on a SNAPSHOT version of something (which implies that the build should always check for a newer version).
Not really. Building from local sources only or "living off the grid" as parent poster put it typically isn't just about temporary loss of Internet connectivity, but instead serves as a shorthand for a more complicated set of requirements.
Generally speaking there will be (at least) disaster recovery requirements such that a computer with a fresh OS install and access to the local very-well-backed-up archives can produce a build. And the local archives typically have exactly one entry point so that legal and process controls can be applied as they go in.
So the problem isn't how to cache dependencies for a working directory, but how to get, archive them, and distribute them to a development team easily. A lot of package mangers approach supporting this kind of use case, but still need a fair bit of customization and manual steps if not new tooling built around them. And all that infrastructure also has to meet qualification criteria (through testing).
It gets to be enough that any tool that decides to add in their own half-assed download manager will be quite annoying to anyone that doesn't use "YOLO" as their quality management system.
I think that this is an issue of properly covering a use-case, and having simple configuration options to avoid manual and cumbersome steps... But I don't think this means that a build system shouldn't default to downloading its deps.
Basically, if "build systems should only build" means
- build systems would be better off with an offline build mode available
I agree 100%... Otherwise, if it's
- build systems shouldn't have an option to automatically download the dependencies and then build
I'm still in disagreement
Of course they get intermixed in practice, package management provides build prerequisites, and completely separate tools are not necessarily conductive to getting information from the build system (or dependency list in the tarball) to the package manager. Or for the download/installation provided by the package manager to be at the perfect scope for the software build.
I, unlike grandparent poster, am not making statements of "ought" for build systems. Just providing context.
However, I can still generically say, the tightness of the build and package management limits user flexibility. I'ts not even a particularly interesting phenomenon. It's just another case of software coupling. There are benefits to high coupling and benefits to low coupling. Typically tight coupling means the normal use case works better at the cost of all abnormal use cases. It's the same-old-same-old of all software design.
----
As a side note, I will say that in my experience an inflexible or awkward to use "offline" build mode is not an offline build mode. If "offline build mode" means:
-The tarball contains (or build system generates) a list of files that can just as easily be fetched with `wget -i deps.txt` and dropped into a subdirectory manually.
That's cool. If it's:
-I have to reverse engineer a one-off package manager architecture, write a script to fetch build-deps externally and hack the build to support my particular flavor of offline build because it's not the type of offline build envisioned by the tool author
Then that's bunk.
I'm not sure they can, or at least not without creating a circular dependency. It is greatly advantageous for build steps to be first-class packages that can use the library ecosystem etc.
On one hand frameworks and code in general becomes more and more bloated and incomprehensible these days, due to an overzealous approach in the aforementioned. On the other hand tools are getting cramped with features which were never in their original scope.
The leftpad issue was only one example of that lets-inject-third-party-code-on-the-fly mentality.