Hacker News Reader

Comment by jcranmer | Hacker News Reader

Perusing this article quickly, it means a CI that is automatically handling things like caching build artifacts so that you don't need to recompile your entire repository every single commit. It's not about a faster program to call exec; it's about a program that knows it need not even call exec.

View on HN

Incremental builds are something even enterprise build systems fail at.

I've had Visual Studio and MSBuild fail on me multiple times by deciding not to recompile a certain .cpp file, because they got the dependency graph wrong. This results in a running but inconsistent executable that's a bitch to debug.

If you're selling me another tool that duplicates this work, but that has even less context on my project than the build system, I'll pass.

One thing these tools should do automatically is at the end log to the console (or whatever system) what was compiled and what wasn't.

This made debugging so much easier for these issues. They're bound to happen, because we're all human and bugs are a thing. Automatically being able to audit the build as step 1 makes this easy to spot

Visual studio does give a fairly detailed explanation of what was compiled each time, in the output>build view.

Bazel does this.

I have to be honest - MS Build does not scale well. Companies use it because it’s the default.

Seems they came up with a way to scale up build toolchains with BuildXL[0]

Doesn't seem fully baked yet though

[0]: https://github.com/microsoft/BuildXL

I don't have a ton of experience in Bazel, but from what I have experienced with Bazel I think it would be more accurate to say that you configure the dependency graph yourself in Bazel. Then Bazel can determine what to build from scratch and what to take from cache

Yes, but Bazel requires an all-in commitment and also tons of work to fit the rest of the world into it.

Yes, exactly.

Earthly is explicit about inputs ( using COPY) and outputs of each build step. This let's it be 100% certain about whether it can cache parts of the build.

If you are familiar with docker layer-based caching and Bazel, you can imagine how Earthly works to eliminate rework in builds.

Here is a small monorepo example: https://github.com/earthly/earthly-solutions

I looked at Earthly just a couple of weeks back. It’s a tool that makes perfect sense - especially when I was deep in a makefile wishing the caching was as automated as docker.

In the end I decided that I couldn’t face the complexity of dealing with someone else’s hosted tooling. Luckily in this case.

except not really.. their tutorial [0] includes no-compile languages like python and JS, with only non-trivial compilation being and Java... and how do they do it?

    RUN gradle build

At least in this case, Earthly has no insight into how repository is organized, and it _will_ recompile the entire repository with every single commit.

So what's the real value of it? this basically seems like a better "docker build" alternative, giving nicer inter-layer caching. Maybe will save some time on dependencies if they take some time to install and devs were too lazy to set up own workers and used ephemeral runners instead.

[0] https://docs.earthly.dev/basics/part-1-a-simple-earthfile

That's step one, 'how to write a basic Earthfile that just does the thing', so they don't delve into the details of caching and how it works, but look at the whole block and not just that one line; it operates like a Dockerfile:

    build:
        COPY build.gradle ./
        COPY src src
        RUN gradle build
        RUN gradle install
        SAVE ARTIFACT build/install/java-example/bin /bin
        SAVE ARTIFACT build/install/java-example/lib /lib

The COPY phases copy files into the build context. If the files haven't changed, then the commands don't need to be run. In other words, if `build.gradle` and `src` are unchanged, it won't run `gradle build` or `gradle install`, and it'll just give you the artifacts from the previous build.

They have a golang example in part three[0], which they redesign to use caching effectively:

    build:
        # Download deps before copying code.
        COPY go.mod go.sum .
        RUN go mod download
        # Copy and build code.
        COPY main.go .
        RUN go build -o output/example main.go
        SAVE ARTIFACT output/example AS LOCAL local-output/go-example

They copy in go.mod and go.sum and then run `go mod download`. If the mod and sum files haven't changed, then `go mod download` doesn't need to be run.

Then they copy `main.go` in, and run `go build`. If the `main.go` file hasn't changed, `go build` doesn't need to be run.

[0] https://docs.earthly.dev/basics/part-3-adding-dependencies-w...

Gradle can do the caching of tasks itself in a level of granularity no CI can. If you keep your build cache "live" in the CI, Gradle knows to only recompile what has changed, and to only run tests which changed (or tests something which changed). I am not sure if many other build systems do that, but this is something I would absolutely expect from the build system, not CI. Do you not expect caching when building locally??

The whole idea with CI doing this is wrong IMO. All I want CI to do is call a couple of commands after setting up the environment (which would include the Gradle cache or whatever to speed up things - no need for your revolutionary CI build system).

A CI system like TeamCity will give you that sort of thing for free though, without needing an Earthfile. It just dedupes builds at the same commit, and keeps checkout directories between builds on the agents, so if the build tool has a cache then it's automatically reused. Works great.

Given that it's hard to understand what exactly this is for? Build systems normally try to avoid redundant work for local purposes anyway. Is it some hack for companies that insist on resetting build environments to zero on every single build, and then decide they want caching back? If so, why not just ... let the builds take place in directories containing previous builds, and fix the problems? That's what we do at my company and it works great (using gradle), there are only rarely problems with unclean builds and it's easy to rerun with a clean checkout if that's suspected to be an issue (the bugs are mostly in our own code which does a layer of caching above the build system anyway).

One of the chief benefits of Earthly is builds that are always clean and predictable. This is especially important on a shared build server.

So they claim. And yet ultimately they do it by caching things, same as every other system.

sure, but in a java project, how likely you are to have a commit with no changes to src/? or for a go project, how likely is it that you want to duplicate you entire dependency tree in a clunky, non-parallel dependency description language, vs just doing "COPY * .; RUN go build" and letting golang figure it all out and build in parallel using all the cores?

Don't get me wrong, there is definitely value in caching dependencies, and I can see Earthly approach help if you have, say, a Python app, with 10 minutes of dependency install time, 0 build time, and 10 seconds of tests - you'll see huge improvement. But you'll also see almost the same improvement if you use Dockerfile on a persistent host instead, and as a bonus you won't have to learn new system.

One place Earthly might shine are monorepos with multiple independent projects that even use multiple build systems... but I am not sure how many of those exist _and_ don't have some other caching solution already.

That only splits dependency-build from app-build. Any change to the app source code still causes a full rebuild, unless you figure out how to share the cache directory between builds (and then parallelism gets sketchy).

Generic CI just simply can't parallelize+cache on a level comparable to systems that understand the build itself (Bazel, sccache, etc).

Isn't this a complex solution for a problem that shouldn't exist?

Reply on news.ycombinator.com