Show HN: Earthly v0.6 | Hacker News Reader

Show HN: Earthly v0.6

earthly.dev

116 pointsvladaionescu37 comments

Hello HN. Earthly is a tool for build automation and we just had our biggest release yet.

Earthly is a modern build solution that is similar to a sandboxed makefile. It works with your existing CI and the #1 reason people use it today is to able to reproduce a CI build locally.

Today we promoted a number of important features to GA status in our 0.6 release including:

    - WITH DOCKER : Earthly can execute any number of containers in parallel, via an isolated Docker daemon that it starts up within.
    - User-Defined Commands: Extract common repetitive build steps into a command you can use across your projects.
    - Shared Cache:  Earthly v0.6 now provides shared caching. This  extends our existing build caching to work in ephemeral CI runners by storing the cache in regular container registries.

We started working on this project in 2020[1] on GitHub [2] and while time has gone quickly the number of people and projects using Earthly now is truly exciting.

Let me know what you think! Feature requests always welcome :)

[1] https://news.ycombinator.com/item?id=22890612

[2] https://github.com/earthly/earthly

I love earthly! It has been a joy to use.

One thing I haven’t gotten right is caching. Some of my builds have targets they depend on that are very expensive but rarely change, and I haven’t been able to make sense of when those targets are (re)computed and when they aren’t. This results in many builds taking a long time, when the targets could be reused from a cache.

Earthly is also missing something that I find to be a glaring omission from make—give me a command to simply list the targets in an Earthfile. ‘earthly ls’? Maybe have a column or two that indicates whether the target saves an image or artifact. When building a project I didn’t author, it’d be so helpful for the targets to be discoverable.

+1 on `earthly ls`. I've also thought it would be valuable to have "public" and "private" targets. When looking at a new codebase it can be hard to know which targets I should use when developing, and which are purely for internal use. Good docs help with this too, but they are not always there.

We have a PR for the ls feature up:

https://github.com/earthly/earthly/pull/1472

Public / Private is interesting.

Private would not be callable from command line and not be listed in `earthly ls`?

Yeah, exactly. Or even just not listed in `earthly ls`, maybe you'd still want to be able to call them manually for debugging purposes.

The `earthly ls` idea is great. We do have shell autocompletions but that is not quite the same thing. I will add a ticket for that.

Cache misses can be a bit inscrutable. It could be the buildkit GC is running, because disk space is getting scarce, or that some arg or file change caused the cache to be considered invalid.

Caching is an area we will continue to improve. We have a proposal for extended cache mounts here[1].

Thanks for using earthly!

[1] https://github.com/earthly/earthly/issues/1399

I didn't know about the autocomplete, that is really helpful; and I'm excited to see `ls` PR'd! Huge thanks to the contributors.

I think some of my cache misses were a rookie mistake: I was building a few expensive targets on a `FROM ...:latest` base. The base image changed often enough to make the cache misses annoying, but infrequent enough to make me think it was GC. My builds are much happier now! :)

re ls: `https://github.com/TekWizely/run` displays targets in the usage command (ie `run -h`). It might be worth looking at their output for inspiration. (Runfiles are doing something similar to Earthfiles).

Just a quick drive-by to say thanks for the call-out - I hope you're finding run to be a useful tool and would love to hear more about how you're using it!

> The #1 reason people use Earthly today is to be able to reproduce CI builds locally. Earthly helps create a consistency layer, providing a level of guarantee that the build will run the same on your machine as on your colleagues machine, and as in CI. We call this consistency repeatability (note that this is distinct from reproducibility as full determinism is not guaranteed).

> This helps avoid cases where tweaks to the CI script take long interaction cycles to test (git commit -m "try again..." anyone?)

Now you have my attention. I haven't done much CI work and the long iteration times are the main reason why. I have to edit a file, commit, push, wait for CI to start, wait for CI to complete, search through 10k lines of logs to find the error, repeat... seriously? This is the best we can do? I assumed that I was missing something.

I should be able to just `gh run-ci` in a repository on my machine. Or take one of those fancy online dev environments that everyone's making now and specialize it for editing CI files. Open an editor, let me click Run to run CI on-demand, have live error messages for syntax and missing variables, show a pane with a description of the CI jobs, show what config options are available for each job as it's being edited. Then once the changes are working I can commit and open a PR.

Yep. An Earthfile describes your build steps in a containerized format. Because of this sandboxing it can run identically in CI or on your local. So you just describe your build in the earthfile and call it from CI. This means you can iterate locally and it also means switching CI vendors is simple.

You can see this in action with the Elixir Phoenix framework. They have an earthfile[1] and use github actions[2] to run it for CI.

Even if you don't use Earthly, using a vendor neutral format to describe your build process can help with iteration time.

[1] https://github.com/phoenixframework/phoenix/blob/master/Eart...

[2] https://github.com/phoenixframework/phoenix/actions/runs/146...

Nice. Been looking for something like this - a Dockerfile equivalent that isn't governed by the Docker folks, with whom I generally do not see eye to eye with.

Bookmarked for later. Glad someone is thinking about this problem space!

What is your gripe with the Docker folks?

Docker's codebase is a mess, the implementation is a mess, the data formats are a mess, the performance is a mess, it deadlocks under pressure (at least it did several years ago when I worked at Vercel), and I don't like its recent power moves to monetize people already somewhat locked into Docker.

Just my opinions.

But the dockerfile format has nothing to do with the docker codebase right? I run all my docker containers with podman (I recently moved over due to the monetization issues you mentioned), but I still use dockerfile format because they are (imo) good enough and also ubiquitous

The dockerfiles are technically specified by OCI but the Moby project has a lot of sway in that committee. The Dockerfile spec has a number of deficiencies in it that aren't being addressed. Until recently, Docker was the only fully supported, ubiquitous way to build container images from a Dockerfile. The file even being called a "Dockerfile" is, in my opinion, annoying too.

Podman of course allows `Containerfile` but meh. I just don't like the tight coupling of the Dockerfile spec and the Moby project, since I don't like the Moby (Docker) project for the reasons listed above.

I tried using Earthly, great product.

Unfortunately, we rejected it at our organization due to how verbose the commands are. I want my users to be able to say `earthly plan-stage` for a terraform repo, and for it to work. But you cannot.

You cannot have earthly automatically use environment variables from your env, you have to explicitly list the commands yourself using flags:

$ earthly \

  --build-arg AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID \

  --build-arg AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY \

  --build-arg AWS_DEFAULT_REGION=us-west-2 \

  --build-arg AWS_SESSION_TOKEN=$AWS_SESSION_TOKEN \

  --build-arg AWS_SECURITY_TOKEN=$AWS_SECURITY_TOKEN \

  +plan-stage

This means if I want to `terraform plan` something, I need to explicitly set all of my AWS environment variables.

Compare this to `task plan-stage` or `make plan-stage`.

It is much too verbose and will only get worse as we add features to our pipelines.

You may want to use secrets like this:

     earthly \
      --secret-file config=/home/user/.aws/config \
      +build

But yes, Earthly optimizes for reproducibility, so implicitly pulling in ENVs is not something it does.

The required build arg and env var names are already listed in the Earthfile as `ARG` and `RUN --secret ...`. So I think specifying it in the commandline is redundant.

I didn't use secrets because it is a prototype, and I need to explicitly give RUNs access to secret values, and I got sick of the boilerplate.

Interesting. Dev tools post always catch my attention. For me, your comment regarding `build-arg` is in the same bucket as the comment regarding the need for a `ls` command. Ultimately I predict that the requested feature set will eventually be equivalent to what is already in Bazel -- like `.bazelrc` for setting up build variations and environment variables, and `query` for figuring out what is going on with the project. If Bazel has been working on these features and issues for years, I'd expect the same here.

Earthly seems very useful to work around issues like cross-compiling. A simple example that I'd expect a lot of people to run into is running into libc/stdlibc++ differences with Alpine containers. That is easy to come across when doing something as simple as pulling in a Python pip dependency that wants to compile native C/C++ code. I find this extremely difficult to learn to do in Bazel.

However if you're in the league of complex cross-compiling, then you may already be on CMake and/or Bazel and have the dev resources to define the toolchains as needed.

It’s a build tool, I’d be surprised if something like it worked well for non build related tasks. I suspect bazel would be equally verbose (if not even worse or impossible).

I think you’re better of using something like taskfile w/ earthly for a really good workflow, it gives you the best of both worlds.

(Actually I think I saw this on their slack)

Yeah, that was what I was doing in the prototype stage. But then I thought "why do we need two task runners? " and ended up just going with task files.

They sell it as a make replacement. Bazel constantly says "you don't need hazel, it is a big tool" and earthly does not. Which is fine.

Either way, I already said earthly is a great tool. This is just my feedback around why it doesnt fit my use case.

Ah sorry, that’s fair. We went the other way from having just taskfile to having both and it seems quite nice so far.

We may go back! Earthly is quite nice

That's my biggest complain about earthly as well. There is `.env` support that ease the problem a bit: https://docs.earthly.dev/docs/earthly-command#environment-va...

Worth watching the reported issue at https://github.com/earthly/earthly/issues/707

We do love getting all this input. Finding a way to make most use-cases ergonomic while still being reproducible is very important to us.

Thumbs-ups on tickets and comments help us prioritize features.

We already have a PR up for the `earthly ls` feature.

https://github.com/earthly/earthly/pull/1472

could combine it with something like: https://github.com/bbugyi200/funky

Oh geez.

Dockerfiles are reason #1 for docker's popularity.

Shipping 'the lot' so the artifact works everywhere is #2. This is incredibly inefficient for most use cases.

`docker pull random-shit` is #3.

Everything else it gives is largely immaterial, or maybe even a negative.

You guys are solving #1 and #3, and providing a better alternative to #2. I'm very excited for you and your project!

I haven't tried the product but looks interesting.

I've actually been looking into nixos to generate docker containers after that repl post the other day.

Aside from the software itself I wanted to compliment you on this image.

https://earthly.dev/blog/assets/images/earthly-v0-6/earthly-...

I really appreciate how much context is added there for what earthly actually does or is trying to do. I'll probably save that just to have it as an example to ask people to recreate when I'm baffled by their landing page.

I think in your Articles / Tutorials you should show how to migrate(?) e.g. a "Hello World" Maven project running in GitLab CI to Earthly. This will help people understand it better (from a concept they're already familiar with).

Great suggestion! We have a walk-thru in the docs and some examples in GitHub.

https://docs.earthly.dev/basics

https://github.com/earthly/earthly/tree/main/examples

Do those help?

Why is it called "earthly"?

How does this differ from Mesos?

I saw that you were downvoted for asking a legitimate question. Can you expand on what you mean?

Having done a lot of work in the past with Mesos and Docker and having done the deep dive on Earthly I just don't see the corollary between Mesos and this. Maybe there's more context you can provide :)

xkcd #927 irl.

Reply on news.ycombinator.com