Just: A Command Runner
github.com
github.com
Make never stuck for me - I couldn't quite get it to fit inside my head.
Just has the exact set of features I want.
Here's one example of one of my Justfiles: https://github.com/simonw/sqlite-utils/blob/fc221f9b62ed8624... - documented here: https://sqlite-utils.datasette.io/en/stable/contributing.htm...
I also wrote about using Just with Django in this TIL: https://til.simonwillison.net/django/just-with-django
Do any of these make alternatives reinvent the paradigm? No, but they do offer some quality of life improvements I wish were within reach without jumping through hoops.
I never thought about .PHONY as a workaround to anything. Just a slightly unnecessary annotation that you may add to the makefile if you want to be pedantically correct.
Maybe it's just that my brain has such a strong negative reaction to that PHONY hack that I was never able to get past it.
Maybe Just's tagline should be "it's Make, but you don't have to remember what PHONY means".
Make’s built around creating files. Phony works around that.
Both are possible in make, but are extremely non-obvious and come with a bunch of caveats and require some heady code blobs at the beginning of the file.
From what I can tell, Just accomplishes what a set of scripts in a directory could also do, but in a more ergonomic fashion. Since the justfile is a single file, you're not cluttering up your directory. You don't have to look for all *.sh files, but can instead do a "just --list". And using "just target" is probably easier to type than "./target.sh" since the just version has no punctuation.
What are some of the other benefits of Just that makes it superior to "a set of scripts in a directory"?
And nice stuff like command/options parsing + nice help messages.
That's exactly the value proposition here. It's a small quality of life improvement, nothing more.
I'm not a software developer, but a scientist, and eventually like 5 shell scripts fill a directory. Having them all be in one place sounds neater and easier to document and come back to a few months later without needing to either open up each individual file to read its own comments or to make a separate readme file, seems good for small projects.
Knowing that 'Just' is an alternative to Make for running project utility scripts, it may gain popularity and become as well known.
Accepting arguments alone seems to be a pretty big value add.
$ make HOST=dev1.example.org deploy
But I guess, enough of the time, what people want is just a nice tidy way of organising tasks to run. Not drastically different to a set of shell scripts, but organised differently, with a few extra features and less boilerplate per task.
What often happens is that people should use `make` (or `just`), but don't, and instead end up writing a poor replica of `make` as a custom python script for example.
And then that python script perhaps shells-out to a subprocess because you can't run something in python directly as a lib, so now you have to import and invoke subprocesses, and so on. Building and deploying a static website using just or make is 4 lines.
You can do the same with any other scripting language, because with most popular languages it's quite easy to handle cli-parameters on that level.
And using a single messy file is usually not a good pattern. But on the other side, "just" comes with builtin tab completion which you would not get for any random script out-of-the-box. And if you are only using half oneliners or very short scripts, the messiness of a single file does not matter that much. Though, how well this will scale over time is a different topic.
I find my Justfiles rarely ever get large enough to become messy, and then I just factor out the large tasks as scripts and it's all nice and tidy again. Almost all of my projects with a Dockerfile have docker-build and docker-run-local just tasks and those are pretty much always one-liners, and usually that's as complex as it gets. I treat Justfiles more like runnable readmes than a framework for homebrew build systems; if I need a proper build system, I'm probably going to invoke it via a one-liner in a just task.
It’s easy to install on any internet connected server that you have install privileges to.
There is a nice trick that gives makefiles this ability: http://marmelab.com/blog/2016/02/29/auto-documented-makefile...
Adding .PHONY targets and so forth is a bit inelegant, but I can share a makefile with confidence that any Linux/Mac OS/BSD user can use it without needing additional software, and I will never have to worry about make becoming unavailable or no longer maintained. Just my personal opinion.
Makefile portability can be tricky. Especially if you try to do something fancy with the makefiles.
GNU Make has features and syntax that the other Makes don’t. Likewise there are features and syntax that some Make programs have that GNU Make doesn’t.
I'm sure you're kidding, but in the case that you're not: Make portability is gross.
We have ./configure steps precisely because Make is difficult w.r.t. portability, but even if that wasn't the case and you were just using make as a command executor: you still have enormous warts.
Oh, and yeah, you'd need whatever additional software too.
Be it: headers, linters, formatters, libraries or test suites that you've bundled.
Valgrind is a popular make target, but "Make" does not bring in valgrind (for example).
Honestly one of the most backwards things the Go community did was adopt "Make", it's so kludgey even as a pure command executor that I can't really take anyone seriously who argues for it's use.
I'm not saying "Just" is a replacement, I don't know what is.
Most of the arguments for using Make boil down to "I enjoy typing `make <something>`" and "you probably have it installed already?".
Make portability is bad enough on UNIX-like OSes, to say nothing of what a crapshoot it is on Windows. Even if you do have make installed on Windows, there's no guarantee that what it shells to is going to be able to run all the commands people tend to put in there.
Plain old shell scripting is much more portable than make, because a shell is definitely installed on every UNIX-like OS, and there is a very clear baseline of functionality that works in every Bourne/POSIX descendant. And it's quite likely to exist on a modern Windows developer machine too because bash is bundled together with Git.
My theory of writing developer scripts is to prefer the tool that already exists in the language you're developing. Gradle for JVM, npm for Node etc. Otherwise just use shell. Make feels like wrong tool for the job.
Let me know what shell scripting selectively builds only the artifacts whose dependencies have changed (honest question).
Where is the best place to learn what this is? I'd love to make sure I'm writing portable shell scripts when I do have to.
There's also the issue that "shell scripts" often involve using binaries which you might not even realise are binaries (is "echo" a shell builtin or a binary? I forget) and which may differ from system to system. I've been bitten by grep issues before writing scripts across Ubuntu and OSX.
If I was about to ship an open source application that came bundled with some shell scripts, then I agree portability is good, so that I know the script would run for people who might not have Bash installed.
But at ${DAYJOB} I much prefer to let Bash run all my scripts, and I make that explicit via ‘#!/usr/bin/env bash’.
Bash is still evolving and the Bash devs are adding new features that I would miss in pure sh. Case in point: A (somewhat) reasonable way of working with arrays.
> Be kind. Don't be snarky.
But, I'm not being snarky or rude... I'm just not 100% sure he is being sarcastic.
Given that Make is not portable and does not include _any_ tools that it is commonly used to execute, I consider it hugely sarcastic.
However, other lines were true and it sounds like it was written seriously, so, can't tell if sarcasm or not.
I went into more detail for you here: https://news.ycombinator.com/item?id=34319254
(btw, most of times I've used "Make" has been times where the pain has been done for me or I'm using Python, Go, Docker or in one case: Rust. I have never touched Autotools or C-compilation myself with Make, my arguments have nothing to do with C compilation at all.)
Not make.
Not the case.
I could just as easily talk about the pathing issues between MacOS and Linux: or the multiplicity of issues surrounding `sed` (GNU) and `sed` (BSD), or that the `black` python formatter won't be installed by default.
It is not as easy as the author claimed, you need whatever dependencies you call on, obviously.
But you could be referencing any number of other things in the guidelines. It would be incredibly helpful if people on HN who engage in community moderation actually explain their reasoning. Here I’ll help:
> Comments should get more thoughtful and substantive, not less, as a topic gets more divisive.
If you’re bothered enough by someone’s comments to link the guidelines, sharing a thought about which part of them is pertinent would be more thoughtful and would facilitate a healthier discussion.
> Please don't post shallow dismissals […] A good critical comment teaches us something.
This is one I’m working on too. If it’s worth challenging something, it’s probably worth challenging it with some proverbial meat. It might seem obviously wrong to you, or to me, but it doesn’t always seem that obvious to everyone else.
I’m certain you meant well with this, but I also think OP would benefit from clarification. I know I would too, and I expect the discussion would benefit as well.
Autotools, which generate configure scripts, was built to work around the specific issues associated with old-school C cross platform compilation (with shared libraries, version differences, and misc libc editions). Ditto valgrind, et.al.
So, yeah. Make’s fine. You just don’t like C. Which, that’s cool, just unrelated.
To go into issues though:
Make itself executes by default with `sh` which is wildly different between platforms.
Even if you write portable enough shell; Paths are still incompatible between OS's and distros.
You still must ship your tools, which is a direct contradition of what is mentioned.
In fact; I just googled it and this chapter from Managing Projects with GNU Make; talks about the issues in making Make (GNU Make, as opposed to BSD Make, which is different enough to have broken my things!): portable
https://www.oreilly.com/library/view/managing-projects-with/...
Fixing that within Make would require it to be platform aware. Not just “is this Linux” but “which flavor of Linux is this and what version of that flavor”. It’s also highly specific to C.
Perhaps your other complaints here are valid, but they’re issues I’ve never run into myself, in my 20 odd years with it.
EDIT: Would you blame Just for not handling C cross compilation capability built in? Would you blame Just if someone automates the creation of Just command files to work around a particularly nasty workflow?
Look, I'm not taking your tools away, there is no need to be defensive.
Make isn't going anywhere, but it is a bad tool, the syntax is completely arcane and it's designed for things few people actually need these days.
The most common case I've seen of modern Make usage is `make docker` and for Golang, where it doesn't get an opportunity to stretch its legs as a dependency manager at all -- making it a glorified task runner.
The portability aspect is all I mentioned, because that was all that was in question.
But if you really want me to get into it, I can be quite cruel.
Just because you spent 20 years learning or using something does not mean it is a good tool, I'm glad it works for you, truly, but it is an abomination and people only continue to use it for a sense of sunk cost fallacy or by telling themselves that "most people have it installed already".
One of the reasons autotools is so complicated is it is used to build GNU Make so has to work with whatever crummy make your Unix vendor shipped with
What exactly are you talking about? Are you upset that make isn't a cross-platform package management system?
You don't know what alternative there is to Make and then in the next breath you say the only arguments for it are personal preference?
and, if you hate yourself: https://bazel.build
> What exactly are you talking about? Are you upset that make isn't a cross-platform package management system?
Ok, so you were serious with your claims of portability, that is concerning.
The majority of times I've seen Make used it's primarily been as a task runner.
For example:
TAG=some-service
SVC=website.com/$(TAG)
BUILDER=golang:1.19-alpine
export REVISION_ID?=unknown
export BUILD_DATE?=unknown
RUN=docker run --rm \
-v $(CURDIR):/opt/go/src/$(SVC) \
-w /opt/go/src/$(SVC) \
-e GO111MODULE=on
build:
ifeq ($(OS),Windows_NT)
# Workaround on Windows for https://github.com/golang/dep/issues/1407
$(RUN) $(BUILDER) rm -rf vendor vendor.orig
$(RUN) $(BUILDER) rm -rf vendor vendor.orig
endif
$(RUN) -e CGO_ENABLED=0 -e GOOS=linux $(BUILDER) \
go build -o service -ldflags "-s -X main.revisionID=$(REVISION_ID) -X main.buildDate=$(BUILD_DATE)" \
./cmd/some-service/...
# $(RUN) $(BUILDER) rhash --sha256 service -o service.sha256
docker build --tag="$(TAG):$(REVISION_ID)" --tag="$(TAG):latest" .
run:
docker-compose up
serve:
go run ./cmd/some-service/...
dev:
ulimit -n 1000 #increase the file watch limit, might required on MacOS
reflex -s -r '\.go$$' make serve
Edit: We're actually using this - and I remember there was a modern version with colors and a cool short form. But I can't find it - anybody else got nice examples?
Make is not _that_ portable. If you're using high level languages and only need a task runner to kick off your compiler, watch rules or similar and need portability, you could write an executable bash script with functions that serve as your commands;
#!/bin/bash
set -e
function build() {
echo "Your build steps"
}
function clean() {
echo "Your clean steps"
}
eval $1 $@
$ ./task clean
$ ./task build
Let me know what a bash script would look like that does dependency checking for each build artifact
In my experience, `make` also needs to be installed. (On ubuntu it's part of `build-essential`).
I guess more precisely, `just` might not be available in all package managers?
https://gist.github.com/prwhite/8168133
The linked blog's solution is also mentioned in the gist comments.
https://gist.github.com/prwhite/8168133?permalink_comment_id...
https://github.com/lpsantil/oop0/blob/master/Makefile#L87-L1...
Take the example they've screenshotted in the readme[1]:
alias b := build
host := `uname -a`
# build main
build:
cc *.c -o main
# test everything
test-all: build
./test --all
# run a specific test
test TEST: build
./test --test {{TEST}}
build.bash:
#!/usr/bin/env bash
set -o errexit
cc *.c -o main
#!/usr/bin/env bash
set -o errexit
"$(dirname "${BASH_SOURCE[0]}")/build.bash"
./test --all
#!/usr/bin/env bash
set -o errexit
"$(dirname "${BASH_SOURCE[0]}")/build.bash"
./test --test "$@"
[1]: https://github.com/casey/just/blob/master/examples/screensho...
Last time I checked, git bash doesn’t ship make, either.
My personal favorite feature is the ability to load environment variables from a `.env` file and set them for all commands run. Just have to add this to the top of your `Justfile` to make it happen:
`set dotenv-load := true`
make can do that too. Put
include .env
export
Looks like there are some caveats and a slightly different approach: https://unix.stackexchange.com/a/235254
The simple fact that a build target isn't executed if the build file already exists, requiring a workaround to run it, points to the difference in purpose and philosophy. Those little details, together with the more modern design, accumulate to make Just a tool that is simpler to use for that purpose.
I get the reverence around make but don't blind yourself to potentially better ways of doing things.
I would even go so far as to say that intentionally avoiding new ways to approach these things will necessarily blind you to better ways once they do come along, and I wonder how many good tools have died because "[x] does all I need." (replace "x" with whatever you like.)
I wrote a post about that here: https://rosszurowski.com/log/2022/makefiles
No, its not, and Just supports Windows, where that is particularly true.
When something doesn't work as expected, I dive in as deep as the rabbit hole goes to get across the line.
Curious what led you to arrive at "Aha! They must be an ops person", will you humor me with an explanation?
It is possible to reimplement a relatively easy tool like make, and having learnt from its historical shortcomings it can be better irrespective of the implementation language. But that’s a different point.
It's a bit bizarre to me that their example involves building code, since that application generally benefits from this "idiosyncratic" behavior of `make`.
You probably would want the behavior of `make` for the `test` command too, to avoid running potentially time-consuming tests unnecessarily. Bazel caches test results to avoid this, for example.
[0] https://github.com/casey/just#what-are-the-idiosyncrasies-of...
I've also rarely ever touched software that is actually free of nondeterminism, so I am deeply skeptical of caching anything but the simplest test case. And those are fast.
Mock the entire socket API, so you're not actually testing your software?
At the end of the day you need to run some tests with external interactions, and you shouldn't cache those.
Also keep in mind any time you perform syscalls, you have external interactions. Their behavior could change and you want to know if they have intermittent failure scenarios. Using threads? You're nondeterministic, you should keep running tests.
Keep tests that open sockets and otherwise interact with resources not controlled by the test environment itself separately, what most people like to call integration tests. Only those cannot be cached - and it's a huge trade off and while you get a lot of value from those tests, exactly because they can't be cached, you want to keep those to a minimum.
Certainly, you can optimize some of your tests by caching them! But they should be fast anyway, because your software should be fast. The software I currently work on has massive test suites containing around 50k tests that run in about 3 minutes per suite on my PC. It's time consuming, I suppose, but caching would mean that I'm not actually flushing out latent bugs. Instead, we run all of the tests every time and do things like randomize the execution order, so we find bugs before customers do.
At the point where your software interacts with threads (or the kernel!) you've already brought in non-determinism and trying to act like your tests are really deterministic is naive.
Learn to separate tests that use outside resources and behave undeterminastically from the ones that are never flaky, make sure to cache the latter. If you don't do that you're still in the dark ages of testing when tests are a pain to touch and fail all the time for no reason.
Then the destination of that socket should be managed by the test too.
For example: https://pkg.go.dev/net/http/httptest#Server
If you are not sure of what I am talking about, try typing
`make -p`
those builtin rules can be disabled if you are not building ancient artifacts but that we have to is why one of these work-a-likes is going to win some day.
My general approach is every repo should have something that follows https://github.com/github/scripts-to-rule-them-all, written in sh (maybe bash, its 2023), linted with shellcheck. When you need something fancy Rake is great or grab some nice bash command line helper and source it from all your scripts. Is a command listing really worth another dependency over what you get from `ls script` or `cat script/README` ?
I like gli because it gives you subcommands, like `gli database refresh` etc.
It’s very simple so isn’t good for everything, but works well as a simple command runner.
Glad to see people are finally going back to ahell scripts since they are very ergonomic and with a little portability in mind, they can be cross platform too.
I've been using it for a while and it's pretty flexible:
- dependencies
- parallelism
- programmatically generated tasks (since the config file is just a Python file)
- "udf" to specify when a task is up-to-date and can be skippedThere are a lot of language specific tools like this. I love Ruby's rake.
How is having to use Python different from having to use the Justfile language?
The alternative Taskfile can do that https://taskfile.dev/usage/#including-other-taskfiles
I would rather lose my hair screwing with a makefile like thing than add more yaml to my day to day; I currently say “I hate yaml” at least 4 times a day.
If it’s not too much trouble and I don’t need comments I just write json as yaml.
Just doesn't seem to even support caching task results (by declaring inputs/outputs)?
Stuff like `make` is already there, always. In my case, I assume everyone has Python 3 and include a `task` script using this template, which does something similar: https://gist.github.com/sirikon/d4327b6cc3de5cc244dbe5529d8f...
Also, with Python you have... Python, all the language's power to extend your scripts as needed. `Just` works with a shell like bash, and I pretty much prefer python for scripting. Bash scripts get complicated very quickly.
Your package manager likely also has just: https://github.com/casey/just#packages.
Not that I fault it. It’s supposed to be for making programs hence the name. We’re abusing it by turning into a script collection.
I'm done trying to cast spells at Make
If there's was a short Make: The Good Parts O'Reilly book, then I would probably read it.
It makes life a lot easier when onboarding new folks, or for remembering how to do something months later.
I also use make, but limit it to just building software. All of my repos have a Runfile and a Makefile.
Someone at the company introduced Just in their projects. I’ve used it quite a bit now, it’s great EXCEPT that you cannot include other Justfiles. So abstraction is impossible. If I want to implement something like a push feature, that has to be implemented everywhere, with no way to centrally update all projects.
* https://github.com/TekWizely/run
Support for including other Runfiles was recently introduced, with support for globbing and the ability to indicate if an error should be generated if no files are found.
Can anyone comment with their experience using this? (in particular the social ramifications)
With nix, all sorts of niche packages are available, so installing just isn't difficult. (& packages nix installed only go in /nix/store, so the filesystem isn't messed up).
If you want the same programs (the same version of programs, even) across different Linux distributions, and macOS, nix is the best tool for that.
If you're worried about your environment being an unreproducible snowflake, nix's main advertised feature is reproducing package installation.
Although, yeah, Nix suffer the same cost of "1 more niche thing to install".
All sorts of tools use xxxxfile for their config. But the tool “make” it’s actually short for “make file”. That’s what make does, it makes files (and has modified date dependencies)
RUN : A Task runner that helps you easily manage and invoke small scripts and wrappers
* https://github.com/TekWizely/run
Do you find yourself using tools like make to manage non-build-related scripts?
Build tools are great, but they are not optimized for general script management.
Run aims to be better at managing small scripts and wrappers, while incorporating a familiar make-like syntax.
Some features:
* Auto generates command list
* Auto generates help for commands
* Supports defining command-line arguments, which are presented in help text
* Can be composed via includes
* Each command can define its own shell (or even python, etc)
If you're interested in task runners, I hope you'll take a look!
As far as unseating Make as a command runner, I think that might just take Just being available in more places, since one of the main advantages of Make that many users cite is that it's available everywhere. Just is already available in a lot of package repos, but not all of them. Finally packaging Just for Debian[0] would help a lot.
- https://github.com/casey/just/issues/1163 - https://github.com/casey/just/issues/503
Manually installing stuff sucks. It should be cross platform, automated and local by default whenever possible.
I ended up using it over just because it felt easier to use cross platform, and toml seemed like a right choice
The intended way seems to be to invoke just with explicit --justfile and --workdir arguments, and wrap that invocation with a shell alias/function.
The installation section has no direct body, but there are subheaders with the actual instructions.
Sure you could accomplish something similar with direnv to maintain the project-level scope. But Just makes it more explicit.
I think looking at the features, you see "minor quality of life UX improvements", and it's not mind-blowing. (just commands implicitly run with the workdir of the justfile, just --choose uses fzf, soft tabs not required, etc.).
The effect does end up being a nicer tool to use.