Show HN: Minimal build system using just /bin/sh | Hacker News Reader

Show HN: Minimal build system using just /bin/sh

notabug.org

100 pointsakkartik63 comments

Open articleView on HN

> This repo illustrates a way to manage a project using just plain Bourne sh, available on all Unix systems since 1977.

No, this code uses functions, which didn't exist in the original Bourne shell. They were introduced only in 1984.

Source: https://www.in-ulm.de/~mascheck/bourne/#variants

Thanks for the correction!

It also uses the "local" keyword, which is non-portable.

People like to bash the autotools, but they exist for exactly this reason - they already spent decades working through all these gritty details to ensure the scripts generated are portable. To paraphrase Spencer "Those who do not understand autotools are doomed to re-invent them, poorly."

Uh-uh. Pray tell me an actual extant platform without `local`.

Debian dash deems 3 POSIX extensions basic enough to be in /bin/sh -- including `local`. https://www.debian.org/doc/debian-policy/ch-files.html#s-scr...

I don't care about some spherical-cow ideal of portability. It has its own costs -- like autotools. This is why OP makes no mention of POSIX or portability.

(This exchange regurgitates https://www.reddit.com/r/tinycode/comments/6md2l8/make_gmake..., albeit with more mutual sneering.)

Any OS which use the Korn shell as /bin/sh, for example Solaris 10, 11, and illumos:

  $ /bin/sh -c "fail() { local f; }; fail"
  /bin/sh[1]: local: not found [No such file or directory]
  $ /bin/sh --version
    version         sh (AT&T Research) 93t+ 2010-03-05
  $ uname -rsv
  SunOS 5.11 joyent_20160721T174127Z

It wouldn't work on Solaris 9 and older either as the original /bin/sh also doesn't support "local", but as those releases are EOL it's fine to discount them.

Whether you regard these platforms as extant depends on your point of view. Most people wouldn't, and that's fine. We'd naturally disagree ;)

That's good to know, actually. Thanks! Sorry about my tone before. Are there any other platforms that actually benefit from autotools, that you're aware of?

OpenBSD uses the Korn shell for /bin/sh and this is not true there.

    $/bin/sh -c "fail() { local f; }; fail"
    $

OpenBSD's /bin/sh defines several aliases as standard.

    $alias local
    local=typeset
    $

M. Agaram has stated elsewhere that xe was using OpenBSD sh as a poor man's POSIX conformance test.

> M. Agaram

Who's that?

That's my last name. Perhaps the M stands for "Monsieur" :)

That's the argument for autotools, but I wholeheartedly disagree. Untested code can be assumed not to work, so if a build script doesn't work on a certain platform, what's the likelihood that the C program it's building will work on that platform? When you see a poorly-written autoconf script checking for basic things like size_t, you have to wonder if the program it's building really supports pre-ANSI C. This wastes real time on not-so-uncommon platforms like Cygwin and MSYS2, where spawning subshells is slow and running ./configure can take minutes.

So if you've never tested your program on a platform where /bin/sh doesn't understand local, feel free to use local in the build script. Bashisms like local are safer anyway. Variable scoping is good. (Though I guess the author's claim of it working in a 70s era Bourne shell is still wrong.)

> This wastes real time on not-so-uncommon platforms like Cygwin and MSYS2, where spawning subshells is slow and running ./configure can take minutes.

That's true, but in my case it turned out to be insignificant. While changing all the autoconf junk in > 100 plugins could move the total build time from 50 minutes down to 40, I'd need an order of magnitude decrease in build time to warrant the work. I say order of magnitude because its only at a 5 minute build time that the entire dev process would see a paradigm shift-- at that point I could, with a straight face, stop shipping Linux binaries and just suggest that users compile the software themselves. (Of course that's not great usability for a whole class of Ubuntu users, but at build time == 5 minutes some other dev could easily come along and package things up with ease.)

Also, the extra wasted time under Windows isn't the fault of autoconf. Try doing `git any_subcommand` in msys2 and you'll get latency so high you'd think it were a practical joke.

Edit: added parenthetical

you're missing another huge problem with autoconf and friends.

if it doesn't just magically work, you're largely sol.

sometimes you can put in a basic build in its place, or find some environment variables to fiddle with. but usually..

I just rewrote it without functions in about 5 min. It probably takes less than a minute for a faster typist.

He may have used functions to keep this to a single file and I doubt his 1977 comment (now changed to "30 years") was to be taken literally.

Anyone aware of the overzealous use of more complex build systems in situations where they are not necessary would understand his point. It is possible to build portable software without autoconf and makefiles.

Example: http://nacl.cr.yp.to

I seriously wonder if the author ever worked on a project with more than around 10k LOC, multiple languages and tools, automatic testing, deployment, version control (merging this build script is going to be a huge PITA).

No, you have to realize you're working on one side of the graph (source code), but your build produces something on the other side (your artifact). Without automatic dependency resolution of you DAG, you're constantly traversing from one side to the other, making the build 'work'.

Maybe they're not using a language with insanely bloated compile times?

It's not just about compile time. Couple of examples:

- Parallel downloading / uploading of artifacts

- Multi-step transformation (old-style: lexer to C to Object file to linker to deployment). You can turn this around, but generally you want each output of your lexer to be input to the C compiler and so on.

- Testing (especially brute-force testing might take time)

And in modern situations you have the same, with Typescript => Javascript => Minimisation => Combination => Deployment (roughly).

It really helps to think of this as a dependency chain and use a partial-order planner to find a possible linearization and execute.

There are a lot of projects that don't meet those criteria. Scale is bidirectional. Tools designed for big projects can have features and designs that don't scale down well. Tools designed for big projects often depend on institutional type knowledge where expertise is spread among multiple individuals...at Google, there teams of engineers managing the single code tree is what allows a monorepository.

Alternatives are not replacements.

Yes, it's a team of engineers. But interestingly, Google started with a mono-repo. All through their explosive growth they maintained (mostly, except Android and Chromium) their vision of mono-repo. It clearly shows that their vision of mono-repo works both for small companies, as well as large ones.

A build-script without dependency resolution clearly does not scale. It was the reason to design and implement Make, CMake, Maven, SBT and many others.

To add to this, Google spent a lot of time and effort on building a custom build system (Bazel) to make this sort of dependency tracking in a mono-repo easy. It's pretty great, and I'd rather write BUILD files than Makefiles/shell scripts any day, but the OP appears to have been focusing on extremely simple projects, where shell scripts are okay.

Right, "works for me" doesn't mean "you must do it.".

My apologies for not expressing myself as clearly as I would like.

Some systems don't need to scale. Make and SBT etc. have an impedance mismatch with some projects due to their learning curve and the current sophistication of the development team.

The most important feature of any development tool is not whether it is best practice, it is whether it works. For example, Tarn Adams doesn't use version control for Dwarf Fortress. [1] That doesn't mean Microsoft should have switched the Windows codebase to zip files instead of Git any more than a single developer should start boiling the small lake of SBT just because it is better.

Sometimes, it is easier and faster and better to just build the tool that is needed. And of course sometimes it isn't.

[1]: https://www.reddit.com/r/IAmA/comments/1avszc/im_tarn_adams_...

I applaud the effort, but is "pure /bin/sh" really a good feature in today's systems?

To my knowledge, today every Unix system has Python, usually at least 2.7, if not Python 3. Some even replaced most of their system shell scripts with Python scripts. (Formerly, Perl had this status, but never really caught on in that area.)

Python is a much cleaner language than any shell language, especially with regard to error handling. (yes, I'm aware of "set -eu") Also, if you occationally need some more elaborated data structures or algorithms, you can express these in a straight forward way, rather than a series of complicated (and usually very fragile) text processing steps.

Totally agreed; I'd use a different language if I needed even an array.

I'm having a hard time thinking of a use case for complex data structures in a build script. Can you give a real world example?

Array is the only one that I have found lacking, in my experience.

If you aim for slightly larger projects and efficiency, you'd need at least a DAG (because that's the fundamental nature of a dependency graph).

You can find a nice explanation in the "tup" build tool, although there are of course many others: http://gittup.org/tup/

And yes, you can express DAGs with text and/or filesystem operations (plus symlinks), but not efficiently and writing code that operates on that level is cumbersome.

Ah, thanks for the pointer to `tup`! I'd spent a while looking for it a few months ago but couldn't remember the name.

Tup is pretty darn amazing. I've never had such a high success rate while writing build rules under it. "Once it builds, it has no bug". Not to mention the great speed to begin with.

Is there any similar build system? I know of https://github.com/droundy/fac, but it's not cross-platform

tup sounds similar to qbs. I found qbs to be a most amazing C++ build system, it's the only one I used that is

1) reasonably easy to use 2) very fast 3) very correct

Most build systems fail 3), usually 2) and often also 1).

What makes tup amazing is it traces which files a program touches, so assuming your programs obey the rule "I need to rerun if I have different command line arguments or something I read has changed", you get perfect dependencies for any program. I use it for running experiments with custom programs and scripts all the time, and it just refund the parts I need.

Regarding DAG: tsort (1) sorts input lines by interpreting them as a list of edges. I guess that's what you where referring to when you said "cumbersome" :)

    $ cat > f
    a c
    a d
    b c
    ^D
    $ tsort f
    a
    b
    d
    c

The problem is not tsort, it is adding, removing, updating items in such a text file. (not to mention nasty corner cases like filenames containing whitespaces, or the security implications of filenames containing newlines)

I've used a project in the past that was a number of shell scripts that were written to handle cross-platform and (at least on Linux) various shells (bash, zsh, etc.). It can be done, but it becomes a complete kludge very quickly.

Any large project needs a build system with complex data structures to express all the dependencies just due to sheer amount of them. Add various types of source generation, tooling, non-trivial build steps and simple build systems quickly become infeasible.

> I'm having a hard time thinking of a use case for complex data structures in a build script. Can you give a real world example?

Look at how a modern build system (like qbs) organizes the build. It builds a declarative object graph describing the project and maps that to a dependency graph where each node can be a relatively complex object ("compile C++ file X using flags Y" or "link LargeSetOfFiles") annotated with modification data (like input file hashes). (Then it compares the dependency graph to what is found on the system and creates a list of commands to execute in order to bring whatever is on the system to the desired state, i.e. create an up to date build).

Hash tables are useful for expressing build variants -- some examples at the end of this comment:

https://news.ycombinator.com/item?id=15043633

debug/release, internationalized builds, coverage builds, profile-directed feedback, running parameterized test suites, etc.

Also most current Unixes have at least Bash. Bourne shell is surprisingly rare outside BSD.

Or Korn, which is better than Bash, IMHO.

I see you've not yet encountered Alpine-based Docker containers.

  $ docker run --rm -ti alpine sh             
  / # bash
  sh: bash: not found
  / # python
  sh: python: not found
  / # perl
  sh: perl: not found
  / # ruby
  sh: ruby: not found

Sure, you can install all of these with a single `apk add`, but it makes your container much fatter, i.e. every CI job that has to pull the image will be taking a longer time. Same for when you push the image into a far-away datacenter.

It's one of the reason that alpine should not be used outside of tests and toys projects. It cannot be substituted to an usual distribution.

In a base Alpine container, it would be awfully hard to build anything interesting. You are probably going to need to install some build tools first anyway. (Thus the two-stage container pattern.)

Python never works right for me, having to deal with yet another dependency just to use the build system is obnoxious.

Just use make. It's not hard and it works even for fairly large projects.

Speaking of using Python or Node.js as a language for a build process, there were so many times it failed when I trying to run the build script included in the github open source project I found interesting and downloaded to my local file system, and usually it's a project not written in my familiar language, and since I'm NOT familiar with Python nor node.js, so when the build script emit something like a dependency error, after several tries without success, I usually just give up.

This leaves the topological sorting of the tasks to the programmer? This is neat, but that's one very useful feature of make that's missing.

Yes, that's true. I'm still noodling on that, primarily to enable parallel builds.

Outside of parallel builds, however, I actually prefer an explicit sequence of steps to so-called "declarative syntax". The frequency with which I find bugs in the dependency graph of `Makefile`s [1][2][3] suggests that people are starting from a sequence of commands anyway. And it's better for the reader as well to see the commands laid out in order, when they need to debug breakage on their setup. People should stop pretending to be Vulcans and just write a script for building their project, and then sprinkle in `older_than` directives as an optional optimization. Building from scratch should be the case to optimize for readability, rather than some exceptional situation.

[1] https://github.com/zetavm/zetavm/commit/4e5e2e9d8b

[2] https://github.com/tekknolagi/stackx/commit/5999202423

[3] Fun fact: when you try to build OpenBSD's userland from source, the top-level Makefile always unconditionally calls `make clean`. I've seen this pattern a few times: large projects find themselves up shit creek in the build system, and it's difficult enough to test and debug the dependency graph that they give up all performance and retreat to just recompiling everything, just to be safe.

> The frequency with which I find bugs in the dependency graph of `Makefile`s [1][2][3] suggests that people are starting from a sequence of commands anyway

This is why the more elaborate build systems recognize dependencies automatically. For Makefiles, usually some sub-Makefile is generated automatically and then included. For language-specific build systems like OPAM (OCaml) or Cargo (Rust), this is one of their core features: As a programmer, don't care about maintaining dependency definitions! Whenever a source file is changed, the build system recalculates the relevant part of the dependency graph as well.

How does this compare to e.g. @apenwarr's minimal `do`?

A true build system really needs to build out a DAG of dependencies to be efficient and fast. Compose this with https://github.com/ejholmes/walk and you're good.

Thus ninja https://ninja-build.org/

I'm also using (for now) something based on these time checks (also pure posix + local, but local is extremely portable (tests well on 7 shells), though mine's a little more involved (automatic build job timing, controlled concurrency, robust failure/cancellation with autodeletion, fancy logging, filterable implicit dependencies, dependency + counting). It scales well to about 1000 source files. Then the lag starts to feel noticable.

One of the benefits I see for this approach is the possibility to handle spaces in filenames (AFAIK very difficult with gnumake). Sadly the provided function does not accept spaces in filenames.

Somehow I find it slightly amusing that I see ./a.out in 2017.

"notabug.org"... interesting.

Anyway, how does this compare, features-wise, with GNU Make? For example with make you have `./configure` files, do I also have them here?

Configure scripts are part of autotools, not make.

Looking as the sh code, it appears that it will work without tab characters. https://stackoverflow.com/a/2131227/1144060

So there's that.

As it happens, I was moved to create this standalone repo showing off my current build setup after writing this comment: https://www.reddit.com/r/vim/comments/6l8z34/vim_betrayed_me...

So yes, eliminating tabs (or rather the execrable requirement of using tabs) was absolutely a prime goal.

Put this in ~/.editorconfig

    [Makefile]
    indent_style = tab

And install the plugin, which is available for most editors and IDEs.

http://editorconfig.org/

It's not a requirement - I give you: .RECIPEPREFIX

> If you prefer to prefix your recipes with a character other than tab, you can set the .RECIPEPREFIX variable to an alternate character

That said, how hard is it really to disable tab expansion in your editor? I have a block of configuration dedicated to the different indentation requirements of different programming and configuration languages. In more modern editors, I don't even have to do anything at all to edit Makefiles with tabs - they come pre-configured to do the right thing.

The whole goal of this approach is to get out of the vicious cycle of build files generating other build files, so no `configure`.

`configure` is not in itself a feature. What purpose does it fulfill for you? Then we can discuss how we may serve that purpose in some more lightweight manner.

I have the feeling your way of thinking is 'remove every abstraction so that we end up with a simpler system'. This works in some cases, but certainly not all of them. configure is one of them.

Why? Try removing a configure script for even a simple autotools project, and ensure that it still builds fine on all previously supported platforms by writing a cross platform Makefile. Good luck with that.

"I have the feeling your way of thinking is 'remove every abstraction so that we end up with a simpler system'."

You're certainly right about that:

http://akkartik.name/about

http://akkartik.name/post/libraries2

https://lobste.rs/s/to8wpr/configuration_files_are_canary_wa...

"This works in some cases, but certainly not all of them."

This too I'm totally willing to accept. I believe most abstractions are prematurely frozen and so poorly designed. But this makes me liable to err too far in the other direction.

"configure is one of them. Why? Try removing a configure script for even a simple autotools project, and ensure that it still builds fine on all previously supported platforms by writing a cross platform Makefile. Good luck with that."

Here we part ways. I think `configure` is one of the easiest cases for me to be sure of, and it's because I don't care about "all previously supported platforms". Many if not all of them are obsolete. Also, as rossy pointed out elsewhere in this thread (https://news.ycombinator.com/item?id=15045076), it's impossible to know today if there's a bug in autotools for some rare platform.

We live in a world very different from the one autotools were built for, and it's almost a monoculture at this point. All that crap is utterly unnecessary today. I don't care about building software on all possible platforms, all I care about is building it on this one platform in front of me right now.

The most frustrating thing about autotools is that it's unclear which platform each individual check is concerned with. I have a whole mess of rules from 20 years ago -- and zero rationale for any of the rules. That makes even the few rules that are worth having a net liability because I can't separate them from all the useless ones.

> For example with make you have `./configure` files, do I also have them here?

The concept of `./configure` is 100% orthogonal to Makefiles. You can have Makefiles without `./configure`, and you can have `./configure` that outputs something other than a Makefile. For example, I could easily imagine a `./configure` that prints a shell script like the one in the submission.

has anyone tried tup? http://gittup.org/tup/

> The build script is easy to repurpose to your needs

No, it's not. This isn't under a free or open source license. The lack of license means that only an idiot would use it for anything.

Reply on news.ycombinator.com