Knit: Making a Better Make | Hacker News Reader

Knit: Making a Better Make

zyedidia.github.io

87 pointssigna1136 comments

I wonder if the author has read the paper "Build Systems A La Carte" [1], which defines a taxonomy of build systems (of which Excel is part!) and their fundamental concepts.

This:

> I have to use hacks like .PHONY to mark rules as virtual.

makes me think not (.PHONY is required because fundamentally Make uses the filesystem as the "Store", or cache, for what is built, and rules just define Store targets)

It's been 5 years since the paper, and I'm still hoping for a a production-ready project that leverages its lessons to create an agnostic, distributed build system.

Nevertheless, excellent work in Knit! I'm always happy to see interest in the build system space. I particularly appreciate the use of Lua as the scripting layer. Make's scripting layer is certainly... limited. Also, great to see consideration for generated dependencies (like extracted header files)!

[1] https://www.microsoft.com/en-us/research/uploads/prod/2018/0...

Oh and tangentially, because it's amusing: why does Make use tabs? Because the original Lex was troublesome and it was convenient in the prototype. Then, there were dozens of users and it was too late to change!

https://beebo.org/haycorn/2015-04-20_tabs-and-makefiles.html

Thanks! Perhaps I shouldn't have characterized phony rules as a hack, but just that I think it is cleaner to express that information directly in the rule rather than in a separate rule. The paper you link is great, and I should read it in more depth -- are there any particular lessons or insights from it that you think Knit or other build systems could benefit from?

It's been a while, but one of the more powerful insights I remember, in this era of compute farms, is abstracting out the Store to be cloud-aware (kinda like ccache, but integrated in the builder. I think Bazel does this?)

If you haven't seen it: https://github.com/rochus-keller/BUSY

> BUSY is a lean, statically typed, cross-platform, easily bootstrappable build system for GCC, CLANG and MSVC inspired by Google GN

It uses lua and config files that are mostly directories and filenames.

You should look at Buck2. One of the coauthors of that paper (Neil) is one of the lead developers, and unlike Buck1 the buck2 binary exposes a very tiny API that consumers then build off of, and define all their rules, using Starlark for everything. This means that your build rules and your code exist together, as they should. And it recently started supporting the remote build API that Bazel does, so distributed builds and caching already work as they should. It's a very exciting project in the space.

Looks OK but adoption will be limited by the pain felt by the users of packages by people that use it.

Ah, this package looks nice, I'll have a look. What's a Knitfile? Oh, that looks quite cool; oh, so I need to install knit of course. Prebuilt binaries from the internet? Ha ha, no, better to build it. Oh it's in Go, so I need to install Go? Ah, a different package that does the same thing ...

> Ah, this package looks nice, I'll have a look. What's a Knitfile? Oh, that looks quite cool; oh, so I need to install knit of course. Prebuilt binaries from the internet? Ha ha, no, better to build it. Oh it's in Go, so I need to install Go? Ah, a different package that does the same thing ...

This is a useful vignette, but I wanna pick at it a bit because at least sometimes, I don't see things going this way with me.

> oh, so I need to install knit of course. Prebuilt binaries from the internet? Ha ha, no, better to build it

Do you really see most devs thinking this? I don't, even though I'm someone who would build it myself.

> Oh it's in Go, so I need to install Go? Ah, a different package that does the same thing ...

I don't understand what the problem would be here. I maintain several out-of-tree Go packages for Nix for private use and all of them are trivial because Go packages are super easy to build. Are there really many devs out there who are so hesitant to build Go projects, of packages in all languages?

With C and autotools you may already have the toolchain installed, but it's generally harder to build than Go packages.

I try to use Make to bootstrap project-specific tooling when missing, by getting it to download (then verify) the tools, then delegate everything else to something like knit (or just/cargo/npm).

That way, there's very few prerequisites for a new dev to install (plus you have consistent build/test/run targets for all projects).

If someone wants to manage tools themselves (e.g, the no prebuilt binaries example), they can do that, then skip the Makefile entirely.

This will unfortunately be a pain point for any build system that isn't already entrenched in the ecosystem. I think Knit does as well as it can to mitigate this:

* There are fully static prebuilt binaries for a wide variety of platforms (including Windows, where I think it is actually more complicated to install Make than Knit).

* Knit can generate a shell script or Makefile for the build, so these can be used for users who don't want to do development but just want to build from source.

* It is easy to build Knit from source (thanks to Go). Yes you need a Go toolchain, but once you have that it is just `go install github.com/zyedidia/knit/cmd/knit@latest`. Of all the languages that could have been used, I think Go is one that makes the "build from source" experience very straightforward.

How does it compare to just?

https://github.com/casey/just

compare to just what?

what?

just what?

just

what?

Knit, like Make, is designed to provide incremental builds. In other words, a build that takes into account the graph of dependencies involved. Their goal is to only run build steps that need to be run. If a build target has already been built, and its dependencies have not changed, then the build rules associated with the target can be skipped.

Just, as far as I know, does not support incremental builds. Just is more intended to provide a set of project-specific scripts that are convenient to list and run. Make is sometimes used in that way with "phony" targets, but that is more of an evolved usage. Make's original purpose was to allow for defining builds in terms of dependency graphs.

the list of inconveniences is missing the biggest (and sadly, unchangeable) pain-point of Make: targets/deps/filenames cannot have spaces in them..

and, it cannot have multiple wildcard-patterns, like %xyz%, only single %xyzabc or abcxyz%, and this brings much complicatedness into picture..

Knit supports regex rule patterns so I assume the latter issue is resolved in knit in one form or another.

I haven't run into these pain points much with Make, but Knit does support spaces in targets/prerequisites (use single or double quotes to surround the name). For the second point, Knit has regex rules (a Plan9 Mk feature) that can allow multiple grouped matches. The example from the article is:

    $ ($name-(.*)-(.*)):RB:
        GOOS=$match2 GOARCH=$match3 go build -o $match1

which is a metarule that matches targets like `$name-linux-amd64` and can be used for easy cross compilation with Go. Of course since it allows full regular expressions, even more complicated rules are allowed.

This project has an impressive scope, especially for a personal project. I think there is a strong demand for a small-scale build system like make, but without its footguns. I will be keeping an eye on this.

I’m in the unenviable position having to study a build system in Make for a very large codebase. Are there existing tools that generate the dependency graph or aid in understanding such a build system?

Might be worth a look at remake, it can help a ton when debugging complex Make setups: https://github.com/rocky/remake

Very helpful. Thanks so much!

This definitely fixes a few "micro" problems with Make, but I think most people use Make these days for the same reason they use Bash - not because it's good, but because it's probably already installed.

If you're going to require installing another tool why not use one that fixes the "macro" problems too? Notably hermetic builds and dynamic dependencies.

E.g. Ninja has dynamic dependencies (IIRC). Bazel et al, and possibly Landlock Make have hermetic builds.

The recently announced Buck 2 has both.

I think it makes sense to combine this with premake (https://premake.github.io/)

Premake is also Lua based and while it does slightly different thing (generates VS,Xcode,gmake solutions) it may benefit from having direct build path - without creating make files and intermediate solutions.

IMO, Rake was the other killer app for Ruby, but Rails sucked all the oxygen out of the room. It's highly underrated as a both a generic build system and a Make replacement.

Nice project. Is it ready for prime time? Btw, are there any other alternatives to be compared with it and Make?

I've seen a few, but the only one I can remember off the top of my head is just[0].

[0]: https://github.com/casey/just

I use just and it is amazing.

Yup! Two well-established alternatives are "rake", in the Ruby community, and "just" in the Rust community.

Rake is fully programmable in Ruby. Just is a bit less flexible, but it doesn't require learning Ruby, and it's quite pleasant to use.

https://ruby.github.io/rake/

Also, many people using rake already are often biased by how Rails uses it, but it's much more than that, and can easily be used as a make replacement, complete with freshness checks and rules.

There was an article specifically avout that on the interwebs but I can't find it anymore. These ones cover some bits as well, although they're a tad heavier on using Ruby code.

https://martinfowler.com/articles/rake.html

http://www.throwtheswitch.org/build/rake

It should be noted that just specifically does something make was not designed for but is often conscripted into: it’s not a build tool, it’s a command runner.

So when you run a command it runs the dependencies then the commands, in full, every time.

I very much prefer that usually, because either the underlying tooling handles checking if all’s right or there is no such checking to do and it would be a problem.

But if you expect just to build your C project and handle not touching stuff which doesn’t need to be, you’ll be disappointed.

The dependency resolution is core to https://gittup.org/tup/ which had been adopted a bit at the time it came out but since faded back into obscurity.

This looks cool, how have I never seen it mentioned before

There is build2

https://build2.org/

I've also written my own alternative to make called Rabs (https://github.com/wrapl/rabs) which uses an imperative language to specify the dependency graph as well as the build instructions. It's designed for really fast incremental builds.

Pretty happy with ccache, though of course it doesn't scale to very large projects.

I'm pleased to see the inclusion of hash-based cache invalidation!

ah, what a missed opportunity for calling this build system katana. or shuriken. it would have continued the analogy that started with ninja and samurai :)

Reply on news.ycombinator.com