N2: Alternative Ninja Implementation | Hacker News Reader

N2: Alternative Ninja Implementation

github.com

79 pointsingve30 comments

The author mentions it, and I highly recommend anybody interested in build systems read the seminal paper Build Systems A La Carte [1], which breaks down various build systems (including Excel!) into their component concepts, and along the way finds a new feature-space that hadn't been explored.

[1] https://www.microsoft.com/en-us/research/uploads/prod/2018/0...

(an understanding of Haskell is recommended to understand the implementations, but not required to understand the concepts)

Corresponding blog post: http://neugierig.org/software/blog/2022/03/n2.html

Nutshell summary of the post:

Instead of relying on modification-time comparisons between dependency and target output, use hashes of the relevant "ingredients" of building a target (cmdline, arguments, meta-data about input files etc.).

I’m a little confused by some of this (mtime vs manifest) because Ninja does rebuild things if things like the command line changes, doesn’t it? Or have I misunderstood how this tool works forever…

Yes, Ninja hashes the command line and compares that hash in addition to comparing the mtimes.

If someone wanted to reimplement ninja and add support for the GNU Make job server, I'd be sold.

https://github.com/ninja-build/ninja/issues/1139

At a certain scale (i.e. package management and not a mono build system), having a common job server is essential to scaling in a healthy way.

Aside: the make job server is very simple and UNIXy. There's a pipe shared between `make` and its subprocesses. `make` writes `N-1` bytes to the pipe. When a job wants to do parallel work, it reads a byte from the pipe. When it's done, it writes a byte back to the pipe.

(Both rustc and cargo support the make job server)

It's really an elegant design, which is nice, other than the way that information is passed into the child process (roughly: MAKEFLAGS=" -j --job-server=3,4"), which requires you to argparse out the fd numbers.

It would be nice if a new, less cumbersome, environment protocol could be adopted.

What happens when the job crashes?

I dunno if they'll accept it but I might actually submit a PR to add this to n2, because I also would very much like "ninja with jobserver" and it being written in rust it should be fairly easy to add since the code cargo uses for jobserver support is broken out into a very simple library.

Do you mean jobserver client or sever? As for "ninja with jobserver client" there's already https://github.com/Kitware/ninja

> Do you mean jobserver client or sever?

Both. But server is more important to me usually (ninja running cargo, in particular -- but also sometimes ninja running make or ninja running ninja). Technically you can obviously kinda trick client-only support into being server support by wrapping ninja in a jobserver like make, and letting that jobserver cascade down, but that's pretty gross.

So while client support is nice, it's not really sufficient. I actually don't think I have much use for client-without-server at all, the main configuration I need client for at all is ninja->ninja.

There's also that kitware's is a one-purpose fork that's just kind of a pain to install in CI because lack of packages, but it's pretty easy to `cargo install n2` on a machine where I'm building rust anyways, and cache the CARGO_HOME so it doesn't build every time.

To be honest, with the fact that compilers are no longer single-threaded almost as a rule, something like jobserver seems like an essential feature in a build system now. It's basically impossible to meaningfully limit concurrency without some level of coordination or fuzzy imperfect indicators like load average. I hope other compilers follow rust's lead here and implement client support eventually.

(and then hopefully a more robust common protocol can evolve out of it)

It's not explicitly stated anywhere, but this appears to be by the original author and one of the later maintainers of the original ninja, which probably puts it ahead of some of the other rewrites out there in terms of its likelihood of adoption.

Shameless plug, I also wrote my own Ninja implementation [1]. I tried to make it easy to understand, while supporting all the Ninja features I used.

In the end, it was able to compile my own kernel and a few other Ninja projects.

[1]: https://github.com/gkbrk/scripts/blob/master/ninja.py

To improve your shameless plug, you should tell us:

* What your motivation for writing it was.

* Why people might like to use it instead of ninja or n2.

and we should not forget about the simple C (C99) re-implementation of ninja from M. Forney:

https://github.com/michaelforney/samurai

What are the benefits of the C implementation? I noticed the adherence to POSIX command-line conventions, but that doesn't sound like a good enough reason for a reimplementation in a different language.

I guess the ability to work in really restricted/minimal environments where you only have a simple C compiler like tcc, and nothing like LLVM. Or paranoid environments where you can only use CompCert. Something like that.

But can't you just cross-compile?

I'm not talking about embedded, I'm talking about e.g. intentionally minimalist systems that have to be self-building, or systems that have to be built with a formally verified compiler (which does not exist for C++), or anything else like that.

Only tangentially related, I'm personally intrigued by redo, but have never had a good justification for trying it out.

It looks eminently reasonable to me, and Avery Pennarun plus DJB sounds like a majorly winning team to me.

https://redo.readthedocs.io/en/latest/

Redo has a conceptual cleanliness that other systems lack. But maybe it should emit a ninja script instead of building directly.

I love build systems and have tried many of them. Redo is fun to play with. It is really easy to understand.

There is some messy reality that causes redo the struggle. For instance it doesn't natively support a single command that produces multiple build targets. You end up needing workarounds. I also find that redo starts to feel like Rails where your logic is spread around is many little files. Each file is easy to understand, but it takes a while to build the full picture.

That's interesting. Thanks for the info!

I also created a minimal Ninja implementation in Rust some time ago. My goals were to implement it in terms of the Build Systems a la carte paper. Of course, hard to compete with the original ninja authors who obviously understand it much better. For example I used a separate lexer and environments, which got a little annoying and is something explicitly called out in their design.

https://github.com/nikhilm/ninja-rs

(n2 author here) This is really cool! I am sorry if weird ninja decisions (like not being lexer-friendly or phony being underspecified) caused you grief, the whole thing was really built as a quick hack before I understood the problem space.

None of these have a historical ETA estimator, which has been requested in ninja maybe a decade ago. I hacked a few lines to show something, but would prefer the correct solution.

(n2 author here) ETA is pretty hard in the presence of parallelization unless you make big assumptions like "all pending build tasks will be able to run in parallel". If you had any good ideas here I'd love to hear them.

I think perfect is the enemy of good regarding ETA. This Ninja PR comes pretty close to the best possible solution I can think of: https://github.com/ninja-build/ninja/pull/1963

Well, HN commenters will like this: Something rewritten in Rust without advertising that as its primary feature.

Reply on news.ycombinator.com