Piper: A proposal for a graphy pipe-based build system | Hacker News Reader

Piper: A proposal for a graphy pipe-based build system

mattsanetra.uk

70 pointsmattsan28 comments

>> Isn’t that just Make?

> Yep! But what if we add mappings and collections?

These are called pattern rules in make:

https://www.gnu.org/software/make/manual/html_node/Pattern-R...

This is the best resource I've found on make to get a low friction build system up and running. Some parts are a bit dense if you're relatively new to make or used to someone else setting it up for you, but if you have the gnu make manual side-by-side for reference you should be good to go:

https://make.mad-scientist.net/papers/advanced-auto-dependen...

If you've ever been frustrated with how time consuming it is to manually write make rules then the above method will probably be a night-and-day difference from what you're used to.

I'd also strongly suggest either disabling builtin implicit rules with -r or reading up on all of them here:

https://www.gnu.org/software/make/manual/html_node/Catalogue...

Great resources, thank you! I think I came across some of these ideas before and it might have been as simple as me resenting the syntax. I'll touch up on my `make` knowledge and try to make Piper more ergonomic!

Please don't do that! Good reasons to borrow ideas from make would be familiarity for those coming from make, but definitely not because make is ergonomic. The entire reason you must put in tons of mental effort to grok this one true method of make usage I linked above is specifically because of how non-ergonomic make is out of the box.

My comment was meant to be practical advice considering make's popularity. A while back I wanted to replace a legacy script that generated Makefiles for a project at work, and I started out by making something very similar to Piper in python so it was easy for coworkers, etc, but ended up finding the guide I mentioned in my above comment while doing gcc dependency generation. Since I was targeting make underneath anyways, and my needs were less complex than I thought, I ended up scrapping my redundant python build tool.

Oops I worded it badly! I meant take some theory from Make and modify it to be more ergonomic before adding it into Piper. A la rethinking it all with new knowledge I gain.

If you haven't already, I'd seriously suggest having a look at redo (the theory but also apenwarr's implementation at https://github.com/apenwarr/redo).

It's not going in the same direction you are exactly but building a mental model of it in my head was salutary in terms of thinking through the purpose and implementation of build tools and hopefully it'll be similarly useful to you.

An idea:

If piper is clearer for you, it can "compile" down into make, and then let gmake do the heavy lift, at least for some of the early prototyping.

Better to target Ninja, no? Since it’s basically designed for that.

Could be, I haven't used it.

There are so many build systems that honestly... it is crazy to me.

(grabbing geezer stick) When I grew up we only had make... and we liked it! /s

Implicit rules and pattern rules have the little issue with header dependencies that I can never get over with. It always feels like I can only choose simplicity or accuracy in incremental builds. I suppose -MMD is the way for C, though the way it pulls in system header checks always feels a bit too far.

I think this is neat! Love the cute lil' characters, and I believe that simplified build systems are still fertile ground. Filtering and deriving from collections will be essential for larger projects.

> $_IN: [...] For collections, this is a space separated list of filenames > [...] > I haven’t written any software for this

Bit early for this sort of nit-picking, but that will need to be either a.) an escaped and quoted list or b.) an argv array that doesn't undergo word-splitting.

Bonus consideration: What if the user's shell isn't sh? Perhaps it's out of scope (ie. this shouldn't support every use case of Make), but for what it's worth, they support all sorts of recipe interpreters.

Thank you!

Re. a), b); I will probably add a "spread operator" syntax or something for collections. Maybe different symbols for the different types. Once I start prototyping I'll choose whatever is the least burden on the user (both in learning, comprehension, and in practice).

Re. different shells, I have yet to experiment - simple as! When I write the code I'll spot the challenges. Obviously I'd like it to support everything, but I have no clue as to what the implications would be.

The hard part of build systems aren't map/reduce operations, but dependency discovery as you go - because suddenly, you're rebuilding your transform pipeline while you're executing it. That's where the fun dragons are. (I really think they're fun. But type 2 fun ;)

amin vadhat wrote a little paper while in grad school about sniffing the workload while you were building it by hand and pooping out a makefile based on what you had done. seems like if you ran a makefile, recorded what the compiler looked at, you'd be in good shape wrt dependencies without having to parse the course or use other tools

There were similar tools to watch commercial installations and reproducing them outside of the proprietary tools.

Also Red Hat labs had something which watched an installation including the users commands including sudo for later replay. If I find it's name I'll edit or reply to this.

That technology was available in the 1990s.

Clearcase ran on a file system that recorded what files the build using clearmake looked at so your makefile just needed to know how to make a .o from a ,c file and all the dependencies were calculated. The build was on a server so you could get the results of other peoples (or automated) builds if they used the same dependencies.

We stopped using that as the filesystem had to be on a server and so the builds were slower as involved network access and we did not have large repos where this might have been offset by a large number of build artifacts. (plus we did more on Windows)

Plus let's not forget you needed roughly one fulltime Clearcase admin for every 10 engineers. It was a good idea with horrible execution :)

Don't think it was that high but yes needed admin.

I believe this is Amin's paper you refer to: https://www.usenix.org/legacy/publications/library/proceedin...

And for shorter reading, there's also djb's 'redo' thing: http://cr.yp.to/redo.html - specifically, the section of "Target files depend on nonexistent files" is an interesting one if you have tools that are allowed to generate dependencies for source files that aren't part of the source files.

(It turns out, if you follow that road, that your build tools and all their libraries are implicit dependencies in the graph as well. Second law of build graphs: You will at some point question if it should be legal to do unspeakable things to people who rely on pseudo-randomness in their build tools)

It is a conceptually simple problem which turns out to be surprisingly hairy in practice. That's why it's fun, and so many folks noodle on it :)

If you're interested in the field, I know Google has talked about theirs: https://google-engtools.blogspot.com/2011/06/build-in-cloud-... - and I'm sure many other companies as well.

Being on the Bazel team, and having previously worked closely with the Piper team, this naming would be very confusing for me!

That said: good to put your thoughts out there before going too deep into the implementation rabbit hole. Good luck with your idea. Build systems are fascinating.

https://www.project-piper.io/ I was confused as well

https://github.com/rhasspy/piper I was confused as well

Also a mouse config tool: https://github.com/libratbag/piper

There's this concept of "forward build system" that other people too have nibbled at. See:

* http://neilmitchell.blogspot.com/2021/09/small-project-build...

* https://dl.acm.org/doi/abs/10.1145/3497775.3503687

Primarily trying to reduce the high learning curve of complete build systems like bazel/please/buck.

The author too seems to be heading in this direction

Also https://taskfile.dev/ seems to be in a similar vein. Simplifying the authoring of the build specification.

I am working on a similar tool started from a filmmaking need to source-control large binary files.

The current (semi-implemented) design is that each file or collection of files can be assigned a step in a pipeline and version controlled separately, and steps can be defined as dependencies of one another.

This makes it possible to add "compute" steps that do not track any file but instead run a task, achieving a similar goal.

I think this will be the eventual goal with a simple syntax.

> Incremental compilation is trivial due to each mapping having a single input and output, with the output file name being known before it is generated.

If the build system only knows about a single input (the .c file), how do you deal with rebuilding when header files have changed?

Very good point! For my next amendment(s) to the idea, I will be attempting to think of a simple way to do M-N. It should naturally also cover 1-1, 1-N, and N-1.

Reply on news.ycombinator.com