Hacker News Reader

Comment by zxv | Hacker News Reader

An 8 line Makefile translates to an 80 line common workflow (.cwl) file.

https://github.com/common-workflow-language/workflows/tree/m...

Honestly the format looks a bit over-engineered IMHO. The task is quite simple: make build system that executes jobs on clusters. So why not get best build configuration format practices and just make them run over network? For example ninja build system [1] format is quite good in my opinion, so just make runtime execute commands over network. Or travis-ci [2] is another example of well designed configuration format, and it really enables developers to write small and powerful configurations. Sure it was even done before (though mostly for C/C++ stuff), like IncrediBuild [3] for example, or FASTBuild [4] or distcc [5]. Though the case with precise control of pipes could be improved in current build systems, but not sure how important it is for this application.

- [1] https://ninja-build.org/ - [2] https://travis-ci.org/ - [3] https://www.incredibuild.com/ - [4] http://fastbuild.org/ - [5] https://github.com/distcc/distcc

Haven't checked ninja, but I've blogged a bit on limitations in common build systems, such as make and its various derivatives:

"The problem with make for scientific workflows":

http://bionics.it/posts/the-problem-with-make-for-scientific...

"Workflow tool makers: Allow defining data flow, not just task dependencies"

http://bionics.it/posts/workflows-dataflow-not-task-deps

The last of which is a limitation of even the most of the "very much-engineered" ones, as the post goes on to explain.

From the first blog post:

> Files are represented by strings

I think it's especially true for make - looks like it was designed to efficiently express operations for transformations of the same type (like .cpp -> .o/.obj). So in different use case it may become a bit clumsy to use. Ninja should help a bit in this case - you can define a rule, and just use rule name when defining inputs and outputs of a build statement, though it still operates on files.

>[Problems with] combinatorial dependencies

Yes, partially this could be fixed with wildcards in make. Ninja doesn't have wildcard support, so I've created the buildfox [1] to fix it :)

>Non-representable dependency structures

I think it's a limitation of this type of build systems, their configuration language oriented on expressing "how" to achieve things, not "what" to achieve.

- [1] https://github.com/beardsvibe/buildfox/

Ouch. I suppose that's the price paid for portability, though.

Anything beyond that trivial example Makefile will rather the reflect the system and environment on which it was written.

Reply on news.ycombinator.com