Hacker News Reader

Comment by Rochus | Hacker News Reader

> a large Starlark codebase is a large Python codebase, and large Python codebases are imperative, untyped, and can get messy even without all the things mentioned above. Even though your Starlark is pure and deterministic, it still easily ends up a rats nest of sphagetti

This brings it to the point. I'm still wondering why the achievements of software engineering of the past fifty years, like modularization and static type checking had apparently so little influence on build systems. I implemented https://github.com/rochus-keller/BUSY for this reason, but it seems to require a greater cultural shift than most developers are already willing to make.

View on HN

It did have influence. Take a look at Gradle, which is widely used in the JVM space. It uses a general, strongly typed language (Kotlin) to configure it and it has a very sophisticated plugin and modules system for the build system itself, not just for the apps it's building.

Gradle has its problems, and I often curse it for various reasons, but I'm pretty glad it uses regular languages that I can reuse in non-build system contexts. And the fact that it just bites the bullet and treats build systems as special programs with all the same support that it gives to the programs it's building does have its advantages, even if the results can get quite complex.

Interesting. Using a regular language has some advantages, but also many disadvantages. One of the intentions of BUSY was - similar to e.g. Meson - to avoid a fully Turing complete language, because then people start to implement complex things, thus leaving the declarative character of a build specification, which again makes the build more difficult to understand and maintain.

The basic assumption behind Gradle, I think, is that people usually implement complex things in build systems because their needs are genuinely complex. Build systems are at heart parallel task execution and caching engines that auto-generate a CLI based on script-like programs, and that's a very useful thing. No surprise people use them to automate all kinds of things. You can lean into that or you can try to stop people using them in that way. Gradle leans in to it and then tries to make the resulting mess somewhat optimizable and tractable.

You can of course get people who are just bad at software and make things over-complex for no reason, but if you have such people on a team then the actual software you're building will be your primary problem, not your build system.

>if you have such people on a team then the actual software you're building will be your primary problem, not your build system.

But if you can't get such people fired, the more tasks you can safely assign to such people the better, thus the advantage of a build system without a Turing-complete language :)

When did it get support for Kotlin? I thought that was comparatively recent and that Groovy was the DSL for forever before that.

2016, https://blog.gradle.org/kotlin-meets-gradle

Build systems are sort of like type expressions, templates, or constant expressions in a programming language. Either a program compiles or it doesn’t. What might happen when you change the code in some unlikely way isn’t immediately relevant to whether the program works now, so it’s easy to skimp on that kind of checking until things get out of hand due to greater scale.

Also, in Starlark, any runtime check you write is a build-time check and calling fail reports a build-time error, which is good enough for users, but not for understanding how to use Starlark functions within Starlark code.

There is also the need to understand a build and to navigate a build system. Try e.g. to understand how the Chromium build works, and which options are enabled in which case. I even built a tool (see https://github.com/rochus-keller/GnTools) to analyze it (and some other large GN projects) but even so reached the limits of a dynamic specification language pretty quickly. This won't happen in BUSY.

Yes, gn is less good than bazel for a variety of reasons, not the least of which is tooling like `blaze query --output=build` and the more restricted evaluation model in starlark which is easier to evaluate.

Since starlark and bazel restrict the amount of "weird" things you can do, type-inference is pretty straightforward (moreso than in regular python), since almost everything is either a struct or a basic type and there isn't any of the common magic.

I think it's a cultural thing. People like to think of a language for the build system as a little language that somehow doesn't "deserve" a type system. And even they do think a type system is necessary, they think such a language doesn't "deserve" a complicated type system (say Java-like with subtyping and generics) which makes that type system less useful.

I'm curious, what kind of type system does BUSY use?

It's a rather traditional type system; the specification is here: http://software.rochus-keller.ch/busy_spec.html. The main advantage are the combination of modularization, types and formal declarations, so that if you make a change in a large build (such as my https://github.com/rochus-keller/LeanQt or https://github.com/rochus-keller/LeanCreator systems) incompatibilites are immediately found by the compiler. Without these features you can never be sure whether all effects were checked.

Which tool did you use to create that busy_spec.html file? They remind me of Engelbart's blue numbering system for documents, if I remember the name correctly.

It's https://github.com/rochus-keller/crossline/, a tool which I implemented and used for many years in my projects. It's inspired by Netmanage Ecco and implements features which can also be found in Ted Nelson's Xanadu or in Ivar Jacobson's Objectory.

I wonder why people love to create languages to be embed in applications when there are plenty of languages that are already useful and well known.

So in the end, you have to fight the half-assed small language that was created and have to find a way to connect to some real language to get things done.

Small languages, if they are suitably constrained, offer far more reasoning power and optimisation potential. This is why we need more small languages, not less. Python aims for maximum flexibility and maximum ease of use. This comes with real and serious trade offs. Python programs are very very difficult to reason about, for both people and machines.

A textbook example for you are (proper) regular expressions. This little language guarantees O(n) matching. The Python and Perl communities added backtracking without truly understanding why backtracking was missing in the first place. Now their misnamed "regular expressions" cause security issues for their users.

Even Thompson didn't use the linear time algorithm that's named after him in Ed and Grep. The Python and Perl implementations were inspired by Henry Spencer's regex, which was in turn reimplementing Thompson's backtracking implementations.

Are you sure it was Thompson who added backtracking to grep? Note also that the POSIX standard intentionally omits backtracking regex. It is a shame that others have not deprecated them.

We're talking about backtracking and not back references, right? The original implementations of Ed and Grep in Unix did backtracking, not back references.

Oops I mean't back references, since that is the problem feature (that requires backtracking).

I don't think it's purely cultural. Starlark is interpreted, which presents some challenges to type checking. You either need to make the interpreter more complex, or have an out-of-band type checking step.

The tyranny of the one-small-change use case having outsized importance. Usually the build system is no one's job, which means that all hurdles grow.

Reply on news.ycombinator.com