Hacker News Reader

Comment by koito17 | Hacker News Reader

The article fails to mention whether Scala can ensure code is deterministic and hermetic. Starlark code is deterministic and hermetic, but the article never mentions this. Unfortunately, Starlark does not have static typing, but I think types would make Starlark one of the best languages today for build configuration.

In the Clojure community, there was a huge push for "builds are programs". I somewhat agree with this assertion, but I also think "one should restrict the class of programs a build belongs to". Neither Clojure nor Scala, compared to Starlark, seem to offer a way to ensure builds belong to a deterministic subset of programs.

Thus I am still wondering "why Scala?". I have never used Scala, but reading this whole article gives me the impression that Mill is the Scala equivalent of Clojure's tools.build. That is not what I would want in a build system.

View on HN

What is special in starlark that makes it hermetic and deterministic?

Scala is deterministic.

If you call functions that have side effects and nondeterministic behavior you can fall outside the comforts of determinism. But you can stumble upon library functions someone wrote in Starlark that accidentally put you there as well.

The Starlark homepage says

> Hermetic execution - Execution cannot access the file system, network, system clock. It is safe to execute untrusted code.

But the last time I wrote Starlark it was to define build targets in Basel. And executing the builds definitely accessed my file system and the network, otherwise builds would have no results.

Bazel enforces a hermetic sandbox for Starlark to operate within.

All input files have to be declared, and the build process can only see files that are declared.

Starlark cannot access arbitrary files at runtime, and it deliberately has no APIs for things like system time or random number generation, or global state.

Scala, on the other hand, has no such restrictions. As much as I love Scala, I think it’s an odd choice for a pure, deterministic system. Although perhaps if you use Scala to build a DSL (an area where Scala shines) you could engineer a pure functional sandbox within Scala.

Do you only declare the file names, or also file content and owner and group and everything up front?

Sure it deliberately has a bunch of restrictions in its base form. But in order to use it for anything you have to write custom actions or download and execute custom actions others wrote. And this will mutate your file system and run arbitrary executables.

As someone who worked a bit with Bazel actions and uses Scala: I think Bazel cuts down on the area where arbitrary I/O happens. Also, many executables come from pre-defined targets (e.g. java toolchains). One way to deal with I/O is if Mill injected its own implementation of a JVM filesystem/network layer, which I'm pretty sure it can do.

Input and output files are tracked determistically via content hashing. It cannot reach outside of its sandbox and touch the network or actual filesystem. Those are all (generally pinned) inputs to the system. For a given version of a script and a tuple of input content hashes, you get the exact same output files every time. No way to accidentally iterate over a hashmap, embed a timestamp, or leave the filesystem in a weird state because you were interrupted.

> Scala is deterministic.

I don't see how Scala is any more deterministic than any other language.

That’s what I’m saying. About Starlark.

Starlark does not expose sources of randomness like date.now()

If you're deliberately using that to prove Scala doesn't work for config than sure

It’s about guarantees vs eternal vigilance.

Bazel splits the build into multiple phases. Starlark only comes into play in the first two, load and analysis. During these phases, Starlark code doesn't have access to the filesystem, except in a few very limited cases like using the glob() function to expand a wildcard to a list of source files. Furthermore, it only generates an abstract graph of build actions. The Bazel engine is responsible for executing this graph in later stages, which might result in non-hermetic things happening but usually not.

Starlark has intentionally limited functionality such as lacking Turing completeness or global variables. This provides guarantees that it can be executed in parallel and will have a finite runtime.

Reply on news.ycombinator.com