The Starlark Programming Language
starlark-lang.org
starlark-lang.org
On one hand, having a "hermetic" subset of Python is nice. You can be sure your Bazel starlark codebase isn't going to be making network calls or reading files or shelling out to subprocesses and all that. The fact that it is hermetic does help make things reproducible and deterministic, and enables paralleization and caching and other things. Everyone already knows Python syntax, and its certainly nicer than the templated-bash-in-templated-yaml files common elsewhere in the build tooling/infra space
On the other hand, a large Starlark codebase is a large Python codebase, and large Python codebases are imperative, untyped, and can get messy even without all the things mentioned above. Even though your Starlark is pure and deterministic, it still easily ends up a rats nest of sphagetti. Starlark goes the extra mile to be non-turing-complete, but that doesn't mean it's performant or easy to understand. And starlark's minimalism is also a curse as it lacks many features that help you manage large Python codebases such as PEP484 type annotations, which also means IDEs also cannot provide much help since they rely on types to understand the code
For https://mill-build.org we went the opposite route: not enforcing purity, but using a language with strong types and a strong functional bent to it. So far it's been working out OK, but it remains to be seen how well it scales to ever larger and more complex build setups
So increasingly, this objection is meaningless, because nobody is using "untyped" that way anymore. The way in which people do use the term, Python is only "optionally" typed, and a lot of real-world Python code is "untyped".
Obviously lihaoyi was referring to static/dynamic when they wrote untyped (as made clear by the reference to type annotations) but kstrauser is objecting to using the term "untyped" since that can be interpreted to mean weak typing as well, which Python is not.
$0.02 anyway.
Typed versus untyped is, on the other hand, a rigorously defined academic distinction, and one that very clearly places pre-type-hints Python in the untyped category. That's not a bad thing—untyped isn't inherently a derogatory term—but because untyped languages have fallen out of vogue there's a huge effort to rebrand them.
What's wrong with universally understood and well defined concepts of "statically" and "dynamically" typed languages?
I have no problem with people using the other terminology in casual usage—I do so myself more often than not. I do have a problem with people pedantically correcting usage that is actually more correct than their preferred usage. I dislike pedantry in general, but I especially dislike incorrect pedantry.
Python is strongly typed (hard to escape the bounds of the type system) but (traditionally) dynamically typed (types are checked at runtime).
C is weakly typed (easy to escape the type system), but statically typed (types are checked at compile time).
> However, there is no precise technical definition of what the terms mean and different authors disagree about the implied meaning of the terms and the relative rankings of the "strength" of the type systems of mainstream programming languages. For this reason, writers who wish to write unambiguously about type systems often eschew the terms "strong typing" and "weak typing" in favor of specific expressions such as "type safety".
Nor does it make sense to conflate "typed and untyped" with "statically typed and dynamically typed". These are simply very different things. Julia is an example of a dynamically typed language with a quite sophisticated type system and pervasive use of type annotations, it would be insane to call it untyped. Typescript is an example of a dynamic language which is nonetheless statically typed: because type errors in Typescript prevent the program from compiling, they're part of the static analysis of the program, not part of its dynamic runtime.
The fact that it's uncommon to use untyped languages now is not a good reason to start describing certain type systems as 'untyped'! A good term for a language like annotation-free Python is unityped: it definitely has a (dynamic) type system, but the type of all variables and parameters is "any". Using this term involves typing one extra letter, and the payoff is you get to make a correct statement rather than one which is wrong. I think that's a worthwhile tradeoff.
> A type system is a tractable syntactic method for proving the absence of certain program behaviors by classifying phrases according to the kinds of values they compute.
And later on:
> A type system can be regarded as calculating a kind of static approximation to the runtime behaviors of the terms in a program. ... Terms like "dynamically typed" are arguably misnomers and should probably be replaced by "dynamically checked," but the usage is standard.
The definitions you're using are the ones that he identifies as "arguably misnomers" but "standard". That is, they're fine as colloquial definitions but they are not the ones used in academic works. Academically speaking, a type system is a method of statically approximating the behavior of a computer program in order to rule out certain classes of behavior. Dynamic checks do not count.
As I've said elsewhere, I don't have a problem with people using the colloquial definitions. I do have a problem with people actively correcting someone who's using the more correct academic definitions. We should have both sets in our lexicons and be understanding when someone uses one versus the other.
The Wikipedia article on type systems is well sourced and fairly well written:
https://en.wikipedia.org/wiki/Type_system
It has this section:
https://en.wikipedia.org/wiki/Type_system#Dynamic_type_check...
Here's a pull quote:
> Dynamic type checking is the process of verifying the type safety of a program at runtime. Implementations of dynamically type-checked languages generally associate each runtime object with a type tag (i.e., a reference to a type) containing its type information.
This accords with ordinary usage in the profession, agreeing with the person you're disagreeing with. You should read it. It provides no support at all for the argument that Python is untyped in any sense. It quotes Pierce several times, including the first citation, so it isn't ignorance on the editor's part.
You are advancing the argument that type theory, as you understand it (there being many paper-publishing respectable academics who do not agree with you on this), trumps type systems in practice, but other than insistence you give no reason why anyone should agree with this premise. Computer scientists aren't discovering laws of nature, nor are they writing proscriptive regulations as to how engineering should be accomplished or discussed. They have their own world and vocabulary to go along with it, but they do not have standing (as you do not) to dictate how terms should be used by others.
I don't accept that argument. If someone wants to make a narrower statement like "according to Pierce you could call Python untyped" I might question its relevance but I'd let it pass. Without such qualification, if someone says Python is untyped I will laugh at them and say "what's it doing when it throws a TypeError then, offering an opinion?". Or on a message board just reply that, no, it's dynamically typed, or unityped if you must, but untyped means something else. Which, indeed, it does.
And the objects inside an expression assuredly care about their types:
>>> '1' + 1
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: can only concatenate str (not "int") to strStrong/weak is not a dichotomy. It's a spectrum. That's why folks argue over where a language lands in the spectrum. OTOH, static (compile-time) vs dynamic (run-time) is a dichotomy. There's not really any in between. It's clear when and where typing occurs. So there's nothing to argue over.
> Typed versus untyped is, on the other hand, a rigorously defined academic distinction
A typed language is one that has a type system. Python has a type system. It's typed.
Colloquially, yes, python has a type system. All I'm saying is it's unhelpful to correct someone for using the more correct definition rather than the colloquial one. Both definitions are valid, but if we're going to be pedantic we should at least use the academic definition for our pedantry.
And you're correct, I should have said spectrum, but the point is still the same: even Wikipedia refuses to define "strongly" or "weakly" typed, suggesting people use terminology that isn't hopelessly muddled.
[0] Here's one: https://news.ycombinator.com/item?id=42368689
I guess the typing would be for the size of the integer that you work with. For example, x86_64 assembly has different prefixes to indicate what part of a larger register you are using: 8 (lower), 8 (upper), 16 bit, 32 bit, and 64 bit itself.
There are other "typed" operations, such as branching for unsigned vs. signed integers (think JA vs JG), or SAR vs SHR (signed arithmetic shift vs. unsigned arithmetic shift—one preserved the division logic of shifting for signed integers by repeating the MSB instead of adding zeroes when shifting).
While I'm not too familiar with them (but have been meaning to learn more for years!!), SIMD instructions probably also have similar ideas of having different types for sizes of arrays.
What makes changing the meaning of "untyped" extra confusing is that dynamically typed programming languages often have types as 1st class objects, and they get used all the time for practical everyday programming. Calling these languages "untyped" is just wrong on the face of it -- they're full of types.
Just to be clear, it's the dynamically typed languages that changed the meaning of untyped. OP's usage is closer to the original and to the current usage of the terminology in the study of programming languages.
Types and Programming Languages, one of the best regarded texts on types, has this helpful explanation:
> A type system can be regarded as calculating a kind of static approximation to the runtime behaviors of the terms in a program. ... Terms like "dynamically typed" are arguably misnomers and should probably be replaced by "dynamically checked," but the usage is standard.
In other words both are standard, but that's because the meaning of "types" has changed over time from its original sense and when it comes to the formal study of programming languages we still use the original terminology.
In the time of the original use, there were only static types. Languages had very little in terms of UDT's. Even a struct in C was barely a type of its own. I don't recall the details, but there was something about struct member names not being local to the struct. Interpreted languages didn't have records or classes at all(*), and certainly not types as first class objects.
We cannot really talk about how dynamically typed languages with rich type systems were originally labelled, back when they didn't exist at all.
(*) I'm looking forward to someone pointing out an interesting counterexample.
> A type system is a tractable syntactic method for proving the absence of certain program behaviors by classifying phrases according to the kinds of values they compute.
And later on:
> A type system can be regarded as calculating a kind of static approximation to the runtime behaviors of the terms in a program. ... Terms like "dynamically typed" are arguably misnomers and should probably be replaced by "dynamically checked," but the usage is standard.
In other words, you're both correct in your definitions depending on who you're talking to, but if we're going to get pedantic (which you seem to be) OP is slightly more correct.
Personally, it feels like dynamically typed language advocates have been getting more and more vocal about their language of choice being "typed" as static typing has grown in popularity in recent years. This seems like misdirected energy—static typing advocates know what they're advocating for and know that dynamically typed languages don't fill their need. You're not accomplishing much by trying to force them to use inclusive language.
Rather than trying to push Python as a typed language it seems like it would be more effective to show why dynamic checks have value.
It was an old discussion before then, even. It has nothing to do with advocacy and it's certainly not recent. It's about accuracy so that people stop hearing and then repeating the same incorrect ideas. There's no common definition of types by which Python is untyped, as though it doesn't have types at all when in fact every Python object has a type.
You mean besides the one used by every programming languages researcher and hobbyist? Sure, you can define "common" to exclude them, but I would give at least some credence to the definitions put forward by the teams of people who invent type theory.
As I've said here and elsewhere, I have no problem with people casually using "dynamically typed" as a term—I do so as well. But there's no cause to correct someone for using the more correct terminology.
If hearing it makes you feel defensive of python, that implies that you perceive "untyped" as a pejorative that needs defending against. In that case, your efforts would be better spent correcting the evolving consensus that (statically) typed is better than they would be spent trying to shout people down for using the academic definitions of typed and untyped.
There is also a semi humorously named "stringly typed" which means weakly typed in such a way that incompatible types are promoted to strings before being operated on.
I'm not aware of any static weakly typed language, but it's logically possible to have one
Starlark has been great to build with. You get the readability of having a simple subset of python, without python's dependency management challenges. It is easily extensible with plugin APIs. Concurrent API performance is great without the python async challenges.
One challenge wrt using Starlark as an general purpose embedded scripting language is that it does not support usual error handling features. There are no exceptions and no multi value return for error values, all errors result in an abort. This works for a config language, where a fail-fast behavior is good. But for a general purpose script, you need more fine grained error handling. Since I am using Starlark for API logic only, I came up with a user definable error handling behavior. This uses thread locals to keep track of error state [2], which might not work for more general purpose scripting use cases.
[1] https://github.com/claceio/clace
[2] https://clace.io/docs/plugins/overview/#automatic-error-hand...
I wish the starklark team had addressed it at this point.
load("@.../result", result=result)
def throw(arg):
return 1/0
if result.Result(throw).map(arg).is_ok:
# proceed
else:
fail("...")A lot of the complaints about Starlark as a programming language, and the proposed alternatives, seem to me to miss out on the UX advantages of having pythonic scripting (which so many folks who have taken a random "coding" class understand intuitively) whereas, e.g., using a lisp or lua would not. Further, having a language and runtime designed for safe use is absolutely critical, and trying to embed another runtime (js/wasm) and manage to lock it down successfully, is a much larger undertaking than I think folks realize.
For what it’s worth both Deno and workerd from Cloudflare give you starting points to run JS in a very locked down sandbox.
This brings it to the point. I'm still wondering why the achievements of software engineering of the past fifty years, like modularization and static type checking had apparently so little influence on build systems. I implemented https://github.com/rochus-keller/BUSY for this reason, but it seems to require a greater cultural shift than most developers are already willing to make.
I'm curious, what kind of type system does BUSY use?
So in the end, you have to fight the half-assed small language that was created and have to find a way to connect to some real language to get things done.
A textbook example for you are (proper) regular expressions. This little language guarantees O(n) matching. The Python and Perl communities added backtracking without truly understanding why backtracking was missing in the first place. Now their misnamed "regular expressions" cause security issues for their users.
Also, in Starlark, any runtime check you write is a build-time check and calling fail reports a build-time error, which is good enough for users, but not for understanding how to use Starlark functions within Starlark code.
Since starlark and bazel restrict the amount of "weird" things you can do, type-inference is pretty straightforward (moreso than in regular python), since almost everything is either a struct or a basic type and there isn't any of the common magic.
Gradle has its problems, and I often curse it for various reasons, but I'm pretty glad it uses regular languages that I can reuse in non-build system contexts. And the fact that it just bites the bullet and treats build systems as special programs with all the same support that it gives to the programs it's building does have its advantages, even if the results can get quite complex.
You can of course get people who are just bad at software and make things over-complex for no reason, but if you have such people on a team then the actual software you're building will be your primary problem, not your build system.
But if you can't get such people fired, the more tasks you can safely assign to such people the better, thus the advantage of a build system without a Turing-complete language :)
That's funny: I've been using bazel (and previously blaze) for well over a decade, but it has never once occurred to me to think of starlark as having anything at all to do with Python! I can't see anything about it which is distinctively pythonic.
To my eyes, starlark bears more resemblance to YAML, or TOML, or any other generic configuration language, than to Python.
If you open other Starlark files that have functions (in Bazel, that would be in .bzl files), you should recognize the Python syntax (e.g. `def` statements + space indentation).
def fizz_buzz(n):
"""Print Fizz Buzz numbers from 1 to n."""
for i in range(1, n + 1):
s = ""
if i % 3 == 0:
s += "Fizz"
if i % 5 == 0:
s += "Buzz"
print(s if s else i)
fizz_buzz(20)
https://github.com/facebook/buck2/tree/main/starlark-rust/vs...
The worst Starlark code I've read has been written not by SREs, but by the Boq team as they have a fetish for accomplishing complicated configuration at build time. This was one of the reasons I've avoided Boq: an incomplete code base that's under development doesn't even begin to build, which is far worse than building something and seeing a real compiler error.
It's pretty easy to add types to Python nowadays. I'd consider it bad practice not to do so in a large project.
For example, knowing the return type of a function is Union[DataFrame,Series] rather than simply DataFrame would save a lot of bad errors.
But I write Python for some time now, and I know what you mean. I have nightmares about codebases with dynamically generated class fields for example (though I heard ruby is even worse)
I find the starlark language is very simple (though inconsistent between the various implementations in bazel, go, and rust) but it takes a bit to understand how the magic between defining rules and implementations works in bazel. and TBH, that is also one place I've really needed auto-completion/static typing in starlark to understand what I can/cannot do.
I'm all for using the right tool for the job, and I'll acknowledge the benefits of hermeticity and parallelism in many contexts. But, I also think that you need to weigh the cost of adding another language to a project. I don't imagine myself using this anywhere outside of a build system, and even then I would first try to take the declarative rules of that build system as far as I can. It's probably better than shell scripts, though, at least most of the time? May depend on your team's prior familiarity with shell and/or love of Python.
There are three implementations (Go, Rust, Java) which could be a good thing if the language doesn't change often. They would have to be kept in sync for any changes, as well as keep from drifting due to changes in the host language's semantics.
I also think that the closeness in syntax with Python can be a disadvantage, especially if users are expecting more recent Python additions like the walrus operator or pattern matching to be available.
I like to design languages, too, so these comments come from a place of love and understanding. I think it would help if there were some more specific justification (or examples) of why this is better than something more established (e.g. Lua or MicroPython) or more distinct (e.g. pure data in configs instead of embedded code). I do like that the language attempts to remain as simple as needed.
There are so many people at Google just goofing around, lol. People just doing hobby stuff for $350k/year.
There is no justification to implement this custom dialect of Python for a build system that drives everyone crazy 3 times. Reminds me of when it was revealed that 400 people were working on Fuschia — a hobby OS that only shipped on a single smart home device.
An increase in edit/build/run cycle efficiency of thousands of employees justifies an investment of a few engineers.
They could have invented a new bespoke DSL. Instead they choose to stick to a well known and familiar language and just limit its expressiveness to a subset that would be easier to optimize. I think that's quite reasonable
I think a new bespoke DSL would have been a non-starter since so many build scripts had already been written by the time Skylark was being conceived.
(This is also why I don't trust configuration languages built by people who _didn't_ observe the years of pain. Cue and jsonnet are notable projects that were able to incorporate a lot of lessons.)
With regards to Fuchsia… I used to think building a new OS from the ground up was madness. Now I think _not_ doing that eventually is madness. I'd be a lot happier if Zircon and Fuchsia were done in Rust though…
I do think for an OS/Kernel, it's worth having everything in a memory-safe language, and possibly worth formal verification too, if the very core of it is small enough…
What do you consider the be the justification for three separate implementations of the same build config language? Genuine question. I am not doubting the need for the DSL itself.
If you're looking to embed a scripting language in a Go program, having a embedded language implementation written in Java isn't very useful. And vice versa.
Carry on, Google
While I'm not sure you are in the same crowd, I always think it's interesting that the HN hivemind tends to be upset with the browser monoculture, but doesn't bat an eye at OS monoculture. It feels like you're really just channeling feelings about the company rather than the projects.
An example application is https://github.com/shac-project/shac
Within Google, there a large number of similar tools which are written in go and harness the starlark language. While there are plenty of other options, I will say I think starlark is often a great choice.
Also I don't think pure data is an option. The point is that you want to generate data, which has many benefits (avoid duplication, easier to test).
It seems that Starlark is a good trade-off given the constraints. My main grief is that it doesn't have type.
I agree that sometimes pure data isn't an option, and I've had to write some Skylark to assist blaze build rules too, but every time it also added tech debt, reduced the number of people on the team who completely understood the build system, and wasn't convenient/practical to test the build extension. The problem I'm referring to above is when the use of a source-controlled data file would have been sufficient but someone had to write a build extension because it's fun or something new to do (or whatever reason seemed convincing at the time).
Then there are people who think that those same data files shouldn't be part of the build process at all and should be part of system turn-up/tear-down, stored in a global config or DB where versioning is alongside the data instead of alongside the program build. Certain kinds of migration are made more difficult if everything is built into the binary. Of course, that strategy comes with its own caveats.
It could make a of sense as a "contract language", or as intended: part of a build system.
It's a bespoke Python subset... functions but no recursion, dicts but no sets, etc.
It's also been adopted by the latest version of Buck (Meta's Bazel analog).
I use it daily.
It's an option for lightweight embeddable scripting language, with implementations in Java and Go. If you want Python familiarity, consider it as an option.
The Logic of CUE is a great read: https://cuelang.org/docs/concept/the-logic-of-cue/
I even built a codemod library that does a very basic python -> starlark so that one can develop using python ecosystem libraries and just copy & paste into a secure execution environment. It was a huge success at my last company.
I'm very thankful to Laurent Le-brun and Alan Donovan -- both of whom are exceptional engineers that I learned so much from. I thought I was skilled but both of those individuals are just on another level.
Starlark Language - https://news.ycombinator.com/item?id=40700549 - June 2024 (49 comments)
An Overview of the Starlark Language - https://news.ycombinator.com/item?id=40573689 - June 2024 (49 comments)
(The) Starlark Language - https://news.ycombinator.com/item?id=39457410 - Feb 2024 (1 comment)
RepoKitteh: Github workflow automation using Starlark - https://news.ycombinator.com/item?id=26674781 - April 2021 (7 comments)
Anyway, I guess "see also:" https://news.ycombinator.com/item?id=42370744
For those purposes, it is simple. The simplicity reminds me of Lua. It is a perfect choice for the device.
(if anyone has ideas in the area, you can reach out to me)
- https://github.com/caketop/python-starlark-go
- https://github.com/inducer/starlark-pyo3
(I haven't tried them myself)
What’s this all about? Don’t most languages?
So that means you can control the APIs, and say opendir() closedir() in Unix returns filenames in different orders. Depending on what the data structure in the kernel is
So many programs in other languages aren't deterministic just because they use APIs that aren't deterministic
(IIRC, I don't believe Bazel actually has fully deterministic builds yet, though.)
id("ab") # not deterministic
hash("ab") # not deterministic
def foo(): pass
str(foo) # not deterministic
Another sneaky one is the `is` operator in Python, where the Python documentation says:
> Due to automatic garbage-collection, free lists, and the dynamic nature of descriptors, you may notice seemingly unusual behaviour in certain uses of the `is` operator
Related to that is the `__del__` method: when exactly is it called?
It's quite easy to get non-deterministic code in Python and in many languages. And of course, there are lots of non-deterministic functions in the standard library (Starlark doesn't provide them).