No Compiler – On LLVM, and writing software without a compiler

I'd like to know what is the downside of using LLVM. Why did they choose to write their own optimizer?

Anyway, the most difficult part of writing the compiler was probably the garbage collector.

Is LLVM that well supported on windows ? Go aims at running on as much platforms and it seems to me a lot of languages built with LLVM don't run on windows (Swift,Crystal,...).

There's an "official answer" in the FAQ [1] ("We considered using LLVM for gc but we felt it was too large and slow to meet our performance goals." Note: lowercase gc refers to Go compiler, whereas uppercase GC would refer to Garbage Collector). That said, I seem to recall other additional/more detailed reasons were given on various occasions too, including:

- ease of iteration and freedom to make huge breaking changes (and they did make use of it, see e.g. various generations of the GC, or the SSA work);

- thorough knowledge of the internals of the compiler they used (given they were it's authors; also this is linked to the reason I listed above);

- one of the explicit goals for the Go toolchain was that it must compile very fast (which I believe is also somewhat at odds with advanced optimizations, like those used in LLVM) - they even quipped something like "we designed Go while waiting for C++ to compile";

- additionally, though I'm not sure if it was ever spelled explicitly, I suppose they wouldn't want to get tied to the LLVM as a huge dependency, including having to adjust to each and every change in LLVM API, regardless if sensible or not (I seem to recall from so me articles that LLVM does introduce breaking changes in API from time to time?);

[1] https://golang.org/doc/faq#What_compiler_technology_is_used_...

Yes, there are no immediate plans for anything other than MIR at the imperative-semantic level (the compiler also uses HIR, which is mostly a desugared AST and not related to LLVM IR).

Currently, initial support for lowering MIR to LLVM IR is close to complete and it might be enabled in the next beta (or the one after that).

None of the analysis passes have started their transition to MIR, though (there is a well-typed check to catch bugs in MIR generation, but that's about it).

There's also talk of using MIR for compile-time (pure) function evaluation [1] and some out-of-tree experimentation with interpreting it [2].

[1] https://internals.rust-lang.org/t/mir-constant-evaluation/31...

[2] https://github.com/tsion/miri

Essentially it's an implementation of the concept that metaprogramming should use the same semantics and syntax as normal programming. This is the case in Lisp, for example, but Lisp is not (or not typically) compiled, so Lisp metaprogramming is essentially limited to defining syntactic sugar. In Terra, metaprogramming (or multi-stage programming, as they call it), can actually result in improved performance compared to a typical C-like implementation of the same algorithm.

The website has a couple papers that provide benchmarks for some interesting test cases.

Terra is lower-level, and on evaluation compiles to object code. Lua is higher-level, and on evaluation compiles to bytecode before being executed. A source file written entirely in Terra would produce a binary, and a source file written in Lua would produce Lua bytecode. In either case, you can run the result immediately, or save the resulting object code/bytecode to run later.

The strength of the two is that Terra can be manipulated using Lua as a metalanguage: for example, you could write a Lua function which produces Terra code, which is then executed. In some sense, this is like C, which has a lower-level language (C) and a corresponding high-level language which can manipulate that lower-level language (the C preprocessor). In this case, though, Lua is much more powerful and flexible than CPP, and Terra is also a smaller, simpler language than C.

It gets more powerful, though, because the Lua parts don't just have to exist at compile-time. You could use Lua at run-time to produce Terra code, which in turn can get executed immediately, producing object code specialized to a given situation or machine: for example, you might write a language, use Lua to parse it, and then convert it to Terra to produce object code which is then invoked immediately: this effectively just-in-time compiles your code. Or, you could write an algorithm in Terra with specific constants that aren't known until run-time, and then use Lua to stitch those constants in before running the algorithm: for example, to take advantage of specific machine features or parameters.

These examples also assume that you have only "compile" and "run" stages; you could also write programs with more stages: for example "compile source", "specialize for machine", and "run" could be three distinct stages. Terra/Lua would afford you the flexibility to stage your computations in this way.

The terminology seems to be tripping you up (and I agree these terms aren't super clear). I think it's easiest to understand this through examples. Say you wanted to create a specialized function in pure Lua:

    function make_adder_func(n)
      return function(x) return n + x end
    end

    local add3_func = make_adder_func(3)
    local result = add3_func(6)

    -- Prints "9"
    print(result)

This totally works: because Lua supports both closures and first-class functions, the parameter "3" is captured in the environment of the inner function, so we have a specialized function for adding 3 to any other number.

Now the way Lua will execute this isn't super optimized. It treats "n" as a local variable which could theoretically be modified (it's not here, but proving that takes more of an optimizer than most dynamic languages possess). Also Lua doesn't even statically know the type of "n", so the "+" operation could be string concatenation, or even a user-defined function for the __add metamethod if "n" was a custom type. So even though it looks like the add3_func() is pretty specialized, it's actually not specialized at all in practice.

(For the record, LuaJIT is smarter about this and does have a trace compiler + optimizer that makes this extremely efficient).

But what Terra gives us is the ability to generate fully specialized and optimized functions at runtime. In Terra we could write:

    function make_adder_func(n)
      return terra(x : int) return n + x end
    end

    local add3_func = make_adder_func(3)
    local result = add3_func(6)

    -- Prints "9"
    print(result)

Unlike the Lua function, the Terra function is strongly typed and does not contain a closure over its environment. The inner Terra function isn't referring to the Lua variable "n", it is taking the value of n and making it a constant inside the Terra function. This allows the Terra function to be more thoroughly optimized (more like what LuaJIT would do).

So getting back to your question, the phrase "typed Lua is being generated" is a little confusing. What it means is that these Terra functions are strongly typed, and we're using Lua code to programmatically generate Terra functions, like in make_adder_func() above.

I know less about this than the other people replying, but will chip in because I have a different perspective that you might find useful.

When I saw what this guy was doing, my first reaction was also "that's like Terra!"

Here's how developers work at the moment: You write some code. Then - you /fork a process/ that reads your code, and puts it into a pipeline to produce machine code or bytecode. It's a ceremony. There's all kind of obscure little technologies to know about - variables defined in your shell language, makefiles, the peculiar arguments to specify when you invoke your compiler.

This article and Terra show that there might be a simpler path than this available. In Terra, you describe a system of code using lua structures. (They could as easily have been s-expressions). Terra then abstracts those into their structural components, and passes that tree to LLVM. And an artifact comes out. This is cool, because now you can easily integrate automatic code generation /into your application layer/.

What is happening here is not revolutionary in its mechanism. But you get access to the whole thing in a way that I haven't seen before. I think this will significantly change the way we think about programming.

You can see this writer struggling with multiple issues. And he's used lua, which is supposed to be an easy platform to integrate with. And he's having problems. Still, once these issues are worked through, I'd expect to see a a hacker language emerge that is a thin wrapper around the LLVM API and allows you to dance close to the LLVM API.

I also had trouble understanding that from their description, but then at some point I think I got it. Is I understand, it's like if you used Lua (actually, a Lua-like language dubbed "Terra") in places where you'd normally use the C preprocessor, or C++ templates, or Lisp macros, or Rust macros, or inline assembly fragments in C/C++. While your "base language" that you'd use to write most of the code would be your regular old Lua. Does it make more sense when described this way?

edit: so, from reading the other comments around, I think the language roles are actually reversed vs. what I thought - i.e. Lua is the "macro language", and Terra is the "main language".

So here's yet another attempt at describing terra :)

Firstly, 'typed Lua' really meant the terra sub language because it looks like Lua with types.

Going back to terra, consider that 'terra' is a language that can be compiled by LLVM into machine code. So you can choose to either compile and emit object code, or compile and immediately run the generated object code. The other choice you have is to either write terra code by hand, or generate terra code from Lua code.

You say 'if typed Lua is being "generated" (i.e., it's the target) how come Lua is being used as a metaprogramming language'. Why is that not possible? Many languages have built in metaprogramming using the same language (notably lisp code can generate lisp code).

Another way to look at this is Lua is the metaprogramming and control language for generating, compiling and running Terra code.

C is not really portable assembly. It's full of undefined behaviour, for one.

It is typed at a very low level. It doesn't cover interesting data structures, like hashes, vtables, closures, etc.

I don't know where you got the impression that he got the impression that he was doing any of this with the intention of putting it on production.

I didn't get that impression. I was trying to say that I admire the author without also advocating putting a bunch of clever hacks into production.