Terra: Low-level system programming language meta-programmed from Lua

In case anyone is still reading this, I'm here now. Still happy to answer any questions.

I'll also add that we're always looking for new contributors. We're an 100% community run project, so everything is driven by people who care about things stepping in to do them. If you're interested in compiler technology, there's a wide variety of things to work on.

Please note: The entire "windows" portion of that article has largely been resolved at this point, because I fixed it myself. Some of the more egregious documentation problems have been fixed, but others remain unresolved.

Unfortunately, we have since discovered even more insane behavior: https://github.com/terralang/terra/issues/460

I used terra at least 6 years ago to create a JIT compiler for database queries.

The language itself is performant and pleasant, but there are not enough people using it, so troubleshooting is really painful.

I develop Regent, a programming language for distributed computing based on Terra [1]. We're one of the larger (largest?) Terra codebases and indirect source of users.

(I also help maintain Terra, but this answer is more focused on my capacity as a user.)

Terra is a workhorse and it gets the job done. Overall, compared to a previous iteration of Regent (that was built entirely from the ground up in Python), it's been a good experience. Having first-class code generation capabilities has been really nice, especially with built-in support for things like parsing C header files, vectorization, and CUDA code generation. The language is stable and generally doesn't have many surprises. Unlike, say, C++ or Rust, there aren't as any corners to hide odd behaviors. (Obviously, there still is some room for such things, as you can see in the sibling comments. But I still think it's better than most languages.)

The main issue is, as siblings have noted, the size of the community. While most stuff works, if you find something broken you're likely going to have to fix it yourself. This is an area where we're always looking to grow.

[1]: http://regent-lang.org/

We're working on 1.0. The language is basically stable but needs a little polish. As usual everything depends on what time people have to put in. Always looking for more help!

Terra macros are hygenic in the sense that they follow the Lua scoping rules. So if you do:

    local add1 = macro(function()
      return `x+1
    end)
    terra()
      var x = 12
      add1()
    end

Then this will give you an error because x is not in scope in the add1 macro.

Instead you'd have to write:

    local add1 = macro(function(y)
      return `y+1
    end)
    terra()
      var x = 12
      add1(x)
    end

Note that I changed the variable name to clarify that only the lexical scope matters (i.e., "x" is a symbol passed to the function as argument y).

You should be able to create towers of languages in Terra. I'm not sure if anyone really does more than 2 levels in practice, but most of the common Terra DSLs are basically written as code generators on top of Terra language expressions.

See e.g. the Regent code generator. Here's an example of some nested metaprogramming---not with a nested language or macros, but with an escape that calls a Lua function to generate more code based on the type of the argument:

https://github.com/StanfordLegion/legion/blob/0cf9ddd60c227c...

I think it is more like this: the Lua code emits assembly code or LLVM intermediate representation or C source code, then arranges for the assembly code to be assembled (then possibly arranges for the code to be executed, then terminates).

Many years ago, someone wrote a Haskell library along the same lines that lets the library's user mimic assembly language with Haskell code. One thing I particularly liked about this library is that (by deviating a little from the standard way of indenting Haskell code) the user could mimic the 2 columns of assembly language. E.g., in

  foo <-  statement
          statement2
  bar <-  statement3

"foo" and "bar" would be a labels, and that mimics the look of assembly language in that the leftmost 15 or so characters (columns) in an assembly-language file are usually reserved for labels. (I.e., if a line has no label, those 15 characters are blank.)

You can interleave Terra and Lua execution. Terra functions are JITted if you call them from Lua. You can call Lua from Terra too, though the language makes you jump through some hoops to encourage code to avoid Lua dependencies in Terra code. So "execution" is more like you'd expect from standard Lua (or Python or whatever), it's just that functions can be compiled if you want.

Of course Terra functions can also be saved out to a file too, and if you do that they're free of any dependency on Lua (i.e., no need to like liblua*.so).

When you use a template language, how do you JIT the code so you can run it in the same process?

How do you parse C/C++ header files so you can introspect their types or functions and take actions based on the results?

How do you write clean abstractions for code generation? Do you build your own C++ AST abstraction entirely from scratch? If not what do you use, and how much power does it expose (and what tradeoffs come with the level of power that you get)?

Terra lets you do all of these things in a clean way with well-defined, predictable semantics that don't leave you scratching your head when you get to 4 nested levels of macros and complex dynamic behavior that depends on program inputs.

Different languages help because the different parts of the program expose different needs.

You want the generated code to be low-level, because you want it to be fast, free of dependencies (interpreter, JIT, GC, etc.), and you care less about abstractions. You want the metaprogram to be high-level because you want nice abstractions, and it runs at compile-time anyway so you care less about performance.

Sure, people are working on languages that can do both, but it's easier to make it work with two different languages that can be tuned specifically to their niches.

Well Rust procedural macros can run arbitrary code, so that's not such a big difference. I'd say the larger one is having access to actual AST data structures available if you want them. Rust forces you to parse the token stream yourself and serialize it again. Also, Terra allows user defined keywords so it's a lot easier to create nice looking DSLs (which is my personal use case).

You can think of it as clang, but controlled by a lua engine instead of the CLI. It's not just macros, you have control over generation of build artifacts. By default you're running in JIT mode, but you can at any time bundle up some procedures and spit out an object file or link them into an executable.

You can also call back into Lua from the native code, and use it to generate more native code. You get back some of the cool self-modifying executable techniques of the days gone by, without having to completely YOLO it.

We use LLVM, so hypothetically yes. I don't think anyone has tried though.

Maybe this

https://pypi.org/project/TMPPy/