We figured that it wasn't worth dealing with the hassle of unaligned addresses because the more portable alternatives worked just as well.
107 karma · joined September 21, 2018
The key difference is that Terra cannot directly manipulate Lua data. Although Terra's syntax is similar to Lua, its type systems and data structures are actually much closer to C. If we want to pass Lua table into Terra, we must first convert it into Terra arrays/structs. This cost adds up if the code uses many Lua data structures; in the worst case it can be worse than just using Lua for everything.
Passing data between Lua and Pallene should incur no cost. If the programmer rewrites a piece of code from Lua to Pallene, it should hopefully make things faster, and never slow it down.
Terra is more focused on metaprogramming. Generating code and executing things at compile-time. Think of C++ templates, but better.
Pallene beats C when the code uses many Lua data structures. Acessing Lua data from C via the Lua-C API has significant overhead that can erase the gains from rewriting into C. Also, rewriting from Lua to Pallene is much less work than rewriting it in C.
Although Pallene is only a subset of Lua, the idea is that you use it together with Lua. It's not meant to replace Lua entirely.
For compiled code, Pallene's performance can be comparable to LuaJIT[1]. Pallene might be better in code with unpredictable branches, which don't fit in a single trace. LuaJIT has the edge in purely-interpreted code, and can inline indirect or cross-module function calls. LuaJIT is also more featureful, and has a better FFI story at the moment.
[1] https://www.inf.puc-rio.br/~hgualandi/papers/Gualandi-2020-S...
LuaJIT is really good software, it is hard to beat it as its own game. Pallene tries to get its edge elsewhere: it tracks Lua 5.4, whereas LuaJIT diverged around 5.1 and will stay that way forever. Pallene's implementation is arguably more portable, because it compiles down to C and doesn't need hand-crafted assembly.
Pallene, on the other hand, is more focused on performance. The tradeoff is that it doesn't cover the entirety of the Lua language, just the features that we can generate fast code for. The intention is that you can write the performance-critical parts in Pallene and glue everything together using Lua (or Teal).
That said, I certainly wouldn't be surprised if we saw more cross pollination of ideas between these two languages in the future :)
https://gist.github.com/hugomg/73e91ebca7ced3c5c6f955e9e830b...
For a pure Lua comparison LuaJIT is typically going to beat Lua by a large margin. For code that can be JIT compiled you might see ~10x performance improvements and for code that doesn't you might still see around a ~2x improvement because the LuaJIT interpreter is written in hand-crafted assembly language (PUC-Lua sticks to portable and standards-compliant C90).
But things get a bit more complicated if you add foreign C code into the mix. Under LuaJIT, the JIT compiled code that uses the LuaJIT FFI interface is the fastest, but interpreted code using the FFI is slower than interpreted code using the traditional Lua-C interface, and can even be slower than PUC-Lua using the traditional Lua-C interface. This means that when you are using LuaJIT you need to be very careful to make sure that your code is the sort of code that can be JIT compiled. If you hit a "not yet implemented" feature you can have a big performance degradation.
------
Nevertheless, Lua 5.4 has brought significant performance improvements compared to Lua 5.3, specially in integer operations (including for loops). Another big change was that the `#` is now faster. For example, on my machine the following loop runs in 1.96 seconds in Lua 5.3, in 1.26 seconds under LuaJIT, and in 0.96 seconds in Lua 5.4
local a = {}
for i = 1, 1e7 do
a[#a +1] = i
end
print(#a)
I wrote the hexiom solver that got linked above and the motivation in that case was that I saw that someone tried to solve the puzzle with a hand-written backtracking search, but that their algorithm wasn't able to find a solution to one of the levels.
My version of the SAT solver managed to quickly find a solution for every case, including the highly symmetric one where the usual search algorithm got stuck. The nicest thing is that the implementation was very declarative, in that my job was to produce a set of constraints for the SAT solver, instead of to produce an imperative algorithm. The symmetry breaking was one example of something hat was very easy to do in a SAT-solver setting but which would be trickier to do by hand.
If you want to produce a standalone executable, at the end of the day this executable will contain a copy of the Lua VM.
If you want to produce a library that can be "require"-ed from Lua, then you will produce a ".so" shared object file with just the Pallene functions in it.
Pallene uses garbage collection, and shares the Lua garbage collector. The idea is to make as easy as possible to call Pallene from Lua and Lua from Pallene. Meanwhile, Terra only uses Lua at compilation time for metaprogramming and template building. At runtime, if you want to call Lua from Terra (or vice versa) you need to use the C API, just like you would do with C.
I wouldn't really try to compare the performance of Pallene with Terra. They are two very different languages, doing different things.
Roughly speaking, from the point of view of Lua, Pallene is a system language like C. It is good for performance and calling C system libraries, but isn't particularly suited for dynamic code loading and sandboxing via interpreter hooks.
Terra is designed for high performance numerical computing. As you mentioned, it has no garbage collector, uses C-like datatypes and its executables are completely detached from the Lua runtime.
Pallene, on the other hand, is all about being able to seamlessly interoperate with Lua at run-time. It is possible to directly manipulate Lua tables and functions from a Pallene program. From the point of view of Lua, Pallene modules can be loaded as a C extension module would, except that Pallene is designed to integrate even better with Lua (sharing its garbage collector and having better C API performance).
As the other commenter pointed out, we are currently compiling down to C. However, it is possible that we could switch to LLVM IR in the future.
If you don't mind, I'm curious about what applications you currently use Lua for.
For us, the main motivations for the Pallene approach are performance predictability and implementation complexity.
To get maximum performance with a JIT you need to write your code in a way that ensures that the control flow always stays inside the compiled parts of the code. If you are accidentally "too dynamic" then the JIT will fall back to the general bytecode interpreter, with an order of magnitude slowdown. The Pallene type system should ensure that if your code typechecks then it can be compiled to something reasonably efficient.
The type system also lets us use a simpler ahead-of-time compiler instead of a just-in-time one. JIT compilers are notoriously tricky to implement, which in turn means that it is more difficult to evolve them together with the language, or to port them to additional platforms.