Luau – A fast, small, safe, gradually typed scripting language derived from Lua
roblox.github.io
roblox.github.io
We've been growing fast, and are looking for strong C++, systems, and application engineers. As a highly vertically integrate gaming platform, we've got a lot of hard problems to solve - from programming language design (see OP!) to debugger tooling.
You can email me at my username plus roblox.com.
> It’s similar to Lua interpreter in that it’s written in C, but it’s highly tuned to yield efficient assembly when compiled with Clang and latest versions of MSVC. On some workloads it can match the performance of LuaJIT interpreter which is written in highly specialized assembly.
One thing I've been experimenting for my own project, is using tailcalls. All main compilers (gcc, Clang, MSVC) are able to compile that into actual tail calls (i.e. `jmp`, not `ret`, so that the stack doesn't overflow) (with optimizations enabled), even for indirect calls (i.e. dispatching depending on the next instruction). Function parameters (VM state) are assigned to registers, alleviating Mike Pall's main complaint about writing interpreters in a high-level language, that the compiler cannot optimize the huge loop as well as a human can. I need to test this on a full interpreter with 50 or so instructions, to see if the optimizations still apply.
Passing VM state via arguments is suboptimal since argument registers are often clobbered and one would normally use callee-saved registers. I have developed a patch for llvm allowing to specify registers used for passing arguments and making more registers available for allocation without spilling.
I intend to reimplement LuaJIT VM in C with IR augmentation to verify the performance.
I'm also big fan of the LuaJIT compiler explorer you made https://luajit.me
https://llvm.org/docs/LangRef.html#calling-conventions
My understanding is, for "fast path" instructions (e.g. integer addition) there's no register spilling necessary, so passing VM state in registers makes sense. For more complex instructions, you rely on LLVM to optimize registers as it would normally, with callee-saved registers. Of course, one must still pass as little state as possible - e.g. just sp, cp, maybe top of stack / heap, and then a pointer to *vm_state for all the rest.
GHC allocates registers for arguments from the fixed list where most but not all registers are callee-save in other common CC-s. This is a big improvement; still there are valid reasons to want it customisable.
E.g. in LuaJIT next instruction is decoded as a part of an instruction handler before dispatching to the next handler. Instruction arguments are passed to a handler; they shouldn't use callee-saved registers.
Another complication is the stack frame. There's a requirement to keep the stack pointer aligned (16 byte on x86_64). Call instruction pushes the return address and the pointer becomes misaligned. If the called function wishes to call further functions it has to adjust the stack pointer to make it aligned, even if it doesn't use the stack. The stack pointer is adjusted in a function prologue, hence there's a slowdown even if a nested function call is on a cold path.
Whether these micro-optimisations are worthwhile remains to be seen.
In case of GHC, the functions could return, but the fast-path is that most functions access VM state, and many functions have very little need for registers; therefore, forcing registers be caller-saved would just cause most functions to push and pop the stack without any reason. If non-standard calling convention (e.g. C calling convention) functions are called only rarely, then using caller- or callee-saved registers doesn't really matter.
1: https://gist.github.com/zeux/bb646a63c02ff2828117092036d2d17...
The performant variant is to squeeze it to have room for other code, keep important registers asis, and jit the bytecode array into sequences of static calls. They next best variant is to call the next op statically. Indirect func tables are hard to predict by the CPU.
They explicitly removed tailcall for better user experience, as it messes up debugging and stack traces. Maybe it could be added back in perf mode.
[0] https://mediastream.cern.ch/MediaArchive/Video/Public2/weble...
1/ Code source is not available, so it feels like the purpose of the page is just to brag about how good your interpreter is.
2/ s/publically/publicly/
Seems like this runtime was announced to Roblox developers almost a year ago. But i don't think they've put much thought or effort into open sourcing it and it just happend to pop up in peoples feeds now since Arseny published an article(#1) about his 8 years working for Roblox 2 days ago and somebody probably took the link from that article (Quite an interesting read if you're into gamedev).
I maintain the Haxe Lua target, and have a good understanding of how Lua relates to other languages (at least through a "Haxe" lens).
I think the main things I get hung up on is the 1-indexing scheme, and the lack of true "null" fields on tables (Lua treats these as missing). This makes Lua tricky to integrate with other languages, especially if they serialize/de-serialize data. You essentially have to pick a work-around, and unless you know what you're doing (and what you're doing 5 years from now), you're going to feel a lot of pain.
I realize changing these things would make Lua less "Lua", but virtually every other language Haxe supports has straightforward support for these behaviors. It's really the only two nits that have marred an otherwise pleasant experience working with Lua.
I'm personally hoping tl or something will take off as the de-facto TypeScript for Lua. I'm already convinced that types are better. I'd use types all the time if I could. It's just that not a whole lot of people seem to care about Lua, even with how fast LuaJIT is, so you have to compromise with projects adding types to Lua that are only in their infancy, assuming they'll still be around after a couple of years. TypeScript has proven that gradual typing can be done right and had a bunch of innovations I never would have thought of (the keyof operator for creating a type of "the set possible key names of T" for instance). Having that kind of power in Lua on top of one of the fastest JIT compilers on Earth is my dream.
Also Moon+ (even more expression version of MoonScript) is considering implementing the teal checker natively: https://github.com/pigpigyyy/MoonPlus/issues/15
Jokes aside, and not to boast about what I did, honestly, but Lua is (in hindsight - I wish I realized before) full of gotchas which are definitely not easy to implement (just one word which still makes my stomach hurt: COROUTINES).
I honestly think there are simpler (and better) languages to target. With hindsight, experience and pain, of course.
This may be a place where “technically correct is the best correct” doesn’t hold, at least in terms of community psychology.
I have a pretty small data set (5 people) and three of them are guile users.
Scheme is however willfully underspecified, whereas most other languages lack specifications of even a proper base language (except for code). Implementing scheme is pretty easy apart from continuations and macro hygiene, but those parts are very well specified somat least you know what you are aiming for.
https://github.com/teal-language/tl
It implements types on top of Lua. In a single (middle-sized) Lua file. And it is open source.
Also look at the developer site [2], where you can find documentation, tutorials, and an awesome, active community in the developer forum [3].
[2] https://developer.roblox.com
[3] https://devforum.roblox.com
(disclaimer: I'm an engineer on the Roblox Studio team)
Thanks for the links, so far from what I can see Roblox Studio is pretty good!
2020 dev conference talks are now avail https://developer.roblox.com/en-us/resources/rdc
I first learned programming due to Roblox. It's really easy to pick up, and you can make almost whatever type of game you want. I honestly believe that highschool programming classes should universally use it for introducing how to code rather than Javascript or Unity.
Roblox has tons of innovation. It has a volumetric/voxel based lighting system. It has networked rigid body dynamics built right in. It lets you take your avatar between each game, keeping your look. And you can late-join any game -- you don't need to be there "at the start."
Some games have tens of thousands of moving pieces, yet load reasonably quickly. Check out the tornado in Natural Disaster Survival for a good example!
That would teach the value of iteration, failing forward and how to deal with imperfection at a very early age.
Of course reality is not a vacuum and there would be a lot of noise though :)
https://github.com/teal-language/tl/blob/master/docs/about.m...
I actually hadn't heard of Teal! I don't think it existed when we started.
Judging from the Teal documentation, I'd venture that it looks pretty similar to Luau in everyday use.
If I were to guess, I'd venture that the biggest differences are probably the kind of type inference we do (Luau's inference engine draws inspiration in equal parts from OCaml and TypeScript), a bunch of Roblox-specific features, and that our inference engine is all C++ for performance.
I've been meaning to try out roblox for a while and lately I've been getting pretty hyped about teal (for context, the guy behind it is the creator of luarocks and htop, among others) so I might see the differences up close sooner rather than later :D
EDIT: (unrelated) Already not looking forward to people asking "Lua" questions about typing and constantly having to explain that Lua is not just a roblox thing xD
The real difference tho is age, Luau has been around since 2006 or so (used to be known as Rbx.Lua).
Many other game-focused scripting languages use reference counting instead of tracing GC to eliminate pauses at the expense of throughput. Would that help in your situation?
Here's an older tweet with some examples (I'm not the one tweeting but I made the optimizations):
It will be interesting to see whether they make any major changes to Lua's garbage collector, which is an incremental tracing GC.
Several gaming-focused scripting languages use reference counting instead of tracing GC, even though throughput is lower. This is because RC (almost?) eliminates the need for stop-the-world pauses, making it easier to maintain a consistent frame rate.
I don't know anything about it, but just looking at it, it seems like a giant mess. 1-based indexes. Semicolons required here, but forbidden there. Like... why?
cons:
- 1-based indexing
- you basically always want a local but you have to ask for one, it's a bit noisy. 'strict mode' is easily applied to the global environment, and catches most outright errors here.
- there is an adequate amount of library and extension code, but less than you'd expect coming from Ruby or Python. The lack of a blessed object system sometimes means taking more effort to understand what other people's code does.
pros:
- variadic functions and multiple return values make for a truly dynamic language
- first class functions which are fast and work well: everything is a closure, a whole file is just an anonymous function called a chunk
- the table as a single compound data structure works very well, and metatables are a unique and excellent solution to extending behavior. Array portions are required to be compact (non-nil from 1 to #tab), which can be worked around on the few occasions when it's inappropriate.
- asymmetric coroutines.
- first class environments, which are just, yep, tables. this allows for a few really nice things.
- fast. LuaJIT is very fast. Both Lua runtimes fit in a typical L1 cache, and it uses a register VM, which is a good fit for real processors vs. stacks. No global interpreter lock, all C level Lua functions receive a Lua_state as the first argument and are fully reentrant.
- in general the emphasis on minimalism means the whole language "fits in head", and the tools provided are more than adequate to get jobs done.
- the LuaJIT FFI, which parses C declarations directly, is simply a pleasure to use, and is the only one of which I'm aware where you pay an absolutely minimal penalty to use it. There's just nothing like it. More languages need to have this.
miscellaneous:
Semicolon insertion is simply a non-issue, unlike JS it was implemented properly. I have well under one semicolon per file, less than one per project, even. It's very, very occasionally nice to write more than one statement per line; that's what the semicolon does.
It's a nice language.
If you are using a word "dynamic" as "once-good-but-now-increasingly-considered-worse-and-worse", I agree. Lua is definitely too dynamic to be safely used.
Weeks ago I had to debug a Lua code looked like this:
commands = commands_template:gsub("<arguments>", arguments:gsub(" ", "\\ "))
[1] https://github.com/devcat-studio/kailua/blob/323caab59c06230...
I tend to use lpeg for any complex string manipulation. gsub has bitten me once or twice by not thinking through whether something is a pattern nor not, sure.
In some sort of deploy scripts, converting a command "template" into the actual command by replacing a fixed string "<arguments>" into the escaped arguments (I know, this is not sufficient but also not the exact code, I've simplified it for the easier inspection). Now guess what's the problem without looking at the manual.
Also, don't say "use lpeg". Unless Lua supports a proper package system for every circumstance it's not a solution. Note that this deploy script runs in an embedded context, so no to luarocks and kins.
Still can't quite understand why they would choose the 1-based indexing, it's just such an unforced error, unless the intention was literally just to fuck with people.
I suppose I could get over being fucked with, though.
> Both Lua runtimes fit in a typical L1 cache
That is pretty sweet!
Civilians, and data scientists, tend to count from one. Julia uses 1-based indexing for the same reason.
But yeah, it was a mistake. No good way to fix it, alas.
Also, semicolons forbidden? I don't think there's anywhere you're disallowed from using them to separate statements, though they're largely optional / redundant with newlines.
Put another way, if I don’t play Roblox is there a reason for me to care?
It's Lu-au. It's like a party in your development environment!