RaptorJIT 1.0: Lua implementation for high-perf low-level system programming | Hacker News Reader

RaptorJIT 1.0: Lua implementation for high-perf low-level system programming

github.com

156 pointsingve31 comments

An interesting part of this project, other than improving the already excellent LuaJIT is the Studio[1] tooling (made with Pharo smalltalk[2] and nix[3]) that has been built around debugging JIT performance issues.

Luke Gorrie, one of the main contributes has been live streaming its progress.[4]

[1] https://github.com/studio/studio/

[2] https://pharo.org/

[3] https://nixos.org/nix/

[4] https://www.youtube.com/watch?v=MQyxXSPXcwg

The level of experience, talent and expertise required to dive into LuaJIT development and following Mike Pall's stepping down from active development of it was cause for concern that this would be the end of the road (some even decided to abandon Lua as a result). My optimism grew when I saw this project (and the quotes at the end of it's github page, which capture the issue perfectly) - very glad to see this release and hope the momentum just gets bigger.

That's pretty much how I felt as well. I've come across this some weeks ago, and I'm still not sure if this will be a worthy LuaJIT successor.

I'm very happy to see this. For those who are not familiar with Luke Gorrie, he has done many interesting things whilst tightly orbiting around languages that lack popular appeal.

Erlang, Common Lisp, Forth and now Lua and Smalltalk, it takes a certain kind of thinking to go against the grain like that. Besides following his output for years and being inspired by him, I've always admired Luke for blazing his own path in the programming wilderness.

So great to see this ... The slow roll of LuaJIT, made me despair a bit that Lua would ever see a thriving JIT project. Hopefully this will be it...

If you haven't tried Lua, do. You can be up and running in about 3-4 hours, writing useful things. Try writing your next utility script or command line tool in Lua.

I've looked at luajit, terra, and now raptorjit; and I always come to the same conclusions.

1) this is the wrong optimization (more below)

2) Rio's runtime is very good (enough for now, at least)

3) the next big step like this is to make an interpreter in Rust, not necessarily a parser/ast generator, but just a runtime, and I'm not in a rush for this

the right optimization is to do low-level work in a systems language.. in my case Rust, where rLua has a great wrapper for puc Lua. only use Lua as a scripting language to call those lower level functions.

right now on github, you can see my Lua bindings to Rust's libraries for web server, web client, libsodium functions, jinja rendering, string transforms, set operations, and more. all of that work is done in Rust, and controlled by Lua. it's faster than any Lua you've ever seen.

https://github.com/foundpatterns/torchbear

I don't agree with this at all. First of all, something feels off when you talk about right and wrong optimization without taking context into account. RaptorJIT grew out of the Snabb switch community. With that in mind, Rust+Lua isn't anywhere near pure Lua in terms of development speed, minimalism, simplicity and flexibility. Luke Gorrie has come out of the Lisp community so I have to imagine that these criteria are very important to him.

Systems language doesn't really mean anything by itself, so you sound very dogmatic when you talk about "right" here. The Lisp machines were implemented in Lisp. Is Lisp a systems language?

> Rust+Lua isn't anywhere near pure Lua in terms of development speed, minimalism, simplicity and flexibility

I hope you're wrong about this one. I guess I might have some trouble gaining stake holders from even Lua supporters who view it as a best language for every layer. But Rust has an incredible amount to offer Lua, especially when used primarily as a higher-level scripting language, for glue code.

Except rlua goes out of its way to look like anything but the Lua C API. So you've immediately lost all productivity for people who know that abstraction well. Additionally, anyone whose spent any significant amount of time in that space knows of the amateur alternatives, and this looks no different.

You clearly don't understand terra then. Terra is a system language that mimics Luas syntax and uses it as a preprocessor. The only thing that makes it different from, say, C, is that the compiler can actually run the code it generates immediately, and outputting an enclosed executale is an option, but not the default.

As for PUC Lua, yes, it's perfectly ok for most things when you're mostly scripting and using C for the heavy lifting anyway, like when you're building a scripting language into an already existing game engine or program.

LuaJIT was always meant to fill the in-between role (which is, arguably, the broadest one) where you can't be bothered to write a full C application to host your Lua code but still need more speed than you get by just running a whole program in PUC Lua.

> where you can't be bothered to write a full C application to host your Lua code

This Rust code I linked to is an application framework. It makes several of Rust's most powerful libraries available to Lua, which should greatly improve development speed. I'm using it to build an event-based Lua web framework on top of it. (also there) Development of other network applications should also be very smooth, afaics.

Yeah thats a different use case somewhat like a modern version of openresty but built around a set of libraries rather than inside Nginx. There may still be a case for some applications for a jit compiler for the Lua parts but that depends on the application.

Does this still only support 2 Gb of RAM?

(https://stackoverflow.com/questions/35155444/why-is-luajits-...)

No, that restriction is gone. Peter Cawley did the hard work and added optional support for 64-bit heaps to LuaJIT upstream [1] and RaptorJIT adopted that exclusively i.e. dropped the 32-bit heap option which involved a LOT of #ifdef'ery in thousands of lines of assembler code.

[1] https://www.youtube.com/watch?v=ZeaDk9ElKws

Last I checked (which was a year or more ago), 64-bit GC for LuaJIT is an experimental and buggy feature. Do people actually use it in production? Since which version of LuaJIT?

The bugs have been shaken out now and it works fine in production. That's true at least on x86-64 with common build options and the latest LuaJIT prerelease.

Most of the changes are removals of architectures other than x86_64. What a joke.

Are you willing to contribute to the port? Don't rubbish other peoples' work unless you've got something better to replace it with.

good point, but i think we can admit the title is weird if the only support is x86-64

RaptorJIT is good for writing applications like Snabb. Snabb is a networking framework that’s very low level: cycle-counting loops, device drivers, DMA, etc, in Lua. In this domain of high performance software networking the advantages of supporting other architectures don’t currently outweigh the costs.

That’s why it’s a low-level system programming language that prioritizes working very well on x86-64 over supporting other platforms like 32-bit heap, MIPS, Xbox, no FPU, etc. Just got bigger fish to fry right now.

> no FPU.

How much of a hit does floating point math take? I have a desire to build some realtime audio tools.

The wording is confusing it does support fpu it doesn’t support machines without fpu.

It seems pretty clear to me. It doesn't support "no fpu". It's a lazy way of putting it though and "Architectures without FPU" would have been clearer.

AFAIK it is not about float perf.

LuaJIT uses NaN-tagging to represent Lua values in memory to save space.

Float64 numbers can be left alone, and other types are represented as NaN with the pointer bits stashed in the mantissa part of the float. Only one, canonical NaN value is recognized as such by the language, the others are disguised 32 bit pointers.

On microcontrollers, you might as well change to lookup tables if you are trying to do something real-time with more than 100 operations per millisecond.

This varies by microcontroller obviously.. this comes from reimplementing GLSL shadertoy samples for a neopixel grid.

Realtime audio sounds like a good domain. You might also want to check out LuaRadio if you haven't already. http://luaradio.io/

Yeah, 32-bit, Xbox and no-FPU platforms can be dropped, but dropping FreeBSD, aarch64 and powerpc64 is kinda sad.

Does that mean ARM64 is still supported?

From the comment on commit which removed 50,000 lines of code: `The only supported platform is now Linux/x86-64 with JIT+FFI+GC64.`

And the readme: `Reduced code maintenance footprint ~50% by removing #ifdef features that are not required for Linux/x86-64 e.g. Windows support, 32-bit heap support, and non-x86 backends. This is a necessary short-term expedient to make the code maintainable while we bootstrap the project.`

I wonder how that 50,000 lines was broken down by feature...

To be fair, when I was using Lua a decade ago, LuaJIT mostly only supported Linux and x86-64. Mike was adding new architectures and variants on request and when people would pay him to do so, while the core was both mostly sufficient and still improving. Those 50,000 lines did come after x86-64, if you wanted to move the project in a different direction it seems sensible to strip them out, make your changes. As long as your changes a rent too drastic, it should be comparatively easy to put them back in.

Each architecture had a hand coded interpreter. The plan is to write a portable interpreter in C.

Reply on news.ycombinator.com