Lua 5.4.0 | Hacker News Reader

Lua 5.4.0

lua.org

256 pointssmlckz80 comments

It would be interesting to know how many projects are using regular Lua vs LuaJIT, as the two implementations have diverged.

When both were on version 5.1, there didn't seem to be any downsides to LuaJIT. Many Lua projects could benefit from it as long as they weren't trying to run on a platform that disallows JIT compilation (e.g. iOS or a game console).

However, now that LuaJIT is 2-3 language versions out-of-date, and has different best-practices (e.g. around FFI), it's almost like the two are different languages.

Every major release of Lua has always been a different language, and embedding in general encourages not upgrading. If you ship something based on Lua 5.2, how to handle all existing user scripts if you upgrade Lua?

> When both were on version 5.1, there didn't seem to be any downsides to LuaJIT.

LuaJIT has (had?) weird memory limits due to its GC implementation. This matters if you're running scripts on a server.

I think we do have to call this a legitimate language fork at this point. Prosody (XMPP server) runs on 5.2 and does not require LuaJIT. The LuaJIT thing is more of a symptom than a cause.

Performance-wise Lua is not too far behind LuaJIT that you should just "never" use Lua. I used Lua for a long time without problems, and in the cases where you need that extra performance you are just one "system call" away from native performance for that specific operation. I used to benchmark these things (for my use cases) for a long time, and Lua was always somewhere between 50-100% slower than LuaJIT, which is NOT a lot. It's a mistake to think that people implement Mandelbrot as a script callback. :)

It's unfortunate that LuaJIT fell behind so much, but as far as game engines who need a scripting language goes: You are either safe with Lua, or you have other alternatives: JavaScript, WASM, dynamic libraries and so on.

100% slower seems like a lot.

not as much as 200% slower

I recently redid the graphs of the luajit benchmark results to highlight performance across architectures: https://gist.github.com/cellularmitosis/55ff55b55008b2f1c42b...

Nice work, man. I was doing microbenchmarks where I among other things measure the 90ns (JIT) and 120ns (Regular) function call overhead of Lua. My benchmarks was compared to my own scripting backend where the call overhead is 4ns, and it always feels a little bit like cheating when you have a gigantic headstart with each test.

I'm not surprised the JIT isn't that good, but this is crazy almost! Did you compare against the latest version of Lua?

https://gist.github.com/fwsGonzo/f874ba58f2bab1bf502cad47a9b...

In my microbenchmarks LuaJIT was always faster than Lua, but not so much that I wouldn't stop using Lua. In my mind once you stop developing a language it stops being interesting, because you are always looking for ways to simplify and remove whole classes of bugs.

EDIT: I just built lua5.4 from sources and ran my benchmarks again, and it's markedly worse than LuaJIT still, but it's better than before for sure. My conclusion is that Lua5.4 is the fastest PUC-Lua yet.

https://gist.github.com/fwsGonzo/2f4518b66b147ee657d64496811...

Some of these benchmarks are maybe not how one would do things if they were trying to go fast. An unfortunate idiom I wrote a thousand times at a company that uses Lua and tries to go fast is

  local n = 0
  local t = {}
  -- to do an insert
  n = n + 1
  t[n] = blorp
  -- to delegate some inserts to another function
  n = insert_some_stuff(t,n)

A less messy thing you could do is

  local table_insert = table.insert

but this is not as great, because lua will still do a binary search to find the end of the array each time you insert something.

That might be, but the C++ array append is bounds-checked which makes it half as fast as it could be too. It would be 16ns if there is no checking. So, you are right but I am trying to do a balanced approach here. These benchmarks are made for me, and I want to write (more or less) normal code.

I don't want to write local table_insert = ... I'd rather drop Lua for something else then. That said, it's cool that there are things you can do to speed things up if you really have to.

Fair enough! I think t[#t+1] = x is not as offensive and will get you a few nanos less than the way with two table lookups, but if that's also not to your taste then that's fine.

Lua's weak point is around doing things with arrays, IME. It's just not very fast when you actually need something simple.

I actually started poking around at a language design over the past month that tries to leverage Lua as the compiler for something that can run in its own separate bytecode interpreter, with more of an emphasis on low level primitives. It started off as a simple adaptation of Forth ideas, but yesterday I started playing with a revision that shifts the data structure from plain old stack towards growable arrays containing three cursors(thus, "tricursor array") which can be purposed to describe bounds, destinations, insertion points, read and write, top of stack, etc. In Forth the top three values of the stack tend to get dedicated words for their manipulation because they are used quite often; this model runs with that idea plus methods I've often used for array data(text edits, sound sample loops, blotting sprites) and tries to extrapolate that into a language that can ease certain forms of array programming, while falling back on using the array as a data stack. Still just sketching it now.

We chose to go with Lua on our embedded project ~2 years ago, and chose lua 5.3 with the following rational:

1. Lua embeds well on a small system and gives us a higher level language than C. lua or luajit is ok

2. Lua isn't being used for performant code. If we have performance issues use C.

3. LuaJit is looking for a maintainer and is stagnant. This isn't desirable.

Not sure if this is "best practice" but it summarized how we worked through it.

Lua 5.4 has some appealing benefits for us, we'll give it a few patch releases, but I look forward to moving to it.

Re Performance: We're actually running from 50Hz code in Lua on an old embedded CPU. We haven't re-written that in C yet since it runs just fine. We're been constantly surprised at how well it runs and works for us.

If you use LuaJIT, you can generate bindings to C libraries, versus handwriting bindings for PUC Lua. The productivity difference is staggering. Additions to Lua since 5.1.5 have not helped me write more or better software, as much as I love Lua.

LuaJIT:

+ Super easy FFI to C

+ Very fast (although profile to make sure this matters)

PUC Lua:

+ Possible to run untrusted code (e.g. user scripts) in a sandbox

+ Will run on a huge variety of platforms, including many microcontrollers (only limiting factor is a C compiler and a mostly-complete libc)

+ Has some additional language features (although some features, including integers, are controversial)

+ Actively maintained, expected to receive new releases and updates

What does PUC mean in “PUC Lua”?

https://en.wikipedia.org/wiki/Pontifical_Catholic_University...

PUC-Rio[1] which is the university where the original authors and current maintainers work. See "Where does Lua come from?" here: https://www.lua.org/about.html

[1] https://www.puc-rio.br/english/

This release brings many performance improvements, including a new garbage collector.

In terms of language features, the biggest new feature are the "to be closed" variables. They can be used to ensure that cleanup routines are always called, similarly to RAII in C++ or try-finally blocks.

<close> is equivalent to golang's defer (either can be implemented in terms of the other) except at the block level. imo calling it RAII mostly leads to confusion. One mailing list user recently asked a bunch of questions around using <close> for RAII. They expected RAII to work even for resources that are never bound to a particular type of local variable, for example if one writes do_thing(io.open("foo.txt")), where do_thing is not responsible for closing files because sometimes it is used with files that will continue to be used. They eventually concluded that the closest thing to RAII available was __gc.

Some users presented separate complaints about resources kept in to-be-closed locals in coroutines not necessarily being closed. You can do something to try to solve this, like setting the __gc and __close metamethods of coroutines to call coroutine.close. A "real" solution would look more like dynamic-wind. Notably golang doesn't attempt to solve this one, so maybe not solving it is fine.

> <close> is equivalent to golang's defer [...] except at the block level.

Does that mean it’s the same as C#'s `using` and Python's `with`?

I wasn't deeply familiar with python's `with`, so I looked it up[0]

<close> differs from `with` in at least the sense that it doesn't have any equivalent to __enter__ and doesn't create its own block. It creates a local variable whose __close metamethod will be called when it goes out of scope. Since Lua has lexical scope at the block level rather than the function level, this works similarly to the way Python calls __exit__.

These snippets of Python and Lua are roughly equivalent. They both open a file, read one line, and close it, or do nothing if there was some error opening the file.

  try:
    with open("foo.txt") as f:
      print(f.readline())
    # f is now closed.
  except:
    pass

  local f,err = io.open("foo.txt")
  if not err then
    local f <close> = f
    print(f:read("*l"))
  end
  -- f is now closed.

C#'s `using`[1] seems much closer, except that it handles nulls by not trying to close them and lua's <close> does not include any such handling.

[0]: https://docs.python.org/3/reference/compound_stmts.html#the-...

[1]: https://docs.microsoft.com/en-us/dotnet/csharp/language-refe...

The reference manual https://www.lua.org/manual/5.4/manual.html#3.3.8 seems to specify that "nil" and "false" values are ignored, so it behaves similarly to C#'s using.

Oh, that's great!

Do file descriptors come with a default __close metamethod or do you have to create your own?

They come with one.

There are many lightweight, embeddable scripting languages available, yet Lua remains dominant. This is despite the language being "quirky" at best (and "dreaded" at worst [0]).

Is this just inertia, or is there a technical reason for new projects to embed Lua rather than a different language?

[0] https://insights.stackoverflow.com/survey/2018#technology-_-...

It's very easy to embed, and LuaJit is pretty fast.

What better languages are there which are friendly to embedding? A significant part of Lua's popularity, AIUI, comes from it being easy to embed.

Squirrel [0] seems to have been used in a few places [1], but not many compared to Lua. There are also embeddable JavaScript implementations such as Duktape [2], and a lightweight Ruby in mruby [3].

[0] http://www.squirrel-lang.org/

[1] https://en.wikipedia.org/wiki/Squirrel_(programming_language...

[2] https://duktape.org/

[3] https://github.com/mruby/mruby

Guile scheme, if you have users that don't have the knee-jerk "ewww parentheses!" reaction. The threading model is vastly different from lua, but compared to regular lua you get a lot faster execution.

Guile is great and I enjoy how active it has been. However, it is no more that small embeddable lisp. Its embeddable of course, but not small by any means.

One you start wanting to run complex things over multiple interpreters in Lua (say, to use multiple threads) you might as well just use guile.

Sure, we are talking 2.5mb of library and about the same amount object code. Quite a bit larger than Lua. But that also gives you object code for working with texinfo (and quite a lot of completely unused, undocumented modules). I wonder how much could be stripped without anyone actually boticing

I love Angelscript for that purpose (w/ C++). Very similar syntax to C++ (but w/ GC), very easy to expose classes, functions, and variables. All it takes is a quick compile & link. It also has a JIT compilation if you need it and ability to save/load the intermediate binary code for faster loading.

Wren https://wren.io/

Tcl leaps to mind. It is very easy to embed, too.

ChaiScript [1] For c++.

Almost trivial to add functions or calls in both directions.

- Header only project, with modern c++. Able to compile with and without mutex. Multiple context can run in parallel.

- Awesome integration and binding with c++ bars and functions.

There are two drawbacks. One is the maintenance and some performance quirks with returns and type conversions.

The other drawback, maintenance only [2]. the project accepts pull requests, but no recent activity on lenguaje or performance.

[1] http://chaiscript.com/

[2] https://github.com/ChaiScript/ChaiScript/commits?author=RobL...

I enjoy writing Lua... It's kind of somewhere between Javascript and Ruby in my mind...

Tcl and various scheme dialects were popular, but they have a pretty unusual syntax, at least when viewed from a typical C/Algol-ish programmer. Lua just came at the right time and place and was more "usual" than that, also with a pretty small package and permissive license.

And I would say that this argument still holds true. For extending a game or application, I'd feel pretty bad forcing people to use JS, for example (unless it's EE, where the gloves are off). Wren might be an option, but it's certainly a lot less proven.

Also, there's LuaJIT, if you really need some performance in a small package.

StackOverflow might not be the best source for end-user scripting. (Never mind that I have my doubts about any statistic where Rust ends up winning the popularity contest.)

Not sure why it’s dreaded, Lua is a joy to write code in.

> "quirky" at best (and "dreaded" at worst

I think it depends heavily on the person. Tcl has this problem, too, where lots of people love it and lots of people are completely put off by it.

Lua is weird, but it's competitors are also weird. Tcl, Scheme, Perl and Javascript are all at least as quirky if not more. JS in particular is at least as "dreaded". There are a few more "normal" alternatives, but they're less popular and have even smaller ecosystems than Lua, where package availability is already bad. Python and Ruby are huge by comparison and don't embed all that well.

The technical reason might be that a lot of people like working in this language, and consider it to be well designed.

One interesting change in this release is the move to a stackful VM. This creates a pretty small (edit: actually huge, 1978 stack frames in a lua 5.4 repl installed with luaver just now) limit on the depth of the Lua call stack. I wonder what motivated this decision.

Was it not a stack machine already?

The embedding API explicitly exposes a stack, so I assumed that it was implemented that way.

it had a stack, but that stack was on the heap. In lua 5.4 under normal circumstances, 20 frames of Lua call stack corresponds to 20ish frames of C call stack. In lua 5.3 the number of C stack frames was some smaller number unrelated to the depth of the Lua call stack.

Ah - so it uses the C native stack.

Citation? I don't think anything significant changed in that regard. Coroutines are still "stackless" in terms of the C stack, but "stackful" in terms of the language semantics. The virtual data stack is still clearly on the heap.

Lua 5.4 adds a new function, lua_setcstacklimit, but this merely exposes what was a compile-time constant, LUAI_MAXCCALLS, in earlier versions. Lua can't avoid making some use of the C stack as it supports intermingling of lua_call's from C code with in-language VM calls, which necessarily will use some amount of C stack. Lua has lua_callk for yield/resuming across C code invocations, but not everybody makes use of this and in any event it's only for coroutine semantics, not for recursion.

> Citation? I don't think anything significant changed in that regard.

The limits for recursion have undertaken a significant change. (Though it might not be that concerning, if you see my sibling answer).

> As already discussed, Lua 5.4 is using a "non stackless" implementation. That means that it uses the C stack for recursion in Lua functions. Because of that, the maximum depth of recursion is much lower in 5.4 than it was in 5.3. - Roberto Ierusalimschy [0]

[0] http://lua-users.org/lists/lua-l/2019-06/msg00083.html

See this prior discussion https://news.ycombinator.com/item?id=23297500 and this commit https://github.com/lua/lua/commit/196c87c9cecfacf978f37de4ec...

In CorsixTH we're hitting C Stack Overflow issues on 5.4. I'm having particular trouble with our serializer for saving games which includes recursive calls to lua_gettable.

I believe, but am not certain, that the change was made to help speed up Lua. As an embedded language, and a language that runs on embedded devices, performance is one of the key criteria of design that goes into the language. The compile-to-bytecode step may have seen some improvement with the change.

So instead of speculating on why the change happened, I've found a few bits of information around the limits that make me feel like it often simply won't be a problem.

There's some interesting discussion around the limits across various platforms here [0].

Whilst I'm not entirely clear on the motivation, even Lua running inside a browser under emcripten has a stack limit of over 4000, which is somewhat decent even for recursive functions.

Whereas on Linux you seem to be able to tweak a config when building and safely have it in the region of 20,000, and quite possibly more.

Even under a tight restriction, Lua seems to cope fairly well with it when recursing (the example crashed at 1758 recursions for a ulimit of 400).

Whilst Python has a terrible limit of 1000 for the default recursion limit which is somewhat comparable to this aggressive test, Python also has no tail call elimination - but Lua does.

I have a somewhat large Lua project (6000 lines of C, 10,000 lines of Lua) that is a recursive parser for a format that I passionately hate (think self-modifying LaTeX). With the default 5.4 limits it never hits a point where it crashes, though I had expected it to.

[0] http://lua-users.org/lists/lua-l/2019-06/msg00083.html

Doesn't Lua have tail call recursion built in to the language? Stack overflows shouldn't be an issue on properly written recursive functions.

Yes, it has tail call elimination (which I think I mentioned). But you can still right badly behaved recursive functions that avoid tail call elimination, and that's when you'll hit your stack overflow.

I have a few of those badly behaved recursive functions in the parser I mentioned at the end of my comment, and expected them to overflow, but they didn't hit the default limit.

Ah yeah, I checked just now and the number of allowed frames is quite large. I observed a smaller limit of around 200 with the default config of an earlier release candidate of 5.4.0 and just expected the final version to be similar.

I really like lua. Especially when it comes to game with Love2d beeing so wonderful. I wish more projects adopted it as a scripting interface...

in my knowledge most tools with scripting support uses lua, there's not a lot of options (i'm also a big fan of love2d)

Love2D is indeed excellent, and integrates well with the Tiled map editor. It makes making games almost as easy as Macromedia Flash did!

I wanted to like Lua and Love2D, but struggled with writing code quickly because there's no editor autocompletion that I know of, so I spent a lot of time having to reference the docs. Do others just use it enough that they know the API by heart now?

"const variables" makes me giggle a little

The new random library implementation should stop me dragging in a Mersenne Twister for every other thing.

It was libc's random, which is... Trash.

Now it use's xoshiro256 [a], which I hadn't heard of. (Not my area). [0] It seems somewhat similar to the MT, as it isn't cryptographically safe, but should give you a fast and fairly random set of integers back. It's also significantly faster than the MT.

[a] Followed by two asterisks. I give up trying to make them appear in HN's formatting.

[0] http://prng.di.unimi.it/

To give a very short summary, xoshiro256 is much faster but also has a much smaller period (however the period of the Mersenne Twister generator is way too large for most applications anyway).

Yeah, I was happy to see the switch away from C random/srandom or rand/srand.

Not my area either, but a little surprised why a PCG-family non-CSPRNG algorithm wasn’t used: https://www.pcg-random.org/posts/a-quick-look-at-xoshiro256....

PCG XSL RR 128/64 (MCG) could be used for 64-bit values (albeit with period 2^126 compared to xoroshiro256’s 2^256 − 1) with 128 bits. Half the state size than xoshiro256, and the performance might be comparable (would need benchmarking). LCG version if you want to trade PRNG quality for even more speed.

I just recently started using Lua through the sol2 library. I'm basically providing some flexibility through my input file to my users. I'm very surprised by both library and Lua itself. For my use, I don't feel a lot of performance hit, and gets a lot of flexibility.

[Unfortunately, IMHO] Lua "minor versions" tend to break compatibility with older versions, making it problematic to upgrade to new versions without breaking scripts written for older versions. I believe mpv is stuck on 5.2 due to this. Here's the list of incompatibilities for 5.4: https://www.lua.org/manual/5.4/manual.html#8

Yeah, Lua doesn't use semver, which is ok. Just multiply by 10 and pretend it does. So this would be Lua 54.

Yes, Lua doesn’t do “minor versions”. You can think of 5.4.0 as 54.0.0, and 5.3.5 as 53.0.5.

Likewise OpenResty is "stuck" on 5.1.

And doesn't seem to suffer from it.

"Stuck" might be a wrong word. OpenResty actually dropped support for Lua and is now LuaJIT only. LuaJIT is a fork of Lua 5.1, but it has features from 5.2 [1] (and from later versions where it sees fit). Because of this, Lua 5.1 is a golden standard of Lua world at the moment.

[1] http://luajit.org/extensions.html#lua52

There is more modern alternative to LuaJIT though - MoonJIT [1]. Unlike the former it supports modern Lua standards.

[1] https://github.com/moonjit/moonjit

AFAICT, MoonJIT is just a fork of LuaJIT. Both support many 5.2 extensions. Additionally, MoonJIT also has a few Lua 5.3 extensions, but it is not fully compatible with Lua 5.3 or even Lua 5.2. Quote:

"Note: this provides only partial compatibility with Lua 5.2 at the language and Lua library level. moonjit is API+ABI-compatible with Lua 5.1, which prevents implementing features that would otherwise break the Lua/C API and ABI (e.g. _ENV)."

Some internal language changes in 5.3, like the new integer type, would require deep surgery in the JIT. This also seems to be the main (but not only) reason why Mike Pall doesn't have plans to ever support Lua 5.3. (https://www.reddit.com/r/lua/comments/2zutj8/mike_pall_luaji...)

Yes, and OpenResty maintains their own fork [1] as well.

[1] https://github.com/openresty/luajit2

Am I wrong? I thought the author of LuaJIT has been pretty well over it for the last 5 years and no one else picked it up.

LuaJIT had a beta release 3 years ago and hasn't been updated since.

Looks like Mike Pall is still actively working on the project: https://github.com/LuaJIT/LuaJIT/commits/v2.1

The last commit is from 5 days ago.

<close> seems to be a nice feature. wish it could be ported to luajit.

The only thing you're saying is that it's lamentable that they use the "5.4" version number for something that's not a minor version.

That's what this is, this is the next major version of Lua.

I think new versions of a language, major or minor, breaking compatibility with old code is generally lamentable regardless of version scheme. The version scheme they chose is not the root of my disappointment. Because so many programs will likely never upgrade their version of Lua, this leads to fragmentation of the community.

many projects I am aware of stuck with 5.1 and luajit still and so do I. I still use lua5.1 myself these days.

Reply on news.ycombinator.com