Lua 5.4.0 beta
lua-users.org
lua-users.org
Oh and one thing if you must use Windows... you better quit right away. I had to help my co worker Installing it with luarocks and it is a mess. to be fair it was easy on my Ubuntu machine.
The thing is lua on embedded has no rival. but god did it cost me some nerves.
I'm surprised to hear this. I'm doing a lot of Lua and I always thought the documentation is very succinct and complete. I really curious when issues you ran into.
> The thing is lua on embedded has no rival. but god did it cost me some nerves.
You might take a look at https://bellard.org/quickjs/ and https://duktape.org/. The latter seems oddly familiar if you've worked with the Lua C API.
I expected to be able to do things like tell Lua to run for X milliseconds or Y opcodes or whatever, but it didn't seem that anything like this existed.
So perhaps my disappointment with the documentation was really just disappointment that the implementation wasn't set up to do what I wanted in the way I wanted to do it. But in the course of coming to this realization, I also found the documentation frustrating and opaque.
I remember patching Lua over 10 years ago for a programming game. I essentially counted VM instructions and stopped executing when reaching the limit. It survived malicious players on a hacker conference back then ;-)
The probably badly aged code is still available here: https://github.com/dividuum/infon/blob/c25eb749b17df3fd9b7a5...
There was a while that I wanted use this to design a programming-based video game, where simulated agents worked in real time. Most of the programming games I have seen use a fixed timeout, and kill any scripts still running after that time. What I was thinking was to instead have the script constantly running, and any slow scripts continue running on the next frame (e.g. all players get 100 opcodes per frame).
It's not bulletproof though and the script can still block e.g. when calling into native code.
The advantage to this method over longjmp is that the function call is still valid and can be resumed at any point.
There's a lot of (seemingly) undocumented changes, even between two minor versions that make you question the semver (and your sanity) at times.
For example, the patterns operate just different enough between the two minors, that I had to resort to dumping everything into JSON and back since that was the only way that one odd case worked similarly between the two versions (albeit sharing the same JSON library).
In regards to the new version, coworker of mine glanced at the 5.4 changelog and could not make out how things had changed, only the fact that they had been changed.
I've been using it to extend some C++ code (bsnes emulator) and it's been fantastic. It's so nice to have a C++-like scripting language with natural script-host bindings that I don't have to think hard about creating or spend a lot of time writing marshalling code or worrying about memory layouts. On top of that nice script-host interface you get an actual usable scripting language without silly things like 1-based array indices.
That said, it does not come with any sort of standard packages like luasocket but there is support for defining your own modules.
Depending on the embedded platform you're using, there may not be support for the native calling convention and you may have to fall back to the generic calling convention, or you could roll your own code to accommodate your platform's calling convention.
What about something like Nim which compiles to C?
Comment on all subjects (I will just link it to other commenters): A lot of commenters posted some projects with JS interpreter. But we had to chose something that is stable and is being maintained. The last thing is always hard to argue about because you never know how long maintainers stay commited to the project. But the LUA interpreter is widely used for a lot of (commercial) projects. So there is high chance that it stays for some time. We tried out some other projects, sometimes looking into mailing lists and github issues where we would find open bugs where the interpreter leaks under some circumstances or other problems that you just want to avoid.
LUA seemed a perfect fit and i think still is, but there are still things that didn't work as planned. I really love the concept of the interpreter with the stack. But, again, scheduling is a very complex topic and that is where i found the lua documentation lacking. Especially scheduling between LUA <-> C. I found some projects that solve this issue but then we stumbled upon other issues: Some projects were for 5.1 and didn't work for us. Others used features we didn't have on embedded (pthreads). So we decided to do a very basic scheduling but had to understand more what happens under the hood. And that is where the source code is not easily readable. Don't get me wrong i am convinced that the devs are great at their work, but holy moly you can't use more readable name for you variables? Or the use of macros, is it really neccessary to use it for every conversion?
Yes, my bad.
- new generational mode for garbage collection
- to-be-closed variables
- const variables
- userdata can have multiple user values
- new implementation for math.random
- warning system
- debug information about function arguments and returns
- new semantics for the integer 'for' loop
- optional 'init' argument to 'string.gmatch'
- new functions 'lua_resetthread' and 'coroutine.close'
- coersions string-to-number moved to the string library
- allocation function allowed to fail when shrinking a memory block
- new format '%p' in 'string.format'
- utf8 library accepts codepoints up to 2^31
Maybe I should go read the docs. Killing a coroutine from the outside looks like poor design/bad idea.
The canonical example is that you can mark a variable holding a file handle as to-be-closed and then as soon as you exit the variable's scope the file gets automatically closed (including if exit early due to break, return, or error).
But what is supposed to happen if you are inside a coroutine and pause the execution before reaching the end of the scope, and never resume again? If there are any to-be-closed variables their destructors will never run! Or they might only run after the containing coroutine gets garbage collected, which is not a timely solution. To cover this situation, Lua 5.4 introduced a new coroutine.close function that kills a paused coroutine and runs the destructors of any to-be-closed variables inside it, if there were any.
Interesting. Didn't Unicode restrict UTF-8 to allow encoding only 21 bits? Does it mean that it can now do 6 byte UTF-8 encodings? What kind of restrictions did it have before?
Does this mean + can finally be used for string concatenation and .. disabled as an opt-in C flag?
This has been both great for Lua as a Lua and one of its biggest challenges. It's allowed the language be constantly refined and tweaked; however, at the expense that it's super common for applications to complete break when upgraded to a point release.
Hopefully they update it with a more current Lua version.
For a pure Lua comparison LuaJIT is typically going to beat Lua by a large margin. For code that can be JIT compiled you might see ~10x performance improvements and for code that doesn't you might still see around a ~2x improvement because the LuaJIT interpreter is written in hand-crafted assembly language (PUC-Lua sticks to portable and standards-compliant C90).
But things get a bit more complicated if you add foreign C code into the mix. Under LuaJIT, the JIT compiled code that uses the LuaJIT FFI interface is the fastest, but interpreted code using the FFI is slower than interpreted code using the traditional Lua-C interface, and can even be slower than PUC-Lua using the traditional Lua-C interface. This means that when you are using LuaJIT you need to be very careful to make sure that your code is the sort of code that can be JIT compiled. If you hit a "not yet implemented" feature you can have a big performance degradation.
------
Nevertheless, Lua 5.4 has brought significant performance improvements compared to Lua 5.3, specially in integer operations (including for loops). Another big change was that the `#` is now faster. For example, on my machine the following loop runs in 1.96 seconds in Lua 5.3, in 1.26 seconds under LuaJIT, and in 0.96 seconds in Lua 5.4
local a = {}
for i = 1, 1e7 do
a[#a +1] = i
end
print(#a)
It cannot be emphasized enough that PUC-Lua and LuaJIT have different goals. One goal for PUC-Lua is that it must be portable everywhere and be implementable in pure ANSI C (no platform or architecture specific calls). LuaJIT in contrast contains much handwritten assembly for the specific architectures it supports.
PUC-Lua also cares deeply about interoperating with C and hence why the Lua-C interface is formally part of the language spec, and as the parent thread mentioned, LuaJIT using the traditional Lua-C interface can be really slow.
That said, PUC-Lua has learned a lot from Mike Pall. It was Pall that helped them get fully yieldable coroutines across pcall boundaries. I'm sure Pall's lengthy analysis of unimpressive garbage collection performance in both Lua 5.1 and LuaJIT was read by all of them and a motivation for them to try the generational garbage collector now finally working in 5.4.
But finally, there is a new project called Pallene (not being pedantic, formerly known as Titan). It is inspired by all the lessons of LuaJIT. The name 'Pallene' is a nod to Mike Pall. Pallene is a statically typed sister language to Lua that is designed for performance and easy C/Lua bridging, but in a way that doesn't bring in all the downsides of JIT. The language is glued at the hip to Lua so there is also easy interoperability between Pallene and Lua code. Pallene introduces static types not for the sake of type safety, but for the purpose of inferring what things can be optimized. Pallene brings in a AOT compiler into the mix and generates optimized binaries. But these binaries also look like any other Lua/C native library so they are callable from normal Lua too and the user doesn't know if they were implemented in C or Pallene.
Pallene was previously discussed on HN here:
https://news.ycombinator.com/item?id=18038619
Also look for the original Titan talk from one of the Lua Workshops a few years ago.
It is fantastic software.
For the multiple user data values feature, I've always found lightuserdata more useful than userdata, because it's not often that you need just one bunch of simple memory that can be freed without any other work. Rather, I almost always have more complex data, such as things that need manual cleanup like sockets, handles, etc. Or, the structure has pointers to other things that must also be cleaned up. What I'd like instead is the ability to pass a destructor callback to `lua_pushlightuserdata` so that it get's called when the pointer falls off the stack.
I don't use Lua threads or coroutines currently, so I don't have much to add there. What I do wish is that there were a way to clone or pass objects from one lua state to another to support parallelism. Basically, I'm thinking of it like the work Eric Snow is doing on subinterpreters in Python. Passing values (by copying, no shared memory) between lua instances.
One use case I had is loading a plugin that you then want to run multiple instances of in parallel with different arguments. Ideally, you wouldn't have to load the plugin multiple times (therefore calling the module level code multiple times). So I'd like to copy the entire lua state and then run each one with different arguments. There's no userdata, coroutines, or lua threads, so I don't have to worry about things that aren't possible to copy.
I got as far as looking into how to copy functions and then got busy with other things and stopped. Is there anyone else out there trying to introspect Lua function structures and copy all the opcodes, upvalues, etc?
The syntax is simple and clean, like JSON but without the most commonly cited warts (no requirement to quote keys, and comments are allowed.) The VM is lightweight enough that including it in an application is practically inconsequential. I like its modular syntax as well. There are also bindings for just about every language out there, making Lua code more easily portable and embeddable.
It's not without its flaws as a language but to me, it's better suited as a config language than TOML, YAML or XML for most use cases where you don't actually need their complexity, or JSON if you're not on the web.
My favorite feature is that it's interpreted. The Android app is just a runtime. I can edit source files right on the phone and re-execute. It's awesome prototyping platform.
VM is tiny and can communicate bi-directionally with C / C++ code very easyly, makes it the ideal candidate for embedding in a larger app as a scripting language.
Currently using it to create a tiny Postscript interpreter: https://github.com/wiladams/lj2ps
I use it because it's a simple language that is quick to code in, and trivial to embed inside a C, or C++, host application.
If I were to begin the project again I'd have no qualms about a similar approach. (Writing the core in C++ with a notion of display-modes, email-parsing, etc, and the actual control in Lua. The biggest painpoints were dealing with MIME, and similar broken emails which was largely irrelevant for the lua-side.)
(If you haven't heard of that, then I implore you, for the love of God, don't look it up and try it out and get addicted, whatever you do! And if you do, then don't blame me for getting you hooked. Oh dammit, now I made myself want to play it again!)
* Easy embedding. Compared to Python and ruby which I looked at back then, Lua is so much simpler.
* Fairly easy to learn for users. After all, the complete syntax is one page long: https://www.lua.org/manual/5.1/manual.html#8
* Quite expressive with coroutines, tail calls, metatables, ...
* Trivially allows multiple interpreter instances within the same program.
* More than fast enough, especially once LuaJIT was released.
It’s a simple, but powerful language that uses basically two features (functions and hashmaps) to implement everything. Ex: script files are implicitly functions. Global variables are in a hashmap. Semantically it is very much JavaScript without all the surprises. Semi-technical users (artists and designers) can figure it out and go on to make surprisingly powerful features.
The binary code is small and fast. The standard library is small and most of it can be removed easily. Parsing source to bytecode is quick. Loading bytecode is very fast.
It is easy to embed. It’s easy to contain it’s memory allocations in an arena. It’s easy to restrict what it can and cannot do. It’s easy to interface with C/C++. It’s extremely portable.
I'm no particular fan of the language, but they do the "embed and interface with C/C++" better than anything else I've seen.
It sounds small, but with callback heavy or embedded DSL code it very quickly adds up.
Also, QuickJS is a good initiative, but I afraid it's not production-ready yet. It doesn't even have versioning and/or changelog.
Nobody's a fan of the quirks of JS, but you can almost entirely ignore them and only use the good parts.
I think the syntax use is: local x <const> = 5 local filename <const> = "/etc/fstab" local fh <close> = io.open(filename, "r")
Anyway, you can find a summary of changes since 5.1 here: https://www.lua.org/versions.html
One of the biggest changes was the introduction of 64 bit integers in Lua 5.3
Perhaps my favorite little change is that you don't need to type = in the repl to print an expression anymore.
I've previously opted for LuaTex rather than "base" Tex or XeLaTex and I'm at the point where I'd happily stick to LuaTex and also learn some of it's more technical aspects.