Hacker News Reader

Comment by antirez | Hacker News Reader

Lua is great to embed, but for Redis I picked it as a scripting language with many regrets because I don't like the language itself: it looks almost like designed to have some kind of friction compared to what you expect to write in a language that is at the same level of abstractions. Little design decisions here and there that cumulate into a language that is somewhat hostile (for me, at least). Yet, it is so fast, easy to integrate, small footprint and reliable, that many times it is still the way to go. Btw, the hostility is present even at C-API level, with the stack approach: it's not rocket science but more cognitive load than needed. Even if I was exposed in the past to stack languages like FORTH, I still had to do some mental gym while writing the bindings. Why?

View on HN

Have you seen the Passing a Language Through the Eye of a Needle article, from the Lua authors? It talks about how some of unusual the design decisions such as pcall error handling and the stack are tradeoffs in favor of "embeddability".

https://queue.acm.org/detail.cfm?id=1983083

The main purpose of the stack is to facilitate garbage collection. The GC can find out what Lua objects are currently being manipulated by the C client. The price is that the Lua API can never expose a "Lua object pointer" to C. Many other scripting languages expose such pointers, but then must also expose ways to manage their memory. For example, in Python the user of the API must explicitly increment and decrement reference counts.

The one time I tried to embed Lua in a C project (which included C-based functions I could call from Lua and setting up variables from C that were visible in Lua), I constantly struggled with trying to figure out when I needed to push/pop things from the stack and it just seemed very error-prone and easy to leak memory with. Different functions seem to have different requirements for when/if you should push/pop things and the documentation was not always clear to me.

Has this changed any?

The stack's still there. I agree that reference manual's notation for how many values are pushed and popped can take a while to get used to, but at least it's straight to the point.

One trap with the stack is that it baits you into carefully sequencing the operations so that the output of one feeds into the input the the next. Sometimes values are popped far from the places that pushed them... It can be easier to reason about code that liberally copies the temporary values. Keep one stack slot for each "local variable" you want to work with. Then to work on them you copy that slot to the top, call the stack operation, and then write the result back to the appropriate stack slot.

Essentially, favor positive stack indices over negative indices, because the latter are more sensitive to the sequencing of your operations. Also, consider giving names to the stack indices instead of hardcoded numbers.

> the hostility is present even at C-API level, with the stack approach

> Why?

I suspect it's because implementing control mechanisms is much easier when you reify the language's stack. Especially advanced ones such as generators and continuations which require copying the stack into an object and restoring it later. Making that work with the native stack is really hard and many languages don't even try.

It also makes garbage collection precise. Lua values in C variables would be placed in registers or the native stack. In order to trace those values, Lua would require a conservative garbage collector that spills the registers and scans the entire native stack. By managing their own stack, they can avoid doing all that.

A historical thing:

> Traditionally, most virtual machines intended for actual execution are stack based, a trend that started with Pascal’s P-machine and continues today with Java’s JVM and Microsoft’s .Net environment. Currently, however, there has been a growing interest in register-based virtual machines (for instance, the planned new virtual machine for Perl 6 (Parrot) will be register based). As far as we know, the virtual machine of Lua 5.0 is the first register-based virtual machine to have a wide use.

The API directly reflected the (previous) internals of the VM, I guess [1].

[1] (pdf) https://www.lua.org/doc/jucs05.pdf

The new register-based virtual machine still uses the same stack under the hood. The only difference is how instructions are encoded in the bytecode. By default they can read and write to any position in the stack as opposed to pushing and popping from the top. The Lua stack API already does some of this, because several operations take stack indices as arguments.

Did you consider any other languages, like TCL? If yes, do you remember the pros and cons of that set of languages?

Was the first version of Redis not in fact written in TCL?

Oh, interesting, I didn’t know. ChatGPT “confirms” that a prototype was written in TCL

You can barely do anything before having to extend the language in a way. It's super simple and as such gets in the way as even the simplest stuff needs some extending/(much attention). Admittedly I have little of experience with Lua, did the work and moved on.

> Why?

See https://news.ycombinator.com/item?id=42519033

WebAssembly is a nice way to go for such things in future. You provide the interface, their wasm code runs in a sandbox, and your users get to write their plugins in whatever language they like.

This gets mentioned a lot but has it been used in practice? How would you pass objects and methods etc to the scripting language? Autogenerated C headers?

To be able to use WASM-compiled languages for embedding you first need a small WASM runtime you can depend on (and things like ref types, GC etc which have been standardized only in the last couple of years).

That didn't exist until recently, but now you can use https://github.com/bytecodealliance/wasm-micro-runtime/blob/...

I am not aware of anyone using this yet, but I hope to see that become common in the next few years.

The WebAssembly component model is aimed at having composable components that can call each other. The components can be written in any language, compiled to WebAssembly, and interoperate: https://github.com/WebAssembly/component-model/

Extism is a plugin framework for WebAssembly:

https://extism.org/

https://github.com/extism/extism

Postgres and Extism: https://dylibso.com/blog/pg-extism/

Wasm plugins for Zellij: https://zellij.dev/documentation/plugins

Visual Studio Code can run WebAssembly extensions: https://code.visualstudio.com/blogs/2024/05/08/wasm

A project to bring WebAssembly plugins to Godot: https://github.com/ashtonmeuser/godot-wasm

WasmEdge can be embedded in applications: https://wasmedge.org/docs/embed/overview

Wasmer can be embedded in applications: https://blog.wasmer.io/executing-webassembly-in-your-rust-ap...

Wasmtime can be embedded in applications: https://docs.wasmtime.dev/lang.html

Complexity of writing and building WASM is a few magnitudes higher than just writing a simple Lua script.

It might make sense for building complex plugins, but for simple stuff, Lua still excels, and doesn't require much programming knowledge in many cases.

Reply on news.ycombinator.com