Writing a DSL in Lua (2015)
leafo.net
leafo.net
I’m currently writing a just for me static site generator in Fennel and it’s sad to see just how many powerful packages are abandoned. A lot of it is because they are mostly complete and part of why I like it is lack of churn but a last update on 18 months ago is very different from 10 years.
I want to host knitting patterns with my site generator and techniques like this make it super easy to build a little pattern dsl. It’s stuff like that which makes me wish it was more widespread.
I think it’s gets about the right amount of credit. It isn’t obscure by any means, openresty is used quite a bit in odd places, so many games have used it. PyTorch (well what became it) was originally written in it (it was just torch but basically LuaTorch)
It doesn’t get more adoption because its warts and limitations really start to show once pushed into more into a “real” language. It doesn’t have the browser monopoly. It’s also quite slow, and LuaJIT doesn’t always magically fix that.
Not to mention there are some significant performance issues on aarch64. Love2D, a popular game framework, opted to not use JIT on macOS for this reason.
For a long time a new garbage collector has been an open issue for LuaJIT, which would fix this particular issue, and make the language even faster. Last I checked this was actively being worked on.
No it is not.
https://github.com/LuaJIT/LuaJIT/issues/285#issuecomment-197...
> If your workload fits inside 2 GiB (IIRC) it will be fast.
This has nothing to do with the issue being mentioned here. You are understating the severity of this issue and seem to be confusing it with a different issue that is no longer relevant in 2.1.
There are very valid complaints about Lua. It lacks useful quality of life features. Some of its design decisions grate on users coming from other languages. Its standard library is barebones verging on anemic, and its ecosystem of libraries does not make up for it.
But after using Lua in many places for well over a decade, I’ve gotta say, this is the first time I’ve heard someone claim it’s slow. Even without JIT, just interpreter to interpreter, it’s consistently 5-10x faster than similar languages like Python or Ruby. Maybe you’re comparing it to AOT systems languages like C and C++? That’s obviously not fair. But if you put LuaJIT head to head with AOT systems languages you’ll find it’s within an order of magnitude, just like all the other high quality JITs.
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
Maybe I should say it is "still slow". Of course you can find examples of 5-10x over python performance (not that this is saying much), this is not the norm. There's just no way, PUC-Lua is still fundamentally dynamic hash tables for lookup with a big switch statement, there's a certain limit to how fast this can go. Let's just say there are plenty of game devs that have found that Lua quickly becomes too slow for the task it has been pressed into which then becomes an exercise of exporting more and more stuff to C++ or whatever. There is Luau of course, which sort of proves this point.
These interlanguage comparisons are mostly meaningless anyway. Python may be slow, but you do the heavy lifting in other ways and that ecosystem is huge. Since the Lua ecosystem is small though you are usually on your own. LuaJIT FFI is sorta cool, but wrapping can still be a PITA.
> Maybe you’re comparing it to AOT systems languages like C and C++? That’s obviously not fair.
Or C#/.NET/Java/Kotlin/JVM. Which you can often use as part of a plugin system.
> this is the first time I’ve heard someone claim it’s slow.
Quite ironic to say this, because I'm pretty sure this is why Mike Pall created LuaJIT in the first place. For example http://lua-users.org/lists/lua-l/2011-02/msg00742.html (but there are many other great in depth essays in the archive)
> LuaJIT is also one of the most advanced JIT compilers out there, frequently beating V8 and others.
No it's not these days. LuaJIT is awesome and may have been alien technology in 2005 but has a lot of missing functionality compared to the most advanced JIT implementations.
There were of course many examples of LuaJIT beating V8 on particular microbenchmarks through the years, and many of these are rehashed from over a decade ago and are outdated. I wouldn't say this means it "frequently" beats V8, especially in a practical sense.
Anyone that has much experience with LuaJIT and has stared at a few traces is aware of "NYI" - there is a ton of stuff that is not implemented in the JIT and means fallback to the interpreter, including even pedestrian uses of closures: https://github.com/tarantool/tarantool/wiki/LuaJIT-Not-Yet-I.... There is a ton of stuff that current V8 does to optimize real world larger applications that LuaJIT JIT will bail on.
LuaJIT does well when you write code in a particular style that exploits its strengths. It's not quite as amazing for general purpose use.
Elsewhere you suggest that a GC rewrite is actively being developed, but LuaJIT development has slowed tremendously, the mailing list is not very active and I wouldn't be holding my breath for anything major (this new GC has been talked about for well over a decade).
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
Python and Ruby have had steady increments of improvement over the last decade.
so many times I’ve read an article like this and thought that it could be even more widely useful
Lua is not widely used because it doesn’t exist. Lua {5..1-4,JIT} do exist, but these are different languages between which you and others can easily port code sometimes. It’s sort of cool in a moment, but if (when) you live through a couple of versions, that just gets old. It’s not immediately visible, so you’ll see reviews from 5 stars to zero at all times.
IIUC, they're using Lua's language's function application syntax, combined with the ability to introduce a different namespace environment for function lookup, and then they're evaluating the tree of function calls at run time.
In languages that support doing this at compile time, through compile-time evaluation or compile-time syntax transformation, you can make run time faster. For example of syntax transformation, https://www.neilvandyke.org/racket/html-template/ shows an example of expanded syntax, where it's rendered some of the angle brackets and coalesced that with (escaped) HTML CDATA.
If you wanted to extend that, to support chunks of traditional HTML angle-bracket syntax inline with code (without using string literals), like in React, you could define a Racket reader to parse those chunks and transform them to parentheses syntax above.
It's not a coincidence that Lua supports such syntax for such use cases.
I was building a bunch of html pages for an htmx frontend and a golang back end, and got really exhausted from using the builtin `html/template` library in golang. I kept trying to build reusable components, and it just didn’t work as I wanted it to.
I ended up doing this exact thing as mentioned in this blog post. https://github.com/DBarney/Glua Granted this is just for me, and for prototyping. I put it on GitHub so I wouldn’t loose it.
Writing html got so much easier this way:
local function Component(data)
return div{"my name is", data.name}
end
return function(params)
html{
head{title="this is my page"},
body{
h1{"This is my small page I wrote"},
p{
"some content",
"more content"
},
div{"page has some dynamic values",params.count},
Component{name="daniel"}
}
}
end
Writing a DSL in Lua (2015) - https://news.ycombinator.com/item?id=22223542 - Feb 2020 (41 comments)
function pio_timer(_ENV)
public(start):
set(y, 0)
mov(y, ~y)
wrap_target()
delay:
pull()
mov(x, osr)
loop:
jmp(x_dec, loop)
mov(isr, ~y)
push()
jmp(y_dec, delay)
endThere are two things in play here:
First, the _ENV variable here is special.
It is implicitly used to lookup any identifier that has no visible binding, locally or in the surrounding scopes.
Thus a script just containing print("Hello world") is really _ENV.print("Hello world").
Usually _ENV is just implicitly defined to be the global environment (available as _G), but it can be overridden within a lexical scope to a custom value.
Second, the jump labels just make use of the alternate function call syntax of object:method(args), which is equivalent to object.method(object, args).
The whitespace is non-significant, which allows it to be written like that.
In combination with metatables you can use _ENV to track variable reads, calls etc. within a function, which you can (abuse) to create DSLs.
You can get an idea of whats possible by just tracking what we can intercept with _ENV. Adding the following code after the pio_timer function, and running the script with Lua 5.4, already gives us quite a bit for just the first three lines of the function.
local loggerMetatable = {} function loggerMetatable:__bnot() print("Called bnot on " .. rawget(self, "name")) end function loggerMetatable:__call(...) local name = rawget(self, "name") print("Invoked " .. name .. " with " .. #table.pack(...) .. " args") -- Return another logger table to visualize variable interactions return setmetatable({ name = name .. "(...)" }, loggerMetatable) end function loggerMetatable:__index(key) local name = rawget(self, "name") print("Accessed key " .. key .. " on " .. name) -- Return another logger table to visualize variable interactions return setmetatable({ name = name .. "." .. key }, loggerMetatable) end
local pio_env = setmetatable({ name = "_ENV" }, loggerMetatable) pio_timer(pio_env)
-- Output:
Accessed key public on _ENV
Accessed key start on _ENV
Invoked _ENV.public with 1 args
Accessed key set on _ENV.public(...)
Accessed key y on _ENV
Invoked _ENV.public(...).set with 3 args
Accessed key mov on _ENV
Accessed key y on _ENV
Accessed key y on _ENV
Called bnot on _ENV.y
Invoked _ENV.mov with 1 args
[...snip...]
“But bro, Python has C libraries!!”
and?
Lua is built out of C semantics and has a beautiful C API
if I want native libraries, I’ll just take a little time to write a trivial wrapper for precisely as much functionality as I want from the library
There's even a fork of moonscript, called yuescript, which has a speedier parser, and includes macro's. Effectively, it's moonscript-2.0, and it's bundled directly into the dora-ssr game engine (the same dev is involved with both).
Of course, the "big dog" in the compile-to-lua space is fennel. The great thing with all of these options is that they don't have a runtime component- it's all lua semantics, all the way down. So you can mix & match from project to project, as your whim takes you - and all the while you're really just gaining more experience in building lua solutions (while not having to fight against some of lua's syntax)