Writing a DSL in Lua (2015) | Hacker News Reader

Writing a DSL in Lua (2015)

leafo.net

108 pointsColinWright41 comments

Hey, I'm the author of this blog post. Thanks for posting it whoever did.

I wrote this a long time ago, it's mostly targeted at Lua 5.1 but more recent versions of Lua approach function environments differently, so keep that in mind.

I've written & worked with quite a few HTML generation DSLs at this point. I put them into two groups:

1. Nested object return: the object is converted into HTML after executing the template

2. Evaluate and write to buffer: no return values, each "tag" is a function call that generates code in a background buffer (nested html typically accomplished by blocks or inline functions)

Approach 1 is what was used in this blog post. Approach 2 is what I ended up using for my web framework: https://leafo.net/lapis/reference/html_generation.html

Approach 1 is used by React. Approach 2 I first came across in a library called Erector: http://erector.github.io/erector/

I prefer approach 2. It lets you use your programming language constructs to do things conditionally, in loops, or whatever else your language provides.

I find that with approach 1 you tend to try to bend the language to convert everything into an expression that can be returned into the nested object you're building. The example I have off the top of my head is how React devs will write some pretty nasty ternary operator expressions just to write some HTML conditionally directly inside of a JSX chunk. (The alternative would be pulling things out into temporary variables, which just creates a lot of noise)

As a fun aside for how far you can take things, for my implementation of approach 2 in my web framework, I ended up writing a syntax transformer that could pre-render static HTML chunks to plain sting that can be appended to the output buffer. Essentially removing those function calls and any HTML escaping that would be necessary during evaluation time of the template.

Also, Dart has syntax for making ifs and loops in collection literals. Called collection if and collection for. If only we could bring all really good features into a single mainstream language and focus on creativity instead of routine. Sigh.

https://dart.dev/guides/language/language-tour#lists

Function environments, resumable coroutines and dsl-friendly syntax is what makes Lua feel so superior to js (in a sense of the language power, not in syntax, optimization or minor design details). It is a shame that browser vendors said “nah, too hard” for coroutines and resorted to generator hack for asynchronous operation – and no one reputable complained! I don’t get it why the execution model should suffer just because someone finds it tedious to split their 300 C calls to few continuation callbacks.

The entire JSX thing could be replaced by something similar, seamlessly connecting code and hyperscript. Callbacks could operate directly on “this”, like in “onclick() {count++}” without making every damn “count” a silly lexically visible [count, setCount] couple. Every missing conceptual feature in js is a total collapse of an entire dimension of possibilities. And yet it is “a language of the future”. How we got here?

The reason explicit asynchrony produces more reliable software than implicit cooperative threading like Lua coroutines is explained at book length in the doctoral dissertation of Mark S. Miller: http://www.erights.org/talks/thesis/markm-thesis.pdf

He is one of the main participants in the ECMAScript committee.

If you disagree with his arguments, please explain why. But you don't seem to have heard of them in the first place, simply assuming that the reason JS isn't designed the way you would have designed it is because the designers weren’t as smart as you are, explaining their choice as “someone finds it tedious to split their 300 C calls to few continuation callbacks.” Sometimes people will disagree with you for reasons other than being stupid or lazy.

Would you be willing to explain why explicit asynchrony is superior to implicit cooperative threading? Rather than simply engaging in more namecalling.

Mutable state is always a problem, but it's a bigger problem when there are less constraints on when it can mutate. But Mark's dissertation explains this and some related issues a lot better.

Also, I don't think I did any name-calling?

That isn't a "normative" statement about better, but a "positive" statement about a trade-off of expressive power versus rigor. A bit of mathematical intuition suggests that formally asynchronous approaches tend to be less powerful and hence more rigorous. But if your spec is still in the prototype phase, taking on a lot of expressiveness and permission with respect to your domain model is desirable because it gets you an end-to-end solution sooner.

What is good is not "on time" or "robust", but "on time and robust". Necessary and sufficients.

Yeah, that's why I said “produces more reliable software” rather than “is better”. In this case what you're buying with the flakiness is more a matter of performance than of expressiveness, but everything is intertwingled.

>Mutable state is always a problem, but it's a bigger problem when there are less constraints on when it can mutate.

A very long-time project I’m idly working on with my life-long colleague addresses this issue. It is real, as are all race conditions and threading issues. But it doesn’t have to be solved at the language level.

One of solutions we proposed to ourselves was that an object storage should manage shared access conflicts. You may see it in Redux and other stores that serve a “frontend framework backend layer” duty today. Even with explicit async, you will always have some high-level races that must be dealt with. And these do exactly that.

The basic idea is that local state is strictly local and not a subject of parallel mutation, and shared state is shared via internally locking/serializing controller who knows better how to mutate, merge or throw. It doesn’t push the problem down the code or execution model, since it’s what’s it for.

Sounds like you'd benefit from reading Mark's dissertation. Not that he solved the problem but he did try a lot of candidate approaches, discuss them with others, and comprehensively review the existing literature; and he reports on several years of trying things similar to what you're thinking of.

I’m not going to read this 229-page doctoral dissertation book for two reasons.

First, it is too long to be practical. Please don’t push the doctoral things on me, we all know that 99.(9)% of what we do in js is not a rocket science. If you want me to feel stupid before someone more educated, fine, I’m okay with that. But even if that’s true, people love to experiment and find their best ways to do something. Also it’s not me who implemented coroutines in Lua this way, it’s PUC Rio guys who made it looking at Scheme. I trust them better than a committee that required 30(?) years to make something bearable without transpilation from the future and hundreds of polyfills.

Second, I’ve asked this question before, and it was browser guys who said that they are not willing to do that, because the legacy is real. And webassembly will not support that neither, because it works in a browser and should obey its rules. Whatever that dissertation concluded, it is committee+browser decision to strip all possible source languages from having such coroutines when targeted at webassembly. It is simply not fair, freedom restricting and has nothing to do with someones educated opinion.

JS is designed in part based on lessons learned from the mistakes of past languages.

Scheme doesn't have coroutines as such; it has first-class continuations.

You can definitely implement cooperative threading in a compiler whose target language doesn't have any kind of threads. You know the 386, SPARC, and PDP-11 don't have threading? It's all simulated. (Wasm probably will have first-class shared-memory threads though. It's being trialed in Chrome already.)

>You can definitely implement cooperative threading in a compiler whose target language doesn't have any kind of threads

I can’t find that thread in my comment history, but there was a wasm guy who basically said nope, not gonna have it, loud and clear, exactly in the context you noted. If that was not true, and all browser vendors will allow it, good.

You can just compile to explicit CPS or equivalent.

This makes me want to learn Lua just so that I can see what kind of things I can do with it. There's a DSL in Kotlin for HTML as well that looks syntactically similar. I have not dived enough into DSL land yet, but this looks kind of fun.

Rust's lexical macros look even more interesting: https://github.com/bodil/typed-html

The same could be said about anything Bodil Stokke publishes.

Groovy also has a great HTML DSL: http://groovy-lang.org/processing-xml.html#_markupbuilder

Implementing your own DSL is fairly trivial too: http://docs.groovy-lang.org/docs/latest/html/documentation/c...

As I've been learning more CLISP, posts like these just become more and more confusing to me. LISP is every language that ever has or will exist. Some macros are basically mini programming languages in their own right. I've seen too much. Everyone needs to learn LISP now.

I feel like if you use React right, it’s a pretty good for making DSL-like stuff.

As part of a previous job, I had to redesign the infrastructure monitoring dashboards, but shenanigans did not allow me to install different software on the dashboard server. So I wrote a React app that uses a custom renderer, which, instead of writing to DOM or to a HTML string, renders into the configuration format of the application.

We used a more traditional enterprise "serverful" architecture, so information about services and hosts was important for availability purposes.

I do not recall which was the application, but I could then export that compilation output via API to update the dashboard with new services.

I used a few layout components wrapping Facebook's Yoga layout library (a library to solve flexbox layouts), and put them into high-level components, such that the infrastructure guy could edit a plaintext file and run the compilation to update the dashboard.

It was basically glorified XML, of course:

    <Host id="poseidon">
      <Service name="GlusterFS" id="gluster"/>
      <Service id="IDK" depends="gluster,somethingElse"/>
    </Host>
    <Host id="zeus">
      <Service id="somethingElse"/>
    </Host>

And it would generate a chart grouped by host, with arrows connecting any depends-marked elements. It would also validate the resulting graphs, so you'd get feedback if you missed declaring a service you depended on, and so on.

Conclusion: was a fun "paid side project", but it's way too much effort. In retrospect using a proper XML (or even simpler markup) engine might have been a bit quicker and did the job just as well. Plus, the dependency on JS tooling at "runtime side" is very annoying.

This is beautiful. I never thought of chaining the parenthesis-free syntax or using table arguments for a sequence like that.

You can accomplish something similar in C++: single argument constructors let you skip the parentheses but that single argument can be an initializer_list, for example 'Tag(std::initializer_list<Tag> children)'. Now create subclasses of Tag for actual tags and you end up with a DSL that looks just like the one in the article. There is of course a bit more to it, but it can be done.

How would you do attributes?

You could use the builder pattern:

Div { H1("Title") }.Attr("class", "footer")

Then you dont really end up with a DSL that looks just like the one in the article, though – this is noticeably less nice :)

The best I could think off were designated initializers like:

    Div{{.class = "footer"}, B{"some text"}}

But making tags, attributes and plain text work nearly as nicely as in the lua table example seems not super straightforward to me.

Skip the parentheses? Do you mean like

    H1 x = "Topics";

Or is this an aspect of C++ I don't know about?

Yes exactly. And now if the constructor of your element, let's say a div, doesn't take a string but a std::initializer_list you can write it like this:

  auto doc = Div {
    H1("Topics")
  };

Those look like parentheses though. Also braces, but the Lua version has those too.

While there are a few DSL that won the war, like CSS or SQL, it is very unlikely that your DSL will prove worth it. 99% of the time, what you need is a good high level library/API to deal with the problem with your current technology.

Here is why:

First, you need to design it properly, and most devs are not good language designers. The chances your DSL is going to be good is very low.

Second, unless you are incredibly talented and have little constraints, you will not even remotely provide 1% of the tooling existing languages already have for free. I've seen so many DSL without basic syntax highlighting nor linter it's not even funny, but you need much more to be productive with a new language. Even if you are using macros and claim to be able to use the tooling for the main language you come from, stop a second and think about the last time you had fun debugging a complicated macro magic.

Third you need to test it and document it. Do you even do it properly for your regular code? It takes time and resources. A lot of it. Which could be better spend on the project, or on the library/API you should be writing about.

Fourth, what about training? Because you can hire somebody for language X. But for your DSL, you'll need to train them.

Then of course, you plan to support it for as long as it's used. Right? Right?

SQL is not a “DSL”.

It’s a “Declarative Programming” language and arguably the most popular one.

https://en.m.wikipedia.org/wiki/Declarative_programming

a language being a DPL doesn't preclude it from being a DSL

I’m not certain that is true.

With a DPL, you describe what output you want without describing how to produce the output.

With a DSL, you have to describe how to produce the output.

No, there is no requirement that a DSL be imperative rather than declarative.

Declarative, functional, imperative, stack-based, array-based, etc is a description of the language itself, or rather the primary computational abstraction; DSL is a description of its usecase and purpose. The opposite is a general-purpose language. The two concepts are orthogonal.

For example, your Wikipedia page defines regexes as also a declarative language, and yet it is also the quintessential DSL.

And in terms of it's domain... SQL has a very a restricted domain it was meant for, and is generally used for. You could argue its domain has expanded slightly with things like LINQ, but it's clearly wrong to describe it as an alternative to say python, or c#, as a language, in purpose or usage.

IMO, your comments belie the entire experience of library developers in "DSL-friendly" languages; those with a system of language features that compose well to create what humans perceive as "sub-languages". Ruby and Haskell come to mind immediately, each with a very different set of DSL-enabling attributes.

"The chances your DSL is going to be good is very low." – Nonsense? Yes, API design can be hard. But DSL design at the scale of embedded, ergonomic DSLs is most commonly exactly the same scale as API design. Moreover, there are plentiful examples to draw from these days, especially around builder-type APIs as illustrated in the article. We have quite a suite of well-established patterns to draw from, and they're just not that hard to apply well.

Notably, "language design" doesn't come into it. Almost all proper "language design" decisions for these kind of DSLs are made by the host language – that's why embedded DSLs are so appealing in the first place. There are exceptions when the "language" is really a language (e.g. embedding a Prolog in Haskell or whatever), but most of the DSL examples in the wild end up simply being nice ways to construct rich data structures for consumption/interpretation by some utility library.

Good time to urge people to comment on the article, not on a term you saw in the headline.

> 99% of the time, what you need is a good high level library/API to deal with the problem with your current technology.

TFA exactly describes a solution based on current technology available in Lua.

> First, you need to design it properly, and most devs are not good language designers.

A lot of languages are small and obvious, like the one described in TFA.

> Second, unless you are incredibly talented and have little constraints, you will not even remotely provide 1% of the tooling existing languages already have for free. I've seen so many DSL without basic syntax highlighting nor linter it's not even funny, but you need much more to be productive with a new language. Even if you are using macros and claim to be able to use the tooling for the main language you come from, stop a second and think about the last time you had fun debugging a complicated macro magic.

TFA addresses this not by macros or an entirely custom language, but by using Lua grammar for it entirely.

> Third you need to test it and document it. Do you even do it properly for your regular code? It takes time and resources. A lot of it. Which could be better spend on the project, or on the library/API you should be writing about.

"You need to test your code" is not a problem unique to DSLs

> Fourth, what about training? Because you can hire somebody for language X. But for your DSL, you'll need to train them.

I've skimmed the article for a total of maybe 45 seconds and I know the ins and outs of the DSL it describes. "You need to understand your code to work with it" is again not a DSL specific problem.

> Then of course, you plan to support it for as long as it's used. Right? Right?

Another code-in-general remark.

Completely agree with your replies here, but there are internal DSL's that stray so far away from the regular syntax/semantics that your skills in the host language don't apply when writing in the DSL. (Scala SBT I'm looking at you)

I think that the syntax of a DSL should make a problem easier to work with, like CLISP's format macro. Yeah it's kind of crazy having to learn a whole mini-language just to use a certain library, but my experience with CLISP assures me that this madness is worth it. I prefer to manipulate strings using format rather than try to do it in a real LISP.

That is the difference between an internal and external DSL. See https://martinfowler.com/dsl.html

I don't see the difference.

Reply on news.ycombinator.com