These sorts of languages/language extensions pop up all the time in "ordinary" programming. For example, common DSLs we see often.
- regex is a language for string patterns
- JSON is a language for simple nested data
- SQL is a language for expressing DB queries
- OpenMP contains (essentially) a language for parallelizing loops
- (this might sound crazy because it's so common and basically built in standard in almost all programming languages:) infix notation for math
The problem is that many of these are absolute kludges in existing languages. Anybody who has had the displeasure of trying to interpolate DB table names into SQL SELECT strings knows this. Most languages try to make it a bit better by adding preprocessors, using builder-patterns, abusing reflection, or otherwise. Looking under the hood of some languages' implementations of these ideas will often lead you to see the most cursed of programmer things in existence.
Each of those languages (or close syntactic cousins of them) can be written directly in Lisp, as a library, without shoving them into strings, or using external tools, or making custom file formats. They call can exist as a part of the Lisp language as a syntax extension.
Making DSLs requires exercising that muscle a bit; it's non-trivial to design a new and useful language that fits orthogonally into an existing system.
It's like being introduced to OO programming where, at least for me, it made sense but it wasn't immediately obvious what a good object model for a problem might be. It took me lots and trial and error to understand how OO works in "the real world", where it's good, and where it falls short.
Once you get they hang of making DSLs—or even just small but powerful syntax extensions—it's an amazing tool to have at your disposal.
Maybe a simpler example, what if I wanted this in (pseudo-)C?
with(s = fopen("/foo/bar", ...)) {
...;
...;
println(readln(s));
...;
...;
}
In Lisp, this is a feature you could build and provide as a library feature. In fact, we can do it, straight here in an HN comment.
(defgeneric destroy (x)
(:documentation "A destructor for use with the WITH feature".))
(defmacro with ((var = val) &body body)
"Python-style RAII macro. VAR will be
bound to the value of VAL, and upon
exiting scope, will be destroyed with
DESTROY."
(assert (eq = ':=))
(let ((orig (gensym)))
`(let* ((,orig ,val)
(,var ,orig))
(unwind-protect
(progn ,@body)
(destroy ,orig))))
(defmethod destroy ((s stream))
(close stream))
Now, we effectively have RAII in Lisp:
(with (s := (open "/foo/bar" ...))
...
...
(write-line (read-line s))
...
...)
This is a concept we can use almost immediately after learning about it, instead of waiting for Python's PEP 6363 to pass with consensus from people who may have different programming goals than you do.
https://github.com/norvig/paip-lisp/search?l=Markdown&q=defm... - all references to defmacro in the markdown files
Chapter 3 shows a simple macro, just adding a while loop to the language.
Chapter 9 shows some more complex ones, including a with- macro and a grammar compiler macro.
Chapters 11 and 12 show the development of a Prolog implementation in CL using defmacro to aid in compilation again in Chapter 11.
Chapter 12 shows adding an OO system to the language. Technically not needed with CLOS, but a good demonstration of what can be done with macros.
There are other examples (why I included that search link). Macros let you change the language in ways large and small. Many uses could probably be replaced with functions, though you'd end up having to throw a bunch of quotes about or closures in order to delay processing things. You'd also be delaying that to runtime, which incurs its own penalties compared to macro expansion and compile time (if a compiled CL).
A well written function must also abide by some promise or specification—contracts on the inputs and invariants on the outputs. I don't see why new syntactic structures are much different from that.
I think there's undue spookiness around macros among lots of programmers, especially those who haven't sought to solve some problem with them.
- "It's too powerful for the common person."
- "It's way too likely to make code confusing."
- "It's invariably too hard to reason about."
- "It won't work on a team because everybody is going to make their own weird incompatible language and it's just not scalable."
All of these sentiments are complete bunk, and I argue are against common wisdom around other programming abstractions, like functions or classes.
And a lot of what tends to be macros in Lisp turns into functions in eg Haskell. Eg you can define your own new branching and looping constructs (like 'if' or 'while') etc as functions in Haskell, but in Lisp these need to be macros.
In Lisp, you can also define branching and looping as functions, but you need to concede and have the arguments themselves be (lambda) functions.
(defun while (c body)
(when (funcall c)
(funcall body)
(while c body)))Yes, if you wrap your arguments in lambdas, you can write loops like that.
Though to be as performant as the built-in constructs, you need a rather clever compiler that 'removes' the lambdas again.
I wrote a post about this here[1]
[1]: http://tech.perpetua.io/2022/01/generating-sqlite-bindings-w...