Decades ago I was quite enthusiastic about macro systems, but the experience of TeX programming made me realize that sometimes plain old functional or applicative abstractions are better than macro based ones no matter how domain specific they are.
Decades ago I was quite enthusiastic about macro systems, but the experience of TeX programming made me realize that sometimes plain old functional or applicative abstractions are better than macro based ones no matter how domain specific they are.
A macro lets you build a domain-specific abstraction that not only is extremely readable (matching the domain), but also ensures you just can't produce things that don't make sense domain-wise. In this way, such abstractions are much better than boilerplate you'd have to do with classical languages (though Haskell-like languages can probably get a similar effect through judicious use of types).
Except that over-enthusiastic use of macros replaces well-understood idiomatic boilerplate with project-specific magic.
(disclaimer - I love macros, used sparingly)
If I understand correctly, you use macros to change the syntax of Lisp. But if I'm trying to write a DSL, why would I need new syntax? Why aren't new functions and data structures enough?
(foo (bar) (baz))
If foo is a function, then (bar) will be evaluated, (baz) will be evaluated and then (foo X Y) will be evaluated where X and Y are the results of evaluating (bar) and (baz).If foo is a macro, then none of the above is necessarily true. Macros let you implement new control-flow constructs, which may be necessary for a DSL (or at least a low-boilerplate DSL; you can always make ugly control-flow by wrapping every expression in a lambda, but the idea is to make something easier to write, not something harder to write)
Why the functions and structures might not be enough is that there may be situations where their direct use is too difficult or error-prone somehow.
It can make sense to start with the new functions and structures first and then see if some usage pattern calls out "I want to be a new macro".
It's not that different than abstracting through functions or objects, but it allows you to abstract away the repetitive code structure as well.
The difference is how it's arguments are evaluated, for example if I wanted to make a function that implements a for loop of the form (myfor a from 1 to 10 do (print a)), if it's a function then every argument will be evaluated immediately, so it will try to find a variable called a, from, to, do and it will also try to evaluate (print a) immediately before looping. Macros allows for delayed evaluation, as myfor will receive all arguments (at compile-time) as symbols instead of values, which you can then rearrange in a form that can evaluate properly at runtime. You can also go further if you actually want to create syntax and use reader macros, which allows you to write your own parser and therefore escape writing directly on the AST (then you can even write a C syntax within Lisp).
And if your question is: do I actually need them? The answer is obviously no as many languages do not support it (and there are even alternatives for many use cases, like lazy evaluation). The advantage is that your language can have a very simple core and features that were not implemented (say a pattern match structure) can be added entirely within userspace (which is also good for testing new functionality before adding to the core language). Macros are also very efficient since they run at compilation (so if you use the macro a lot of time you only have to evaluate them once, unlike functions that will usually have to run it's logic every time it's called). And all of that means that for DSL, it's not just making a nice adaptation for your domain within the host language, but effectively writing an optimized language for your domain reusing the compiler of the host language without having to change it's source code.
Or is the idea that, as soon as I go beyond the Common Lisp standard, it's "domain specific", no matter how completely general my extensions are?
I've always interpreted DSL as creating a way to write programs in the language of the problem to be solved. A loop construct, no matter how useful, seems to fall short of that.
[1] https://en.wikipedia.org/wiki/Racket_features#Language_Exten...
It's not a direct comparison of course (nor it is a Lisp), but here is an example of linear optimization in a library that uses macros to make it a DSL closer to the description (in Julia @ before a name means it's macro, so it's easy to see) and one that uses methods:
https://nbviewer.jupyter.org/github/jump-dev/JuMPTutorials.j...
https://www.cvxpy.org/examples/basic/linear_program.html
If in the Julia example they if @variable was a function, then x >= 0 would have been evaluated immediately and it would fail since x was not defined (and if it was x >= 0 would return a boolean). To emulate that you'd probably have to pass a string "x >= 0", which the function would then have to parse (it would be a DSL as well, but one you're writing from scratch), the difference here is that you can just use the language parser directly and compile already with the result.
(my-if condition then else)
Would execute each of those before a call to my-if, so keeping it a function you'd need: (my-if condition (lambda () then) (lambda () else))
(bad example, don't make your own if, but illustrates the idea)2. You want to capture variables from the calling context, same issue as above. Consider the with-resource pattern:
(let ((foo ...))
(with-resource (lambda (r)
(do-something-with-resource r foo)))
Or the macro-d version: (let ((foo ...))
(with-resource (r)
(do-something-with-resource r foo)))
3. You want to do something that can be done efficiently in Lisp but uses very low level stuffs (like tagbody and go). Rather than writing that (repeatedly if this is a recurring pattern, like for various state machines) you can present a lispier syntax that compiles (via macros) down to the low level primitives (do a macro expand on some of the do constructs). See [0] for a variation of this idea in Scheme.4. You want to do something repeatedly and consistently, and want to remove errors. See the definition of defdot in [1]. You could define and register all those functions yourself for each now .<whatever>, or you could let a macro do the heavy lifting.
[0] http://cs.brown.edu/~sk/Publications/Papers/Published/sk-aut... see the linked PDF
You don't necessarily. But you'll need code generation to implement your domain-level abstraction.
Within-programming domain example is OOP. CLOS is essentially a bunch of macros that bolts OOP on top of base Common Lisp. It unified and standardizes what was many experimental flavours of OOP systems that were also implemented as macros. Or e.g. pattern matching and logic programming - both implemented as libraries for CL, via macros.
Outside programming domain, I can imagine working with software e.g. simulating chemical reactions, where you'd want to have atoms, molecules, reactions, energy exchange, etc. as top-level concepts. You can of course model all these as data structures, classes, helper functions, etc. But Lisp macros allow you to take that and close up the abstraction, by building a clean interface that doesn't leak the underlying machinery. On top of that, you can shift that machinery to compile-time execution (but also reuse it at runtime).
The advantages is that this reduces mental load when reading code; imagine if you had to parse a block of code and say "oh yeah, that's just a simple median of 3 quick sort" every time instead of just (sort ...). That would increase your mental load both when reading and writing.
On the other hand, when something goes wrong in sort (even if it's not a bug in sort itself, maybe some garbage was passed in), the fact that it's a function call actually increases the mental load when debugging. Good tooling that lets you print stack frames and such really improves things.
The same is true for macros, but things are worse in both directions since macros are more powerful. You can write better abstractions to decrease the mental load even more, but when things go wrong, more things can go wrong because macros are less constrained than functions.
Again, good tooling can go a long way to reducing the debugging pain. Stepwise macro expansion is a big win and being able to do it in-place is even better.
1: With the exception that lispers will use macros to prevent requiring explicit lambdas. For example, the WITH-FOO macros common in lisp can all be written using lambdas, and it would even be idiomatic to do so in many functional languages. From what I can tell this is originally because of the extra computational expenses of using lambdas, but persists because the syntax is more uniform with LET and friends