The Problem with Macros
ianthehenry.com
ianthehenry.com
(defun f () 1)
(defmacro m () `(f))
(defmacro n () `(,#'f))
(defun g () (m))
(defun h () (n))
; later
(defun f () 2)
(defmacro n++ () `(funcall 'f))
Yeah, that is an important issue. What do you think of the following solution?
1. The semantics of the language are that, every time a function is called, any macros in its body get reexpanded.
2. The language implementation is expected to notice that, in normal cases, neither the macro nor the functions it calls have been redefined, and therefore the previously-expanded (and -compiled) version can be used; but then to notice when that changes (i.e. when you do redefine something) and generate new code accordingly.
The feasibility of 2 I'm not sure about.
Edit: Regarding the printing of functions, I don't think it would be that difficult to make the pretty-printer, upon encountering a function object, detect whether that function is bound to a global variable, and if so, print the name. (Possibly the name with a "f:" prefix, or in a different color.)
[1]: https://common-lisp.net/project/slime/doc/html/Cross_002dref...
> (print #'print)
#<FUNCTION PRINT>
Ironically, the author of the article was correct in their view that he initially described as wrong. Macros are just functions, typically taking code and outputting code. They just didn't understand the evaluation rules surrounding macro invocation.
It seems like Common Lisp has done quite a lot to solve this problem already, which is why it is a non-issue in Common Lisp.
> non-externalizable, non-PRINTable function objects
Yes, this would be a very annoying technique in a language with non-externalizable, non-PRINTable functions.
> This is contrary to Lisp’s DNA of interactive development and is a big no-no.
Redefining functions interactively is actually a great motivating case for cross-stage persistence. Instead of interpolating the value of the function, interpolate the lexical binding of the function itself, and then redefinitions will be reflected automatically in any expansion of the macro.
Given that many useful functions are actually closures over lexical environments possibly shared by other functions, and compiled to native code, I’d be impressed to find a language where such objects are printable and externalizable. I don’t think they are in Janet.
The same problem exists with lists (or any other mutable object). If x and y point to the same list, and you print x and read it back in, and then modify the list x points to, then (naively) it won't modify y. If you did want to preserve such structure sharing, one approach would be to print the entire environment and make liberal use of #n= notation. (I've occasionally done things like this.)
> compiled to native code
Assuming compilation is a deterministic function of the source code and the environment, and assuming the user hasn't changed the environment, it seems printing the source code should suffice, and the runtime can redo the compilation when necessary.
Lists do have issues of sharing data with other lists, but most times the lists that are sharing data are somehow adjacent to one another allowing the printer to collect shared references.
If we allow ourselves the freedom to ignore shared closure environments, L-T-V and other Common Lisp complications, then serializing a function would still be this very expensive recursive procedure to serialize all transitive callees to capture any possible shared state, and I think that is too much to ask for, especially in the context we are speaking (making functions appropriate objects to be printed as a part of an S-expression).
It’s all doable if everything is slow, interpreted, and first-class though. Or maybe you’re serializing FORTH words. :)
Surely these offsets refer to items saved in the lexical or global environment, which were originally named by variables? Then you serialize the variable references and the values they refer to. A shared lexical environment would get serialized like this:
(let ((x 10) (y '(1 2)))
(let ((f (lambda (z) (set! x z))))
(let ((g (lambda (a) (list x y)))
(h (lambda (b) (f b))))
(list g h))))
; serializes to the following,
; with the notation #closure(args body env)
; where env is ((var1 val1) (var2 val2) ...)
(#closure( (a)
((list x y))
(#1=(f #closure( (z)
((set! x z))
(#2=(x 10) #3=(y (1 2)))))
#2#
#3#))
#closure( (b)
((f b))
(#1#
#2#
#3#)))
Yes, most of the time, when you print lists, either there is no shared structure, or the program doesn't try to modify it, so it doesn't matter if you read a portion back in and bifurcate the identity. Likewise, for the majority of functions, either they don't share their lexical environment with others, or they do but not in a way where it matters. (This thread's original use case was printing globally defined functions, which usually have no lexical environment.) The common case should work fine like this:
(let ((n 0)) (lambda (x) (incf n x)))
; serializing to
#closure( (x) ((incf n x)) ((n 0)))
The question upthread seemed to be whether it was possible to have a language that could serialize functions. I've argued yes. Now the counterargument seems to be "well, it's inconvenient and expensive to get full fidelity". This brings us to the question of what use cases we're talking about, and how often you want full fidelity (and what assumptions you can make).
I think one of the use cases was printing macroexpansions in the REPL? And then maybe the readability-requiring use case would be selecting a subexpression and saying (macroexpand-1 '([paste])). The literal functions in the expansion would generally be globally defined—the result of using ",+" instead of "+". Well, I think that's covered reasonably well by having the globally-defined function "foo" get printed as #f:foo or something like that. (Maybe we could use the syntax #'foo to mean that? Hah.) It's less nice than seeing the bare name, but macroexpansions already often contain gensyms and package prefixes.
> (define f (let ((x 5)) (lambda (y) (+ x y))))
f
> (f 4)
9
> (object->string f :readable)
"(let ((x 5)) (lambda (y) (+ x y)))"
That makes me severely disinterested and skeptical about hygienic macros.
They are too weird for the little benefit they provide. You can't look at a piece of code and know what it expands to.
Hygienic macros break lexical scope, because code produced by a macro being invoked in some file "A.lisp" is seeing lexical variables defined in some different file "B.lisp". Lexical scope must be physically enclosed and contained. If you expand it here, it sees the variables that are here, and not some other ones elsewhere that you don't see here.
Given a nesting like (x (y (z a))), if y binds a, that must be like a brick wall; there is no way that the a reference in z can get around it, to connect with a binding of a produced by x. Either that a refers to z's own binding, or else to y. Yet hygienic macros can perpetrate a lexical-scope-destroying wormhole which can do that: the inner a reference can bypass a definition set up by y, and go to the one in x.
It's a downright security issue, like going around your company's firewall, or escaping a container or chroot jail or whatever.
def foo(str):
v = str(1)
. . .
EDIT: I agree that hygienic macros aren't the right way to solve this issue.
CL-USER> (defun foo (list) (list list))
FOO
CL-USER> (foo 42)
(42)
To further illustrate, in the above example the LIST symbol is imported from the package COMMON-LISP and has a function, plist etc. attached to it:
CL-USER> (symbol-package 'list)
#<PACKAGE "COMMON-LISP">
CL-USER> (symbol-function 'list)
#<FUNCTION LIST>
CL-USER> (symbol-plist 'list)
NILThe "certain systems programming language" with its unhygienic macros hides all the resulting horrible mess in reserved symbols that litter both the macro code itself and the code produced by the macros. This is perhaps tolerable so long as you're not maintaining any of those macros and you never need to look at the code after macro-substitution has occurred.
An easy way to see this is: dynamic binding is to environmental variables what lexical binding is to command line arguments.
The unit testing macros for my Racket project were created so I could create snapshot tests for an API. The custom test-suite and test-case macros set parameters (think React context properties) for the suite and case names. There is a custom `check-snapshot` function which save snapshot files under ./snapshots/suites/{suite-name}/cases/{case-name}. It evaluates each expression and pretty-prints the results separated by newlines. It can use an env var flag to fail any test if results differ from the file, but I usually just always update snapshots and check in Git to see if/how they have changed.
One day I'll improve `check-snapshot` to copy the unevaluated expressions into the snapshot file too so they're very easy to review. This will require it to become a macro. The evaluated expressions are generally 1-5 pretty-printed lines long and stateful sequences of expressions, so it's different from React snapshots which can be pages long to display one value.
Anyways, I'm unsure if Racket macros suffer from the problem described in the post. If I'm not mistaken, this problem is kinda like a problem with unhygienic macros only? The idea of hygienic macros is that identifiers are never captured from the environment where they're used.
In Racket there's also phase separation. The tests.rkt file in my project actually has two test-suite identifiers. One at phase 0 referring to the custom test-suite macro, and one at phase 1 referring to rackunit's test-suite. Good thing the macro isn't recursive, I guess...
If you type macrology into Google you get the definition "Long and tedious talk without much substance; superfluity of words. noun." Perhaps accurate, but too vague ;-)
[1]: https://www2.ccs.neu.edu/racket/pubs/scheme2007-ctf.pdf
https://duckduckgo.com/?t=ffab&q=matthew+flatt+macros&ia=web
One minor typo (I thought it was pretty funny): “it seems like it was actually a performance hack in order to make a dynamically typed language performant on the hardware of the early 1890s.” Or, did Babbage’s machine run Lisp the whole time?
(Thanks for the kind words. It was meant as a joke, but it's hard to pick the right decade to balance the funny:confusing ratio :)
Many times these are done through what is known as manual work. In my experience if someone is complaining about laborious manual work. Lack of quick automation, they are basically in a scenario where the language lacks features to generate code to do work, which they have to now do instead.
Its for this reason I have to write adhoc perl scripts to generate shell commands, and sometimes even whole shell/python snippets/mini-programs. All because Perl makes it easy to generate text(code). Perl now acts as a macro facility for Shell/Python.
If you pay close attention these patterns will begin to show all up over the place.
It’s interesting, I’ve spent a lot of recent personal interest time deep diving into code transformation in JS/TS. These concerns about name collision and shadowing are front and center there, but of course tree-walking AST transform is the ~only game in town[2] and it’s had me pining for first class language macros.
1: The first that comes to mind https://github.com/reup-distribution/espalier
2: I have some ideas on this, but nowhere near fully formed enough to elaborate just now; suffice to say they derive a lot from what I do know about macros.
It’s still possible to shoot yourself in the foot (see https://blog.danieljanus.pl/2020/01/21/middleware/ for my personal story), but for that you need `binding`, `with-redefs` or `alter-var-root`, which are all relatively rarely seen.
At least I'm sure it is by fexpr because with them you have complete control over evaluation.
For those who don't know, fexpr are closures (i.e., lexically scoped functions) that are called with unevaluated arguments (just like macros) and are implicitly passed the dynamic environment (i.e., the call site environment). That way you have all the powers of macros and functions with fexpr, and no scoping problem.
But for the capturing problem mentioned in the article, I'm quite sure hygienic macros are enough.
Paul Graham mentioned this (EDIT: something like this, but probably with a different answer) once in http://www.paulgraham.com/ilc03.html :
> What happens when a Common Lisp macro returns a list whose car is a function? (Not the name of a function, mind you, but an actual function.) What happens is what you'd expect, in every implementation I've used. But the spec doesn't say anything about this.
(EDIT: I think "what you'd expect" actually means "raising an error" here. Either that, or implementations have changed since 2003, which is also possible.)
I do suspect that this is the nicest way (from a theoretical standpoint: requiring the least additional complexity in the language semantics) to get "macro hygiene". Then you probably want a different syntax from quasiquote, because otherwise I think you'll put commas on the majority of the symbols. But using "list" a lot (and "append" when you need unquote-splicing in the middle of an expression) isn't very nice. So probably a new syntax is warranted.
Printing primitive functions could be something like #<primitive:+>, and when an implementation read in a thing like that, it would look in its own set of primitive functions for one named '+ and use that; with a reader macro I imagine you could make that exact syntax work. (In fact, you could print #<primitive:+> on an x86 machine and read it on an ARM machine, and this could be fine.)
I honestly don't see why implementations haven't done this, other than "it was more bothersome than worthwhile".
In Common Lisp, what about LOAD-TIME-VALUE?
Recursive printing of the environment just to serialize a single function seems atrocious and hazardous.
Not sure what the author means by "no notes". The transcript they posted does point to both the exact chapters in both the implementation and standard docs:
; See also:
; The SBCL Manual, Node "Package Locks"
; The ANSI Standard, Section 11.1.2.1.2The author was being sincere, rather than sarcastic as your initial parse suggests
This is the section of the standard the error message points at: http://www.lispworks.com/documentation/lw71/CLHS/Body/11_aba...
And this is the reasoning for why that section exists, linked to from that page: http://www.lispworks.com/documentation/lw71/CLHS/Issues/iss2...
"Underestimating the hyperspec" is a persistent mistake on my part sadly, although I at least seem to underestimate it less these days.
The author here is not saying that the message has no notes. They are saying that they have "no notes" to offer the creators of the message on ways to improve it
See, for example: https://forum.wordreference.com/threads/no-notes.3042292/ https://hinative.com/en-US/questions/15238431 https://www.youtube.com/watch?v=5NVQ8v1P4go
People use callback function to achieve almost the same thing
function doTexture(texture, fn) {
beginTextureMode(texture)
fn()
endTextureMode()
}
doTexture(myTexture, () => {
drawCircle()
drawRectagle()
})
I guess why this is less popular in lispy languages is because
1. macro
2. callback functions introduce more indent levels :(> we get the benefit of one thing less to learn
But learning programming language theory is the best part :)
> But learning programming language theory is the best part :)
That's very true :)
There's a QoL difference here with macros - writing out those lambdas can become annoying. That said, the "good style" rule in (Common) Lisp is to prefer lambda-forms in such cases - i.e. cases where the macro parameters are a block of code to be mostly run straight.
In fact, a common pattern for with-macros (of which doTexture would be an example), is the call-with pattern. Example from some random project of mine:
(defmacro with-logging-conditions (&body forms)
"Set up a passthrough condition handler that will log all signalled conditions."
`(call-with-logging-conditions (lambda () ,@forms)))
(with-logging-conditions
(blah blah)
(main game code))
A better use of macros, which you can't replicate in JavaScript[0], would be if you wanted to do something like:
(do-texture texture
o O r R x2)
;; unwind-protect is Lisp's sorta-equivalent of try/finally in other languages.
(unwind-protect
(progn
(begin-texture-mode texture)
(draw-circle)
(draw-circle :big)
(draw-rectangle)
(draw-rectangle :big)
(draw-rectangle :big))
(end-texture-mode))
--
[0] - Well, you can if you have a toolchain. Babel is essentially a macro engine for JavaScript, but you can only use it at build time.
This might be another reason why I find macros less appealing: macros introduce DSL in form of normal s-expression, but they don't actually behave like a function; macros introduce their own mini-language/syntax.
In the last example you provided, I bet the macro implementation would look like a little interpreter? If that's the case, having a function call like
doTexture(myTexture, ['op1', 'op0', 'opR'])
I'm not trying to argue that macros are unnecessary, I really want to like them! Just most of the time, I find functions are sufficient enough.
So "modern" languages without macros have such patches every now and then. Still, it's easy to find boilerplate in programs.
The reason is that a lot of boilerplate is specific to the program's domain, and you're left with cumbersome syntactic patterns and no tools to abstract them.
So yes, you need to learn how to write macros. But, assuming good taste, the simplicity is in their use, not their definition. It's a good tradeoff because macros are used much more often than they are defined.
By the way, the language introduced by a macro often allows arbitrary Lisp code to be mixed with it. The example did not demonstrate this, and that's why you had an easy time thinking up a non-macro alternative (which still has some unfortunate implications, like needing to interpret at runtime).
> macros introduce DSL in form of normal s-expression, but they don't actually behave like a function; macros introduce their own mini-language/syntax*
That's the point. S-expressions are structure notation language. The semantics of code expressed as s-expression is something else. There's the "default" one (as provided by #'eval), but macros allow you to work on the s-expressions as data structures. Ultimately, the macro expansion is still evaluated normally, but the expansion might be wildly different from the macro invocation.
This is a feature, not a bug. It gives you the power to add new abstractions to the language, as if they were part of that language in the first place. It's not something you need often, but there are things you can't do any other way. For example, since we're talking JavaScript - think of JSX. In JavaScript, it's a mind-bending innovation, though committing you to use a big code generation tool (Babel). In Lisp, you can do half of it with a macro[0].
A common use of macros is removing conceptual repetition in code. Imagine you have a concept in your codebase - say, a plugin. Creating a plugin involves defining a class extending a common base class, defining a bunch of methods that are identical in 90% of the cases, and a bunch of free functions. Conceptually, that whole ensemble is "a plugin". Lisp macros let you define that concept in code, and reduce your plugin definitions to just:
(define-plugin some-name
:some-specific-method (lambda () ...))
Such macros are more like compilers. Interpreters execute the code they read; compilers - like those macro - emit different code instead.
> might be able to achieve the same behavior, right?
Yes, except the macro does that at expansion time - i.e. ahead of execution. In practice, this is almost always "compilation time".
> Just most of the time, I find functions are sufficient enough.
Because they are! Even in Lisp, macros are not your default tool for solving problems. Functions are. Macros come out when the best way to do something involves code that writes code for you.
--
[0] - And the other half with a reader macro. Regular macros transform trees before they're evaluated. Reader macros alter the way text is deserialized into trees. Reader macros are very rarely used, because they're a bit hard to keep contained and tend to screw with editors, but if you really want to create a different syntax for your code, they're there for you.
You could just fix the bug?
Game development is a different architecture to 'regular' system development (especially if it is a solo project).
If you are testing to ensure the bug isn't reintroduced then you haven't fixed the bug.
It’s called regression testing, and it’s a (fairly) common thing.