Gödel, Escher, Elisp: The Beauty of Macros | Hacker News Reader

Gödel, Escher, Elisp: The Beauty of Macros

chiply.dev

106 pointsJNRowe40 comments

> To anticipate a common question: why couldn't my-unless be a function? Function arguments are evaluated eagerly, before the function ever sees them.

Interesting, so if you’re using a lazy language then you don’t need a macro here and could write my-unless as a function.

Indeed:

https://hackage-content.haskell.org/package/base-4.22.0.0/do...

Yes, the D language has that as a feature in function arguments!

https://dlang.org/articles/lazy-evaluation.html

It makes it hard to know when things run. In Lisp you also have that problem everywhere, of course.

As the post shows this allows you to do stuff that looks like extending the syntax of the language.

I can’t decide if I love it or hate it!

Props to D for having such a simple implementation of fexprs[1].

The downside compared to macros I think is that, as argument expressions become lambdas, it becomes harder to manipulate them. Check the cond example which needs two fexprs, whereas one macro could do it.

Not sure about D, but if it were lisp, even lambdas could be manipulated as lists. Macro args not having the lambda wrapping just seems simpler.

[1] https://en.wikipedia.org/wiki/Fexpr

Walter Bright is against adding macros to D but D already has really advanced metaprogramming in other shapes:

https://dlang.org/spec/template-mixin.html

https://dlang.org/spec/traits.html

You could do that as well with lisp by passing around lambdas to functions but that adds unnecessary syntax.

Or you could wrap arguments in lambdas (anonymous functions) for eagerly-evaluted languages. Lisp unfortunately has a bulky syntax for lambdas, compared to something like Smalltalk. Of course, you can fix this with reader macros in Common Lisp (but no one does). Clojure has a shorter syntax I believe as well: #().

Unless laziness is pervasive (I guess, no knowledge about GHC internals), you basically have an FEXPR (https://en.wikipedia.org/wiki/Fexpr) which was replaced by macros for good reasons.

The Computational Beauty of Nature it's similar and it has the code of the book there:

https://github.com/gwf/CBofN

Some tools might crash X under CWM for OpenBSD, switch to FVWM or use some tool like mimalloc.

On Lisp, the book basically builds integers based on conses, kinda like Peano Axioms.

Remember the golden rule of Lisp macros: don't write a macro.

Meh, that seems to be a bit of a clojure thing. In the Racket world, with hygienic macros and phase separation, they'll routinely write macros returning macros, etc.

Macros returning macros happens in every Lisp, including Clojure. It’s no biggie. Hygiene has nothing to do with it and there are ways of properly managing unique symbols in all Lisps, even if some dialects are more manual than in Scheme/Racket.

I know, but the point is that in Clojure there's a bit of a culture of not writing macros unless you have to, while in Racket there's no such thing. Agreed that you can write macros just fine in other lisps.

The Clojure community is definitely less macro-crazy than other Lisp communities. But it’s also very programmer-specific. I’ve worked on OSS Clojure code where the original programmer loved macros. In general, I found it more annoying because reading through recursive macros is just more difficult than reading a simple function. So, I think it’s better for the community to de-emphasize macros, in general. Even if I want back to programming in CL or Scheme/Racket, I think I’d be much more judicious about using macros. Again, nothing against macros in general. They are a powerful tool in Lisp. But, to quote Spider-Man, with great power comes great responsibility.

Author here. I think in general this is sound advice, but in Emacs it is an incredibly convenient utility. I use them all the time. Mostly simple things like wrapping function definitions, or even other macros that wrap function definitions. Most of my lisp experience is in Emacs where the config layer is essentially exposed through an elisp API. In that context, I find myself using macros a lot. But if I was building some software from scratch, like say a data processing system, I can't immediately think of scenarios where I would define macros.

Worth mentioning in Elisp that even if you don't define your own macros, you use them more often than you would think. defun, unless, def-custom, etc (more examples in the blog post) are all macros. The fact they look and feel like non-macros like special forms and built in functions implemented in C is part of what makes elisp so cool to me. There are plenty of homoiconic languages, but you rarely feel the distinction between program and data in elisp, and macro-supporting lisps in general.

If you look at the definition of a macro like defcustom, you'll see that it just expands to a function call, where all the logic is implemented. This could also be implemented in the macro, but that is more complicated and brittle, due to the risk of double-evaluations and having to produce code that will (usually) later evaluate with the intended side effect.

Furthermore, macros and functions constitute a kind of function coloring. To use a macro in a function, like defcustom, you couldn't pass the name of the user option you are defining in as a symbol, as that is not evaluated. So instead you'd have to call eval on a runtime constructed expression, which is a cludge.

So I agree with the top comment, to avoid macros is a sign of Lisp maturity. It is easy and fun to admire them when you are coming from languages with arbitrary restrictions in their macro systems like C, but ones you get used to them you don't treat them with any more wonder than any other arbitrary restriction that a language may lack (bad ad hoc example: nobody praises C for the lack of a CALL keyword).

Why?

That is badly generalized proverb. Clojure has unhygienic macros, no expander extension points, and no error-message layer, so "don't" is a rational default there. Yet, for example Racket writes macros that write macros as a matter of course. And that's because the language has the infrastructure: hygiene by default, phase separation, `syntax-parse` with grammar-quality error messages, a module system that actually knows about compile-time dependencies.

But even with some limitations, there are cases in Clojure where macros do achieve things otherwise unattainable. One practical example is multi-host code generation. You can have a Clojurescript macro that parses and analyzes some javascript lib, doing that on JVM side, while emmiting code generated to run in Javascript. Hyperfiddle/Electric is the best example of that kind of macro ingenuity.

Or take core.async's `go`. It takes an arbitrary body, analyzes it into an AST, and rewrites it into a state machine so that <! and >! can park and resume. On the JVM it lets you avoid blocking threads. In Clojurescript it is the only way to have the model at all, because JS has no threads to block.

Nothing about that is expressible as a function. The transformation needs the entire body as data. Same family: core.match compiles a pattern matrix into a decision tree. And there are many more example use cases for macros.

"Don't write macros" rule has the second part: "unless you truly have no choice."

My own personal rule is that every Lisp programmer should understand macros and very rarely reach for them. They are a super-power, but easy to misuse. I think Paul Graham is partly responsible for the “Lisp = macros” thinking and the general overuse of them. On Lisp was a great book but it does lots of things I wouldn’t consider good practice today. After Graham popularized Lisp and macros, there was a period when every Lisp newbie, myself included, was writing macros all the time. Now that I’m older, I know better. Clojure is my daily driver and I haven’t written a macro in years (though I’ve debugged ones other people have written). But I know defmacro is there and I can use it anytime I really need to.

I don't think Paul ever envisioned Lisp evolving into something like Clojure, where macros indeed are a bit less than a first-class citizen. And I think the whole adage of "First rule of macro club - don't write macros. Second rule of macro club - don't dispute the first rule...", etc. became a thing only with Clojure. I don't disagree with you - genuine use cases for macros are uncommon. Yet it's an instrument - nothing's wrong with using it properly. Knowing when and how is a skill, experienced Lispers indeed - rarely reach for it.

Ok, common-lisp newbie here.

Could you explain why Paul Graham's book "On Lisp" is wrong? Or misguided? Or misunderstood by newbies like me? 30 years of c++ with a lot of dealing with templates and I'm not feeling the mandatory function purity here.

I don't think that is a reasonable request for a response to a comment post, but in my experience, many lisp macro advocates come from other procedural / OO languages and aren't well acquainted with other programming paradigms, so they don't generally know what the don't know about other styles, say functional programming, and how to solve problems using higher order functions, etc. instead of reaching for macros. Which is OK. Not everyone has to become a programming language dilettante.

Suppose I accede to every claim you make here. Where may I see the advantage in the long run, in any dimension? I'm willing to go with hearsay.

The word "dimension" is chosen to be so vague that you can surely do it. I would be easy to impress.

>Where may I see the advantage in the long run, in any dimension?

The advantage of what?

I agree! A Lisp coder should define a new macro only rarely. Junior coders should avoid doing it altogether.

Macros make Lisp code hard to read. Every time I see a call to define-minor-mode, a groan a little: that macro saves about 5 tokens (relative to just writing out code involving only functions everyone knows) at the cost of making me bounce back and forth at least 3 times between (a buffer showing) define-minor-mode's doc string and the call.

Why is most AI work done in Lisp? There are many reasons, most of which are somewhat technical, but one of the best is quite simple: Lisp is crisp. Or as Marilyn Monroe said in The Seven-Year Itch, "I think it's just elegant!"

This claim didn't age well.

Haha, this is very true. Symbolic AI is dead for the most part, but I feel like it was an issue of labor scale. There weren't enough code fluent humans to embed all the knowledge. I do think symbolic AI will see a resurgence though. I could see it being a part of enrichment to current models. I don't think it's the case that we'll see diminishing returns THEN reach for symbolic AI. I think that adding a symbolic layer could greatly enhance their effectiveness today and make the models cheaper (by being performant with fewer params and therefore cheaper to run and serve). You already kind of see clues of this as creating graphs / ontologies as part of a RAG or retrieval phase are proven to improve performance of these AI models in specific domains.

At the same time, back then the thought was that symbolic AI was the answer, and for sure Hofstadter's understanding of intelligence, and the emergence of consciousness from matter, had everything to do with symbols. He's talked about this, very interesting, although this is not a 'feel good' artifact from the Hofstadter canon lol. https://www.youtube.com/watch?v=R6e08RnJyxo&t=227s

I always thought it odd that Penrose gets dropped from the Escher conversations.

Can you elaborate? Roger Penrose and his father Lionel did independently discover and popularize the Penrose Stairs and Penrose Triangle but Oscar Reutersvärd had created both years earlier. What do you think should be said about Penrose?

Yeah, I think the story is that Penrose saw an illusion in an Escher print that encouraged him to create the Penrose Triangle, his dad Lionel created the Penrose staircase, and then sent a copy of these in the paper "Impossible objects..." to Escher, who then used the staircase in "Ascending and Descending" and others. It's kind of cool that they cite eachother in these works as inspiration here. I bet Escher could have made an interesting print of them pointing at one another in some kind of impossible object.

And apparently neither Penrose nor Escher knew about Oscar Reutersvärd's work! Oscar got robbed lol.

Yeah, it’s odd that Escher didn’t respond to Reutersvärd’s letters. Also, Penrose found out about Reutersvärd’s work in 1984:

>It wasn’t until 1984 that Roger Penrose (whose father Lionel died in 1972) discovered that Oscar Reutersvärd had invented the triangle and the stairs much earlier.

https://escherinhetpaleis.nl/en/about-escher/escher-today/os...

In 1954, Escher inspired Roger Penrose to create the impossible triangle. Roger shared this with his father, Lionel, who then designed the impossible staircase. In 1958, they published a paper on these illusions and sent a copy to Escher. Escher used Lionel's staircase to create Ascending and Descending and Roger's triangle to create Waterfall. Both Penrose and Escher were completely unaware that Oscar Reutersvärd had independently invented these shapes earlier.

If macros can implement arbitrary language features then how come elisp has never built a type system?

Well, not elisp, but

https://coalton-lang.github.io

or maybe

https://shen-language.github.io

Typed racket too, might I add. I just imagine static typing around stable core APIs would help users understand how eglot and vertico compare to prior art. The way I learned how the completion stack works was to ask claude to show me what the elisp entrypoints would look like if implemented in typescript. It's not because I'm unfamiliar with lisp, I'm not, it's just easier to reason with the world in terms of data structures and their contracts.

CL is the big cousin of Elisp, almost.

I would expect you can.

lexilambda created Typed Racket on top of Racket. https://github.com/racket/typed-racket

Another approach, using macro expansion to perform the type checking: https://docs.racket-lang.org/turnstile/

Turnstile relies on the fact that in Racket, you can have identifier macros [1] which can be expanded in pretty much any location (except binding, naturally). Racket is unique in this regard. Clojure has a mechanism to get roughly the same thing, but it's complicated. [2]

1: https://docs.racket-lang.org/guide/pattern-macros.html#%28pa...

2: https://lambdaland.org/files/2024_ecoop_type_tailoring.pdf

https://typedclojure.org/

> If macros can implement arbitrary language features

Not really your question--others have provided examples of type systems as macros--but they can't, they can only do local rewrites. You couldn't, for example, express mutation or non-local control flow if you didn't have them already; On the expressive power of programming languages [0] is rather theoretical but does set out the boundary of what macros can and can't do. That they're local means everyone has to buy into your type system if you want it to be a macro, though.

[0] https://www.sciencedirect.com/science/article/pii/0167642391... though https://www.youtube.com/watch?v=43XaZEn2aLc might be a useful walkthrough of the ideas being presented

Reply on news.ycombinator.com