What are lisp macros for if not to delay evaluation? I assume you're making a distinction between purpose and mechanism here?
I've always understood the semantics to be:
fn: (a b c) => (call a (eval b) (eval c))
mac: (a b c) => (eval (call a b c))
I've always understood the semantics to be:
fn: (a b c) => (call a (eval b) (eval c))
mac: (a b c) => (eval (call a b c))
https://en.wikipedia.org/wiki/Fexpr
When an FEXPR is called, it gets unevaluated args and can then at runtime decide what to do.
Macros OTOH are a different mechanism, where expressions get rewritten at macro expansion time, which can be for example at compile time. The macro then gets called with arbitrary expressions it encloses (those don't need to be valid code by themselves) and computes new source.
Thus macros are code generators from expressions. In a compiled implementation, macro expansion is done with compilation interleaved: each form gets expanded, even repeatedly until it no longer expands into a macro, and then the resulting non-macro form gets compiled.
Thus in a way a macro does not delay execution, it does the opposite: it actually shifts computation to compile time -> the computation of code from expressions and the computation of arbitrary side effects in the compile-time environment.
In an interpreter version of Lisp, the macro gets also expanded at runtime - but in its own macroexpansion during evaluation. There eval will call the macroexpander repeatedly until it gets a non-macro form.
Now, what can you do with arbitrary complex code generators at compile time? https://stackoverflow.com/a/2563308/69545
Actually Paul Graham wrote a classical Lisp book explaining a bunch of things around macros. Available here for download:
http://www.paulgraham.com/onlisp.html
The classic Lisp article which motivated the move from FEXPRs to macros:
Not sure if this helps you. But let's define a macro A:
CL-USER 32 > (defmacro a (b c)
(print (list :macro-expansion b c))
(list 'print (list 'quote (list :runtime :b b :c c))))
A
Now we can use this macro in some code:
CL-USER 33 > (defun test ()
(a 21 42))
TEST
CL-USER 34 > (compile 'test)
(:MACRO-EXPANSION 21 42)
TEST
NIL
NIL
CL-USER 35 > (macroexpand '(a 20 30))
(:MACRO-EXPANSION 20 30)
(PRINT (QUOTE (:RUNTIME :B 20 :C 30)))
T
CL-USER 36 > (test)
(:RUNTIME :B 21 :C 42) ; <- the printed output
(:RUNTIME :B 21 :C 42) ; print also returns its arg
Thus it has to do with code generation and code execution at macro expansion - nothing about 'delaying' something.
The example isn't useful, but imagine a macro INFIX
(infix a * b + c)
(+ (* a b) c)
And most practical uses of macros involve passing in expressions that will be later evaluated verbatim, e.g. (time (reduce + (range 100)))
I was only intending responding to this:
"That's a common misconception that Lisp macros are mostly used to 'delay' evaluation."
Which just seems to me confuses the matter in response to the article that was posted. At a very low level, macros operate by having the evaluation of their arguments delayed. At a higher level, they're used to implement sugared forms which usually contain valid code (though not necessarily code that makes sense in isolation).
I know you're very experienced with lisp, and I'm not trying to correct you technically, but to me this stuff only clicked when implementing a metacircular evaluator and learning when and how to perform evaluation. The blog post to me reads like the author was going through a similar process.
It doesn't 'delay' anything. It just rewrites at compile time
(infix (avg b) * (sgn a))
(* (avg b) (sgn a))
> At a very low level, macros operate by having the evaluation of their arguments delayed
Not at all. The main purpose of macros is to compute code from arbitrary expressions at compile time. There is no 'delay' -> it's mostly about rewriting expressions at compile time.
> implementing a metacircular evaluator and learning when and how to perform evaluation.
You can build a macro expansion phase into an interpreter. That's what Lisp interpreters do. But the main purpose of macros is to enable code transformations at compile time and thus only to have a cost at compile time, not at runtime. In an interpreter the macro expansions would also happen at runtime and have a cost there. But Lisp compilers are popular and to make the most out of code transformations at compile time, the idea of macro and their expansion at compile-time was introduced. Thus one gets for example new embedded languages, without the need to parse them at runtime.
I'm not arguing with you about what happens in any production lisp. I'm sure you know more than me, but I do understand how they're implemented in practice, and the warts that brings - eg no (apply or <list>), and first class macros are so esoteric purely from the runtime cost that they're barely even written about in the literature.
I'm just saying the mental model of where macros fit in in an interpreter has value, and that view has not changed. In my opinion you're an expert coming in correcting a bunch of people attempting to learn the basics, and you're not wrong, but you are muddying the waters.
If you mean run-time functions that act a bit funny about evaluation order, that mental model is incompatible with common introductory examples like `let`.
((fn [x]
((fn [y]
(+ x y)) 3)) 2)
fn: (a b c) => (call a (eval b) (eval c))
mac: (a b c) => (eval (call a b c))
Everyone jumped on it, as production lisps don't implement macros this way. But it's the natural way to implement them in a SICP style metacircular evaluator, and a good way to learn them.
This is about mental models, not implementation details. And in my opinion this thread is full of experts who have forgotten what it's like to not understand how macros work.
Everyone jumped on it, as production lisps don't implement macros this way. … This is about mental models, not implementation details
It looks to me like you're the one who brought up implementation details, talking about what happens "at a very low level." You're the one who has presented a questionable implementation strategy as a mental model and has been corrected on it multiple times by multiple people. Everyone else has been describing the semantics of macros. Having separate timelines for expansion and execution is a semantic distinction, not merely a common implementation strategy.
And in my opinion this thread is full of experts who have forgotten what it's like to not understand how macros work.
Your confusion is because you insist on a mental model of macros that is incompatible with their semantics. Letting that model go is necessary in order to understand macros. Otherwise, refusing to distinguish expansion time from run time, you are working towards reinventing fexprs.
http://matt.might.net/articles/metacircular-evaluation-and-f...
SICP is a great way to learn lisp and learning macros by extending the evaluator is a natural progression.
I don't care how many lisp experts are "correcting" me on the way it works in the real world, because:
a) I actually do understand how they're implemented in existing production lisps. No, really. I am not confused. In my own toy lisp compiler, I rely on first class macros and am attempting (and so far failing) to use partial evaluation to make the implementation cost somewhat sane. I am under no illusions this is normal.
b) Being an expert doesn't automatically make you good at teaching (often the correlation is reversed), and the article being commented on was a tutorial on macros.
The smug lisp weenie trope exists for a reason, and it's that the lisp community is too busy falling over themselves to prove how smart they are with arcane knowledge to give a shit about the experience beginners face. Clojure gets no end of flak from CL users who bristle at the suggestion that it even be called a lisp, but the community actually gets it and does a great job of teaching.
I have an opinion on how best to teach macros. Maybe I'm wrong. Maybe it's not a good idea to extend SICP with macros, or maybe there's a better way, and I'd love to hear arguments in those directions. But correcting it by explaining how they're implemented in CL is absolutely baffling to me.
I'm not sure why you continue to insist that responses to your comments are just about how macros happen to be implemented in Common Lisp.
But the point of a toy model is that you can read an implementation, build your own, reason about it and send code through it with little difficulty.
Once you've built eval and sent real programs through it, it's not a big jump from that to understanding how real tools work, which can generally be done incrementally.
That was certainly my lisp journey, and that includes macros (which were a separate step). After implementing them in eval, learning how they're in the real world was about as challenging as reading a restaurant menu.
SICP is a book for beginner computer science students, with a focus on functional programming as base.
As a Lisp book it's not really good, since it covers only very little actual Lisp and uses only a core Scheme language.
There is literally nothing about macros in SICP. They are mentioned only in a few places (for example it is mentioned that DELAY would be implemented by a macro).
> The smug lisp weenie trope exists for a reason, and it's that the lisp community is too busy falling over themselves to prove how smart they are with arcane knowledge to give a shit about the experience beginners face
Since you ignore the large amount of literature, teaching & research on macros in the Lisp community, what you write is rather strange. Surprise: people teaching Lisp in the 60s already 'gave a shit about the experience beginners face'.
> I have an opinion on how best to teach macros. Maybe I'm wrong. Maybe it's not a good idea to extend SICP with macros, or maybe there's a better way, and I'd love to hear arguments in those directions. But correcting it by explaining how they're implemented in CL is absolutely baffling to me.
Lisp teachers usually taught macros in context of actual Lisp languages and actual usage examples. Personally I prefer that as a first approach, over trying to explain how to extend a toy implementation with some homebrew macro implementation.
I regret my initial comment, which was written in haste. Had I known the sheer volume of people that would jump out the woodwork to demonstrate their superior technical knowledge while avoiding the argument I was actually making, I'd have been a lot more careful about what I was trying to communicate (or more likely not have bothered, which would have saved us all a lot of time).
That is where the "smug lisp weenie" trope comes from, and it's real. I don't know what it is specifically about lisp that brings it out; I can't imagine having this discussion about C#, TCL or SQL.
Were I disagreeing technically, it would be a different matter and I'd have long ago hit the books to see where I fucked up. The combined lisp experience on this thread is at least an order of magnitude more than mine, and most of you are probably smarter than me to boot.
There is already a bunch of good material out there, teaching the use of Lisp macros for both beginners and for medium-level users. Advanced-level users will find a lot of teaching&research in the Scheme literature/...
Paul Graham himself has written a good and well-known book on it. it does a good job explaining macros and their use. It's called 'On Lisp' and there is no need to replicate its content here.
Now we have gone full circle, again.
A good teacher is engaging and glib, and tells the right sorts of little lies at the right times to lead students through the steps of "Wittgenstein ladders"; but he or she can't be flat out wrong about the subject material.
I don't think it is overly useful to explain people how to extend a SICP style metacircular evaluator with some custom macro system. It's often a part of the Lisp journey, but not something I would recommend for basic Lisp teaching.
If I want people to program in Lisp, I give them a real Lisp and teach them how to program with it.
> This is about mental models
Sure, but you are teaching them the wrong mental models (macros tied to an evaluator), while I would focus on the model of general code transformations and the tools to make effective use of them.
> And in my opinion this thread is full of experts who have forgotten what it's like to not understand how macros work.
Not sure who you talk about, but I have been explaining macros to people already in the 80s.
Edit: In fact, I strongly suspect that this resulting unsuitability of Lisp for proprietary code played a major role in its commercial failure before the widespread advent of services.
Whether the compiler runs inside an 'image' or in some batch mode is also not important for macros.
Because macro expansion does something before evaluating the arguments, you can say that evaluating the arguments has been delayed.
I feel this is just looking at the same thing from two different directions.
Of course, macro expansion does _more_ than just delaying the evaluation of the arguments, and if people say that macros delay evaluating the arguments, you might think that's all they do.
Delayed (lazy) evaluation merely prevents an operand being evaluated at the time of call and until you want to read its reduced value, after which it is automatically evaluated.
It does something independent of evaluation. When the code gets compiled, the macroexpander already has transformed the code. The code might never be evaluated in this Lisp. It might be written to disk and later be loaded into another Lisp.
If something does not get evaluated, gets evaluated later, gets always evaluated or never -> that depends on the generated code.
Thus 'delaying' something is the wrong idea and it limits the imagination of what macros are used for. Think of 'general code transformation', often independent of execution in a compilation phase.
"The primary purpose of the DATA statement is to give names to constants; instead of referring to pi as 3.141592653589793 at every appearance, the variable PI can be given that value with a DATA statement and used instead of the longer form of the constant. This also simplifies modifying the program, should the value of PI change."
Or, quoting the other aphorism, what is constant for someone is a variable for someone else.
What is run time for LISP (which lacked good compiler at early stages - the famous two horse asses width constraint) can freely be a compile time for other language. Or for a LISP, for that matter, in our time.
The compile time expansion allows for better (because faster) error checking.
One class of use cases not covered in the blog post is that a variable appearing as a function's argument can only be a bound occurrence, a use of some name already in scope. A variable appearing as a macro's argument can be the binding occurrence of that name. This is why (using Racket examples) match, define-struct, for/list, and such things aren't functions.
[1]: https://docs.racket-lang.org/reference/match.html?q=match#%2...
[2]: https://docs.racket-lang.org/reference/define-struct.html?q=...
[3]: https://docs.racket-lang.org/reference/for.html?q=for%2Flist...
The closest thing to a macro in (eg. python) imo looks like this.
Instead of
d = c * (a + b)
class Adder:
def __init__(self, left, right):
self.left=left
self.right = right
class Multiplier:
def __init__(self, left, right):
self.left = left
self.right = right
d = Multiplier(Adder(a, b), c) (first b c) -> (eval b)
(if (condition? x) (do-a x) (do-b x))
(call-if (condition? x) (lambda () (do-a x)) (lambda () (do-b x)))
But macros can do more than merely wrap things in lambdas, which alone is sufficient for delayed and conditional evaluation. They can quote identifiers that are passed into them, allowing you to write new defining forms and even introduce new bindings. A nontrivial, but understandable, example is the `define-record-type` special form from SRFI 9, which can define a new disjoint record type and a constructor, predicate, and field accessors for that type all in one form. If you are willing to break hygiene and have `defmacro` or something like `syntax-case` available, you can even introduce new identifiers from out of nowhere, and write something like Common Lisp's `defstruct` which lets you define a type and automatically derive names for the constructor, slot accessors, etc. from the names of the type and slots.
Macros are a tremendously powerful, general code generation and rewriting tool, built right into the language. Underestimate them at your own peril!
Excellent point. i kinda sorta think defun in common lisp injects the new function at top level regardless, but in scheme it's scoped. So in CL you could probably get away with a simple lambda (or function) to make your struct syntax, scheme would need the macro to put all of that stuff at the top level.
Yeah, macros are crazy powerful
(defstruct point x y)
Well, sure; in a sense it is unconditional (which is a condition) that point, x and y are not evaluated (which is an evaluation choice).