Hacker News Reader

But an infix transform does delay the evaluation of expressions you've passed to it. Eg (infix (avg b) * (sgn a)).

And most practical uses of macros involve passing in expressions that will be later evaluated verbatim, e.g. (time (reduce + (range 100)))

I was only intending responding to this:

"That's a common misconception that Lisp macros are mostly used to 'delay' evaluation."

Which just seems to me confuses the matter in response to the article that was posted. At a very low level, macros operate by having the evaluation of their arguments delayed. At a higher level, they're used to implement sugared forms which usually contain valid code (though not necessarily code that makes sense in isolation).

I know you're very experienced with lisp, and I'm not trying to correct you technically, but to me this stuff only clicked when implementing a metacircular evaluator and learning when and how to perform evaluation. The blog post to me reads like the author was going through a similar process.

> But an infix transform does delay the evaluation of expressions you've passed to it. Eg (infix (avg b) * (sgn a)).

It doesn't 'delay' anything. It just rewrites at compile time

   (infix (avg b) * (sgn a))

into

   (* (avg b) (sgn a))

That's all. At runtime only the rewritten statement will be executed.

> At a very low level, macros operate by having the evaluation of their arguments delayed

Not at all. The main purpose of macros is to compute code from arbitrary expressions at compile time. There is no 'delay' -> it's mostly about rewriting expressions at compile time.

> implementing a metacircular evaluator and learning when and how to perform evaluation.

You can build a macro expansion phase into an interpreter. That's what Lisp interpreters do. But the main purpose of macros is to enable code transformations at compile time and thus only to have a cost at compile time, not at runtime. In an interpreter the macro expansions would also happen at runtime and have a cost there. But Lisp compilers are popular and to make the most out of code transformations at compile time, the idea of macro and their expansion at compile-time was introduced. Thus one gets for example new embedded languages, without the need to parse them at runtime.

I believe that if you're trying to understand what macros do from a fundamental level, treating them like functions with different semantics is a very good learning method. You implement them in a SICP style interpreter and then write a few. Totally clears up the mystery.

I'm not arguing with you about what happens in any production lisp. I'm sure you know more than me, but I do understand how they're implemented in practice, and the warts that brings - eg no (apply or <list>), and first class macros are so esoteric purely from the runtime cost that they're barely even written about in the literature.

I'm just saying the mental model of where macros fit in in an interpreter has value, and that view has not changed. In my opinion you're an expert coming in correcting a bunch of people attempting to learn the basics, and you're not wrong, but you are muddying the waters.

treating them like functions with different semantics is a very good learning method

If you mean run-time functions that act a bit funny about evaluation order, that mental model is incompatible with common introductory examples like `let`.

No, it's not. You eval the return value of the call. There is no loss of expressive power. Compile time macro expansion in this model is simply partial evaluation.

If `let` is a function, how and when are its arguments (say, `((x 2) (y 3))` and `(+ x y)`) evaluated?

The call would return:

   ((fn [x]
      ((fn [y]
         (+ x y)) 3)) 2)

Which is in turn evaluated as you'd expect.

How and when are its arguments evaluated?

The first argument is not evaluated but used as data, the second argument is evaluated after the macro function returns, as part of the macro evaluation rules, i.e. (eval (let '((x 3) (y 5)) '(+ x y)))

In conclusion, it is definitely not working like a run-time function with different evaluation order.

In conclusion, it's working exactly as I laid it out in my original post:

fn: (a b c) => (call a (eval b) (eval c))

mac: (a b c) => (eval (call a b c))

Everyone jumped on it, as production lisps don't implement macros this way. But it's the natural way to implement them in a SICP style metacircular evaluator, and a good way to learn them.

This is about mental models, not implementation details. And in my opinion this thread is full of experts who have forgotten what it's like to not understand how macros work.

The sequence of events you describe is a thing that functions do not do at run time: they do not operate on pieces of syntax. Macro expansion time is a different series of events than run time. Macros operate on pieces of program syntax, not on anything from run time.

Everyone jumped on it, as production lisps don't implement macros this way. … This is about mental models, not implementation details

It looks to me like you're the one who brought up implementation details, talking about what happens "at a very low level." You're the one who has presented a questionable implementation strategy as a mental model and has been corrected on it multiple times by multiple people. Everyone else has been describing the semantics of macros. Having separate timelines for expansion and execution is a semantic distinction, not merely a common implementation strategy.

And in my opinion this thread is full of experts who have forgotten what it's like to not understand how macros work.

Your confusion is because you insist on a mental model of macros that is incompatible with their semantics. Letting that model go is necessary in order to understand macros. Otherwise, refusing to distinguish expansion time from run time, you are working towards reinventing fexprs.

I'm not confused, and nothing I'm talking about is novel, eg:

http://matt.might.net/articles/metacircular-evaluation-and-f...

SICP is a great way to learn lisp and learning macros by extending the evaluator is a natural progression.

I don't care how many lisp experts are "correcting" me on the way it works in the real world, because:

a) I actually do understand how they're implemented in existing production lisps. No, really. I am not confused. In my own toy lisp compiler, I rely on first class macros and am attempting (and so far failing) to use partial evaluation to make the implementation cost somewhat sane. I am under no illusions this is normal.

b) Being an expert doesn't automatically make you good at teaching (often the correlation is reversed), and the article being commented on was a tutorial on macros.

The smug lisp weenie trope exists for a reason, and it's that the lisp community is too busy falling over themselves to prove how smart they are with arcane knowledge to give a shit about the experience beginners face. Clojure gets no end of flak from CL users who bristle at the suggestion that it even be called a lisp, but the community actually gets it and does a great job of teaching.

I have an opinion on how best to teach macros. Maybe I'm wrong. Maybe it's not a good idea to extend SICP with macros, or maybe there's a better way, and I'd love to hear arguments in those directions. But correcting it by explaining how they're implemented in CL is absolutely baffling to me.

"Macros" and "first-class macros" are different things. The latter are essentially fexprs.

I'm not sure why you continue to insist that responses to your comments are just about how macros happen to be implemented in Common Lisp.

You can't teach "macros": you have to choose and teach a specific macro system (at a time), whose behavior is linked to the way(s) it is implemented. Each has its own mental models; there is no single mental model for all macros everywhere.

Which is just as effective an argument against the entire of SICP. In a real lisp, a (list? <some-fn>) doesn't return true. You can't dig around inside a closure and walk the entire environment which is made up of a list of pairs.

But the point of a toy model is that you can read an implementation, build your own, reason about it and send code through it with little difficulty.

Once you've built eval and sent real programs through it, it's not a big jump from that to understanding how real tools work, which can generally be done incrementally.

That was certainly my lisp journey, and that includes macros (which were a separate step). After implementing them in eval, learning how they're in the real world was about as challenging as reading a restaurant menu.

> SICP is a great way to learn lisp

SICP is a book for beginner computer science students, with a focus on functional programming as base.

As a Lisp book it's not really good, since it covers only very little actual Lisp and uses only a core Scheme language.

There is literally nothing about macros in SICP. They are mentioned only in a few places (for example it is mentioned that DELAY would be implemented by a macro).

> The smug lisp weenie trope exists for a reason, and it's that the lisp community is too busy falling over themselves to prove how smart they are with arcane knowledge to give a shit about the experience beginners face

Since you ignore the large amount of literature, teaching & research on macros in the Lisp community, what you write is rather strange. Surprise: people teaching Lisp in the 60s already 'gave a shit about the experience beginners face'.

> I have an opinion on how best to teach macros. Maybe I'm wrong. Maybe it's not a good idea to extend SICP with macros, or maybe there's a better way, and I'd love to hear arguments in those directions. But correcting it by explaining how they're implemented in CL is absolutely baffling to me.

Lisp teachers usually taught macros in context of actual Lisp languages and actual usage examples. Personally I prefer that as a first approach, over trying to explain how to extend a toy implementation with some homebrew macro implementation.

That's fine; I prefer the latter. Maybe I'm wrong. But if I am, it's not because I don't understand how macros are actually implemented in the real world, as everyone in this massive thread seems to assume.

I regret my initial comment, which was written in haste. Had I known the sheer volume of people that would jump out the woodwork to demonstrate their superior technical knowledge while avoiding the argument I was actually making, I'd have been a lot more careful about what I was trying to communicate (or more likely not have bothered, which would have saved us all a lot of time).

That is where the "smug lisp weenie" trope comes from, and it's real. I don't know what it is specifically about lisp that brings it out; I can't imagine having this discussion about C#, TCL or SQL.

People with a bunch of Lisp experience have responded to what you wrote and disagreed. That's all.

Like I said previously, I don't believe being an expert automatically makes you a good teacher. Based on the responses in this thread, I doubt anyone here is actually good at it.

Were I disagreeing technically, it would be a different matter and I'd have long ago hit the books to see where I fucked up. The combined lisp experience on this thread is at least an order of magnitude more than mine, and most of you are probably smarter than me to boot.

and I doubt your claim, that extending a meta-circular toy interpreter with some remotely macro-like feature will help any Lisp beginner to understand Lisp macros.

There is already a bunch of good material out there, teaching the use of Lisp macros for both beginners and for medium-level users. Advanced-level users will find a lot of teaching&research in the Scheme literature/...

Paul Graham himself has written a good and well-known book on it. it does a good job explaining macros and their use. It's called 'On Lisp' and there is no need to replicate its content here.

Now we have gone full circle, again.

Yeah, there's no point continuing this discussion. Thank you for taking the time to reply in depth to my posts even though we could not find common ground.

Excellent response. Thanks!

If you tell students that Lisp macros are for delaying evaluation, you're a lousy teacher. Someone in the class will Google you out, send a few instant messages to a few others, and soon nobody will take your class seriously.

A good teacher is engaging and glib, and tells the right sorts of little lies at the right times to lead students through the steps of "Wittgenstein ladders"; but he or she can't be flat out wrong about the subject material.

Yeah, I'm not going to take teaching advice from a person that walked me condescendingly step by step through an example to point out an inconsistency that wasn't there because they misunderstood what I was saying. ¯\_(ツ)_/¯

> Everyone jumped on it, as production lisps don't implement macros this way. But it's the natural way to implement them in a SICP style metacircular evaluator, and a good way to learn them.

I don't think it is overly useful to explain people how to extend a SICP style metacircular evaluator with some custom macro system. It's often a part of the Lisp journey, but not something I would recommend for basic Lisp teaching.

If I want people to program in Lisp, I give them a real Lisp and teach them how to program with it.

> This is about mental models

Sure, but you are teaching them the wrong mental models (macros tied to an evaluator), while I would focus on the model of general code transformations and the tools to make effective use of them.

> And in my opinion this thread is full of experts who have forgotten what it's like to not understand how macros work.

Not sure who you talk about, but I have been explaining macros to people already in the 80s.

While experienced lispers already know this, it may not be clear to everyone that in Common Lisp "compile time" routinely happens repeatedly in a running image. cl-ppcre[1] relies heavily on this. It can take a regexp pattern as a runtime input and compile a recognizer down to machine code on the fly. This technique gets used routinely.

Edit: In fact, I strongly suspect that this resulting unsuitability of Lisp for proprietary code played a major role in its commercial failure before the widespread advent of services.

[1] https://edicl.github.io/cl-ppcre/

That's not a Common Lisp specific thing. The first Lisp compiler in 1962 was already written and running inside Lisp.

Whether the compiler runs inside an 'image' or in some batch mode is also not important for macros.

A normal function call evaluates the arguments and then calls the function.

Because macro expansion does something before evaluating the arguments, you can say that evaluating the arguments has been delayed.

I feel this is just looking at the same thing from two different directions.

Of course, macro expansion does _more_ than just delaying the evaluation of the arguments, and if people say that macros delay evaluating the arguments, you might think that's all they do.

A macro or FEXPR may never evaluate an operand at all. They simply receive the unevaluated forms and decide what to do with them, with explicit evaluation being just one option which can be performed on the operand.

Delayed (lazy) evaluation merely prevents an operand being evaluated at the time of call and until you want to read its reduced value, after which it is automatically evaluated.

> Because macro expansion does something before evaluating the arguments

It does something independent of evaluation. When the code gets compiled, the macroexpander already has transformed the code. The code might never be evaluated in this Lisp. It might be written to disk and later be loaded into another Lisp.

If something does not get evaluated, gets evaluated later, gets always evaluated or never -> that depends on the generated code.

Thus 'delaying' something is the wrong idea and it limits the imagination of what macros are used for. Think of 'general code transformation', often independent of execution in a compilation phase.