Hacker News Reader

Comment by progman | Hacker News Reader

I know both Lisp and Scheme pretty well. Scheme's macros are hygienic, Lisp's macros however are much easier to handle. The hygiene problem can be solved by using (gensym) for local identifiers. Example:

http://stackoverflow.com/questions/267862/what-makes-lisp-ma...

View on HN

To completely solve the hygiene problem on an academic level, you have to use gensyms for all references generated by the macro, including all function symbols.

Lisp macros don't bother doing this because the Lisp-2 separation provides adequate hygiene. Programs which choose to lexically shadow standard functions like cons or list in the function namespace are considered to be sticking a fork in the proverbial toaster, and so that is not handled.

That would't work as well in Scheme because the mere presence of a variable called list in the user's code would interefere with a (list ...) function call generated by a macro.

Also providing hygiene are packages. If macro-generated code references private symbols in the macro's own package, everything is cool. Packages take care of the problem for nonstandard functions. You just don't use another package's symbols for local functions and variables, and that's that.

How are Lisp's macros easier to handle than syntax-rules? On the one hand, you have a function that receives an s-expression, manually deconstructs and transforms the s-expression while (hopefully) carefully, manually aliasing any symbols to avoid inadvertent capture, and constructs a replacement s-expression. On the other hand, you have a pattern-matching facility that automatically takes care of deconstruction and hygiene for you. Yes, syntax-rules is strictly less powerful than defmacro, but in practice it covers the vast majority of use cases very well.

I'd also like to mention that avoiding variable capture is only one part of macro hygiene. The other part is referential transparency in the face of shadowing. Common Lisp partly provides this because names in the standard package can't be redefined, but it's still wide open when it comes to other names. Consider this (admittedly contrived, for simplicity) example:

    (defun square (x)
      (* x x))

    (defmacro hypotenuse (x y)
      `(sqrt (+ (square ,x) (square ,y))))

    ...

    (let* ((square   (make-square s))
           (diagonal (hypotenuse s s)))  ; ERROR!  tried to apply 
      ...)                               ; #'* to a square!

In the expansion of `hypotenuse`, `square` refers to the local variable bound on the line above it, while the macro author clearly intended it to refer to the "global" variable up top. This is a situation that's fairly easy to run into, and horribly confusing when it occurs (especially if the user of the macro didn't write it). Hygiene ensures that the reference to `square` in `hypotenuse` will always refer to the binding of `square` that was in scope at the time the macro was defined, thus avoiding the problem. On its own, `gensym` is necessary but not sufficient for hygiene.

Really, an ERROR? Could you show the actual message that was printed? You can't, because you did not even bother to try. Here is an actual test:

    (defun square (x)
      (* x x))

    (defmacro hypotenuse (x y)
      `(sqrt (+ (square ,x) (square ,y))))

    ;; whatever
    (defun make-square (s) (list :square s))

    (defun foo (s)
      (let* ((square (make-square s))
             (diagonal (hypotenuse s s)))
        diagonal))

Now, let's compile (SBCL):

    ; caught STYLE-WARNING:
    ;   The variable SQUARE is defined but never used.

Not really what you expected, right?

    (foo 10)
    => 14.142136

It looks like it works.

You are trying to apply Scheme's semantics to CL. You forgot to take into account CL namespaces, which consist of (1) function/classes/type/variable namespaces and (2) namespaces defined by packages (or custom mappings).

> namespace n. 1. bindings whose denotations are restricted to a particular kind. ``The bindings of names to tags is the tag namespace.'' 2. any mapping whose domain is a set of names. ``A package defines a namespace.''

(http://clhs.lisp.se/Body/26_glo_n.htm)

If you defined a local function named square, that could have been the beginning of a point. However, I don't buy the idea that inside a single package, someone is going to shadow a function that belongs to the same package, in a macro (if this happens, it is more likely to be done on purpose). As for shadowing function bindings from other packages, this is even more improbable.

Please check your facts and avoid posting made-up error messages that can easily be mistaken for actual ones.

My example wasn't meant to be in Common Lisp, just a nebulous "Lisp" dialect. See my response to the sibling here: https://news.ycombinator.com/item?id=13223290

I'm not trying to be critical of CL. I thoroughly enjoy working in CL. I'm just trying to clear up some confusion about what it means for a macro to be hygienic, that's all.

> My example wasn't meant to be in Common Lisp, just a nebulous "Lisp" dialect.

And how can I believe you? this is just too late.

> I'm just trying to clear up some confusion [...]

You were spreading confusion by using a dialect of Lisp known only by you, that looked exactly like CL, without giving proper warnings.

> I'm not trying to be critical of CL. I thoroughly enjoy working in CL.

Fine, but this would have been the same if you used a nebulous "Scheme" without saying it so.

> And how can I believe you? this is just too late.

Huh?

> You were spreading confusion by using a dialect of Lisp known only by you, that looked exactly like CL, without giving proper warnings.

It's a hypothetical example meant to illustrate an aspect of macro hygiene. Again, it's not a criticism of CL. Also, it's not "a dialect of Lisp known only by me". It happens to be EuLisp, which is what I've been working with lately, and I chose it because it was designed as a mix between CL and Scheme (and Le-Lisp, RIP) and I figured that, as such, it would be accessible to both Lispers and Schemers. Apparently, I was wrong.

> Fine, but this would have been the same if you used a nebulous "Scheme" without saying it so.

The specific language was irrelevant to my point. Only the semantics of defmacro and gensym were relevant.

EDIT - fixed a quote and some formatting

On that last point, you give the semantics of defmacro as defining "a function that receives an s-expression, manually deconstructs and transforms the s-expression".

You then exemplify that with (defmacro hypotenuse (x y) ...) which features automatic destructuring that pulls out x and y. The macro needs to do no further manual destructuring on these arguments.

You're right. I was thinking about Scheme when I wrote part of that comment, EuLisp in another part, Common Lisp in another, Le-Lisp in yet another, and my writing wound up being pretty muddled. I apologize for the confusion. I need to work on improving the clarity of my communication. Part of the problem is that I view comments on the Web very casually, a conversation, but that's not really the right way to approach it due to the differences in format. Anyway, thanks.

P.S., I was playing with TXR a bit last week after you mentioned it in another thread on here somewhere. After I got to the web site, I recognized it and recalled playing with it a tiny bit quite some time ago (at least a year ago, possibly much longer). Anyway, it's really neat. If I get some spare time sometime soon, I'm going to try and pick it up a bit more because it seems like the kind of thing that will come in very handy at the command line.

The ideas in TXR seem really novel to me. Did you come up with that form of pattern language, or did you nab it from somewhere else? The closest thing I can think of is SNOBOL, but TXR is much nicer to use than SNOBOL.

The idea for the TXR extraction language came from simply wanting to do the inverse of what "here document" interpolation does: have some text with variables, and have that match a similar text, binding pieces to the variables. I had a passing familiarity with pattern matching (as in logic programming) without much actual experience using or coding that sort of thing. I wasn't aware of how SNOBOL works, and also wasn't aware of PEGs. I just started implmenting and adding ideas as they occurred to me.

The pattern language is basically equivalent to PEGs, which can be seen from the examples of arithmetic expression and JSON parsing.

When I added functions, and hence recursion, I knew that the matching would increase in theoretical power from just regular languages to context-free. But only when I used it to implement some parsing it hit me that the code is basically just grammar rules that execute. After that when I encountered PEGs in some paper, the material was obvious.

Someone told me some years ago that it reminded them of SNOBOL in some ways. That made me curious, so that I researched SNOBOL a bit.

The directives which capture or assert a chracter or line position might have been influenced by SNOBOL. :)

http://www.nongnu.org/txr/txr-manpage.html#N-02D5D09D

> > And how can I believe you? this is just too late.

> Huh?

See it from my point of view: for all I know, you could as well have made an error, then pretended you were not really writing in CL but a nebulous Lisp; hours laters, after having looked around in the archives, you come back and say "It was actually EuLisp". I am not really saying you actually did this, but I am not ready to believe all you say blindly either. Clarifying things up front would have been way more credible.

> Also, it's not "a dialect of Lisp known only by me". It happens to be EuLisp [...]. I figured that, as such, it would be accessible to both Lispers and Schemers. Apparently, I was wrong.

EuLisp is accessible to both Lispers and Schemers, but you can't blame readers for not guessing what you have in mind when you aren't precise enough in your writings.

> The specific language was irrelevant to my point. Only the semantics of defmacro and gensym were relevant.

Except your example depends on other things than defmacro and gensym, and those things depend on the actual language you use, which makes the language not irrelevant. Simply saying "Take this EuLisp code...", or "assuming a Lisp-1 dialect..." would have prevented the confusion that would necessarily arise. I wonder why you did not just say that.

I apologize for causing so much confusion. My comment wasn't clear at all and I failed to communicate precisely what I intended. Thank you for your critique. I will take it to heart and try to improve my written communication in the future.

Thanks for your comment.

Your example looks like Common Lisp. The square symbol is bound in the variable namespace by let*; the reference in the macro expansion is for a square in the function namespace.

The dual namespace saves most Common Lisp macro-using code from this problem.

In large programs, you'd want to use packages. Those also prevent the problem:

  (defmacro mypackage:macro (arg)
    `(mypackage:function ,arg))

  (flet ((yourpackage:function (arg)))
    (mypackage:macro (whatever))

yourpackage:function and mypackage:function are different symbols, so no interference.

Also, ANSI CL makes it undefined behavior to rebind a standard function; i.e. (labels ((list () ...)) ...) can behave arbitrarily, and implementations are permitted to signal an error like "cannot rebind cl:list".

My example is intended to be nebulously "Lisp". I did mean to mention in my comment that Common Lisp's packages help here (I'm usually frustrated in these discussions when people are always like "Lisp2 Lisp2 blah blah" but that's only part of the story, imo packages play a bigger role), so thanks for pointing that out. I did, however, mention that CL doesn't let you rebind standard names, though.

My point wasn't "Common Lisp (or defmacro) bad, Scheme good", or that you can't write hygienic macros in Common Lisp[1], though it appears that I've come off that way. My point is just that `gensym` alone isn't enough for hygiene (in all cases).

[1]: Another thing that doesn't seem to be understood well enough in these discussions is that hygiene is a property of macros, not macro systems.

You can gensym functions, you know. It's just a PITA.

Take it from a CHICKEN user...

True, you can, though I've never seen anyone actually do it. Not that I've read all the code out there or anything, far from it.

In chicken, you have to (r) all your functions (the rename function is passed to the function transformer, but it's called r, or rename, by convention) if you're writing an er macro, which is what you do if you want expansion to be fast.

I know, I've been a chicken user for some time :)

I don't know why, but it slipped my mind while I was writing the above. I think I was just too focused on my goal and wound up screwing it all up anyway. Oh, well, c'est la vie.

Formally speaking, CL lets anything happen when you rebind standard function names; that's what undefined behavior means. It's a good idea to diagnose that, and not difficult to do.

> that doesn't seem to be understood well enough in these discussions is that hygiene is a property of macros, not macro systems

Can you cite a comment which doesn't seem to understand this?

And it is wrong; of course macros can be hygienic or not; but macro systems which crank out automatically hygienic macros are also themselves called hygienic. Everyone understands what that means.

(Or else countless of Lipers, including notable experts, have been wrong for years in how they discuss macros and macro systems.)

> Can you cite a comment which doesn't seem to understand this?

I don't have any on hand, but I've encountered this misunderstanding quite a few times over the years in discussions of macro hygiene. To be clear, I didn't mean to imply that either you or GP don't understand this. It's just something I've noticed before and it seemed like a good time to bring it up.

> And it is wrong; of course macros can be hygienic or not; but macro systems which crank out automatically hygienic macros are also themselves called hygienic. Everyone understands what that means.

I'm not saying that macro systems can't be hygienic. I'm saying that hygiene is a property of macros. It's not wrong to say that a system like syntax-rules is hygienic, considering that it only allows for the definition of hygienic macros. IMO, the line gets a little blurrier for systems like syntax-case, synclos, or explicit renaming. They're also commonly called hygienic because they provide special facilities for defining hygienic macros, but synclos and ER macros aren't really that different from defmacro+gensym when it comes down to semantics.

I suspect you're just allowing yourself to be overwhelmed by the ambiguity in "hygienic macro system", which can be parsed as ((hygienic macro) system) or (hygienic (macro system)).

The former parse attributes the macros as being hygienic, and the system as having to do with that kind of macro (an not itself being called hygienic).

You're welcome.

Thing is, every macro system is a ((hygienic macro) system), because you can write hygienic macros with any system. The latter parse makes more sense to me, as it relates that it's a system designed to enable hygienic macros beyond what a direct transformer would provide.

Reply on news.ycombinator.com