Ask HN: Are macros still an enormous edge in the power of a language? | Hacker News Reader

Ask HN: Are macros still an enormous edge in the power of a language?

35 pointsrevenant52 comments

I use Lisp (Clojure) a lot and like it, but it seems like macros, except in specialized cases, aren't quite the game changer they used to be. They're much tougher to reason about than functions, can't be passed around as parameters, and can lead to "magic" DSLs that only their owners understand. Obviously they are sometimes very useful, and the built-in macros (e.g. cond, if-let) are essential, but I don't see them as such a game-changer in modern programming. I feel like over 90% of the edge Lisp has over Blub comes from its functional support, with less than 10% coming from macros. This makes the edge small enough that, in many circumstances, I can see languages like ML and Haskell winning; good static typing (e.g. ML, not Java) becomes a benefit on large projects.

I could be convinced that I am wrong and just don't "get" macros yet, but it seems like much of what's shown off in On Lisp et al is available nowadays in non-macro languages. I'd like to see modern uses for macros that aren't easy to implement in macroless languages.

View on HN

I used to like arguing over the Internet about this subject. There are many good technical and management/organizational arguments you can make for and against macros. What I've come to realize is they're all pretty much irrelevant.

The entire point of programming is automation. The question that immediately comes to mind after you learn this fact is - why not program a computer to program itself? Macros are a simple mechanism for generating code, in other words, automating programming. Unless your system includes a better mechanism for automating programming (so far, I have not seen any such mechanisms), _not_ having macros means that you basically don't understand _why_ you are writing code.

This is why it is not surprising that most software sucks - a lot of programmers only have a very shallow understanding of why they are programming. Even many hackers just hack because it's fun. So is masturbation.

This is also the reason why functional programming languages ignore macros. The people behind them are not interested in programming automation. Wadler created ML to help automate proofs. The Haskell gang is primarily interested in advancing applied type theory.

Which brings me to my last point: as you probably know, the reputation of the functional programming people as intelligent is not baseless. You don't need macros if you know what you are doing (your domain), and your system is already targeted at your domain. Adding macros to ML will have no impact on its usefulness for building theorem provers. You can't make APL or Matlab better languages for working with arrays by adding macros. But as soon as you need to express new domain concepts in a language that does not natively support them, macros become essential to maintaining good, concise code. This IMO is the largest missing piece in most projects based around domain-driven design.

Excellent analysis!

Yes. It's part of good Lisp style not to use a macro when a function would suffice, and yet the source of HN is full of macro calls. If you want to see examples of modern uses of macros, that would be a good place to start.

http://ycombinator.com/arc/arc3.tar

If you want a single, self-contained example, here's one:

    (mac zap (op place . args)
      (with (gop    (uniq)
             gargs  (map [uniq] args)
             mix    (afn seqs
                      (if (some no seqs)
                          nil
                          (+ (map car seqs)
                             (apply self (map cdr seqs))))))
        (let (binds val setter) (setforms place)
          `(atwiths ,(+ binds (list gop op) (mix gargs args))
             (,setter (,gop ,val ,@gargs))))))

How do you implement that in a language without macros?

Care to explain it for the Lisp-impaired?

It lets you apply an arbitrary function to a variable or a position anywhere within some structure:

    arc> (= flag t)
    t
    arc> (zap no flag)
    nil
    arc> flag
    nil
    arc> (= foo '(1 2 3 4 5))
    (1 2 3 4 5)
    arc> (zap (fn (x) (* x 10)) 
              (foo 0))
    10
    arc> foo
    (10 2 3 4 5)

BTW, in more idiomatic Arc you'd say

    (zap [* _ 10] foo.0)

Here's a line in HN that uses it:

    (zap [firstn votewindow* _] (uvar user votes))

This expands into roughly the equivalent of:

    (= ((profs* user) 'votes)
       (firstn votewindow* ((profs* user) 'votes)))

Not quite, though, because you only want to evaluate the code that gets you to the location once, and the operation needs to be atomic. The actual macroexpansion is much more complicated.

If that's all it does, then I don't see a problem with implementing it in any language without macros. Let's take simple C. (I didn't test it, or even try to compile - just want to show the idea)

    void zap(void *object, void* (*selector)(void *), void (*action)(void*)) {
        action(selector(object));
    }

Given your explanation: "It lets you apply an arbitrary function[1] to (a variable or a position anywhere within some structure)[2]."

Where object is the structure you're operating on, `selector` is a function taking an object and returning a pointer to the position [2]. The `action` is a pointer to an arbitrary function [1].

Your first example would look like this:

    int foo[5] = {1, 2, 3, 4, 5};
    void *select_first(void *x) {return x;}
    void act_times_10(void *x) {int *i = (int *)x; (*i)*=10;}
    
    zap(foo, select_first, act_times_10);

Or am I missing something you didn't include in the description?

To be consistent across the lisp and c code, I just want to point out that the definition of the c code should be:

  void zap(*action, *selector, *object) { ... }

compared to

  (mac zap (op place . args) ... )

But I also think what the c code really represents is an abstraction. You still need to write out the definitions of the functions each time you call zap. With the lisp code you write one line, which is the whole point of macros: write a single line of code which expands to several lines.

I think you're missing the fact that in the C version you have to write a specialized selector function each time (to find the thing you want to zap), whereas the macro works generically on any place. ("Place" here is an abstraction meaning "settable thing" that is closely related to Lisp macros and is analogous to the left-hand-side of assignment statements in most languages, but far more manipulable by the programmer.)

PG asked "How do you implement that in a language without macros?". I responded with a simplest translation of the example, but a more general solution exists too. Nothing stops me from using GObject instead of void*, which can provide named "places" and as much generic functionality as needed. `selector` and `action` can even operate on property tables, which would be a closer translation from lisp.

I was simply surprised that PG asked about something like that - I still think it's pretty simple and the code in C is as close to the macro as possible (with whatever object abstraction you need).

However I have to admit that I thought about other languages now and it seems that the described zap() is not possible in python... and that's interesting / worrying:

    action(object[place])

cannot assign a new value to `object[place]`, while

    object[place]=action(object[place])

has to do the lookup twice.

By using C you are changing the problem, as your C pointer passes by reference instead of value. The lisp macro works equally well when passing by value, so try rewriting your function to operate on a struct (and, not a struct pointer). It is likely that PG was referring to this functionality when asking how to implement this as a function.

I am suspicious that this macro would not be necessary in a language with better libraries. For example, this code:

  (zap [firstn votewindow* _] (uvar user votes))

In Python you wouldn't need a macro for this. You could just write

  del uvar(user, votes)[votewindow:]

  uvar(user, votes)[votewindow:] = []

I think this highlights a weakness of Arc -- while terseness is probably wonderful once one has Arc in one's brain, it makes things considerably less legible to everyone else. Contrast with presenting CL/Scheme to proponents of the other, where I would assume they could mostly muddle through, Googling one or two odd things. Have you ever tried Googling "arc no function"? (Lisp analogy may not be accurate; substitute Python/Ruby or C#/Java if it makes you feel better.)

C++ version:

  template<typename T>
  void zap (T (*f)(T x), T &x) { x = f(x); }

Example:

  int square (int x) { return x*x; }

  int foo[5] = {1,2,3,4,5};

  zap<int>(square, foo[3]);   // foo is now {1,2,3,16,5}

Admittely, the <int> is a little ugly, but that can probably be fixed with Hindley-Milner (or just plain dynamic typing).

C++ templates are similar to macros: they run at compile time to generate code, but they're a lot harder to use.

True, but in this case, the template serves a different purpose from the macro. It's there to make zap polymorphic, and would be unnecessary in a dynamic or type-inferred language. The main point is that pg's macro can be implemented as an ordinary function in a language that has references.

"Scrap Your Boilerplate" in Haskell: http://www.cs.vu.nl/boilerplate/

You can do the same thing without macros. Granted, the compiler supports by generating code, so it is macro-esque.

The issue of macro is that it is, by itself, not a power. It's like how knife by itself is not food. However, knife can make you food. That's the main point.

Imagine this Christmas, Perl 6 is out with some killing features. Now, your language (Python? Haskell? ML? Whatever) does not have these features. What would you do?

You have 2 choices, basically: a. forget about that features, write a long blog in which you prove how your language can do just the same thing, albeit "a little" less elegant b. Switch to perl. This is not always possible, btw, since you may sacrifice some OTHER features that your language has.

How about a LISPer? Well, simple, let's implement these features. Keep in mind: All languages are Turing equivalent, meaning they can do roughly the same thing. There are two differences: 1. some do some task more elegantly (aka easier to maintain) than others. 2. Some have better support (eg. corporates, community, etc.) than others

Thus, you CAN always write the exact same features in LISP. Now, the definition of the macros WILL be messy. However, as a good practice, you spend 2 days debug that holy mess, and jam it into a file you banish somewhere with a dot precede the directory. Next time, you can use the features, pretending that you are using Perl 6!

This is my strategy. My Scheme code always has for-loop (purely functional, too), foreach-like, OO-like features. Of course, Scheme (remember, it's not even CL) does not provide these, but I can either implement them or find the implement somewhere. I have once even try to implement Icon-like mechanism with Macro + Continuation (okay, the Scheme implementation that time was a bit fuzzy, so I failed. But, with pure Scheme, it SHOULD be possible). That's the main power: you can copy others' ideas extremely fast! Without throwing away your old code, too. Hurray to Macro!

Are macros still at the "top" of the features pyramid?

In http://www.paulgraham.com/avg.html , PG argues that Lisp is more powerful than other languages because it has macros. However, in the 50 years since the invention/discovery of Lisp, is there now some language that has a new feature X that lisps don't have (and can't easily add)? Or are macros still the "pinnacle" of language features?

Lisp is more powerful than other languages because it has macros

It's a bit more complicated than that. The key idea is that programs are composed of lists. But that idea is tied to several others:

http://www.paulgraham.com/diff.html

Fair enough, although I was asking a different question (not very clearly, I suppose). Is there a language X and a feature (or set of features) Y for which the following statement holds?

"Language X is more powerful than Lisp because it has feature(s) Y."

Have you ever use the combination of LISP + Perl? Okay, maybe not. The idea is to use Perl's ideology (aka copy everything) with LISP technology (macro, since perl does not provide that).

A has X? Well, implement that. It takes 2 days, but it will save you months later. B has Y? implement that. Now you have BOTH X and Y. A users will wish for Y, B users will wish for X. You? Both. Sorry, I meant "Lisper? Both".

In 2006 over 2 months, I wrote an OCR (optical char recognition - turning images back into text) system in Lisp using macros.

First it finds all 'features' by traversing a 1-pixel wide representation of the text character's image. Then you describe which features you want to look for, in a macro language.

So (in layman's terms) , a 'c' is a top curve curving up, a lower curve curving down, and a stem at the left-hand side close to the left-hand of those curves. So 'c' would be about 3 rules, in this macro language.

It was fun but monstrously sluggish and I never got the 1-pixel wide algorithm (called skeletonisation) that I used, to work 100%.

One thing it did have going for it was you could identify really small font chars and even 10-pixel-high truetype fonts, as long as you normalise so each char is 60 pixels high.

I ported the system to C++ also in 2006 and had to write a simple Scheme-ish interpreter to turn the Lisp ruleset into C code.

Macros expand into other code, so in the literal, obvious sense, there isn't anything you can do with them that you can't do without them. The question is whether you can do it as briefly, as clearly, or as composably.

Here's another example. I work on a spreadsheet app. A spreadsheet has many operations that apply to either a row or a column (e.g. selecting a row/col or resizing it). The fact that they occur in transposed dimensions make them difficult to abstract over with functions. You probably end up with a bunch of boilerplate that says things like "if this is a column, change the first co-ordinate, otherwise change the second", or most likely would just write the code twice, once for the horizontal case and once for the vertical. In Lisp, we can write such a function for one dimension and then use macros to generate a transposed version of the same function that does exactly the same thing in the other dimension.

That's the first use of a macro I can wrap my head around, but it can also be done elegantly without macros - each function that acts on a row or column of data will have a "list of values" argument that could either be a list of column values from a row or a list or row values from a column.

Not quite. I wasn't clear enough. These operations are purely UI manipulations. They don't operate on lists of values or even need to know what they are. What they care about are things like: should I set a width property or a height property? To which you might reply: fine, just pass the dimension around and have a set-size function that knows whether to set width or height depending on the dimension, and so on. I say: yeah, you can do it that way, but it takes more code and is harder to read, because now you have to write it in a dimension-agnostic way, which is not how humans (or I at least) like to think about these things. It's easier to write or read the concrete version for one dimension and then say "the other dimension is just like this only transposed", which is exactly what the macro does for us.

Do you need macros to make such a program work? No. Can you use them to make the program shorter, clearer, and more generalizable? Yes.

I couldn't live without iterate, a very intuitive, lispy, simple, mostly "backwards compatible" replacement for Common Lisp's clunky, unlispy, ugly loop iteration facility.

http://common-lisp.net/project/iterate/doc/index.html

http://www.lispworks.com/documentation/HyperSpec/Body/06_a.h...

In a language without macros, I'd be stuck with loop and curse every time I need to write some non-trivial iteration code. But thanks to macros, I can use this wonderful iteration facility as a library, and have great integration with CL, just like if it was part of the language.

There's also this macro I wrote that I couldn't live without, I call it fmask.

Problem: I always hated that (loop for element in list collect (my-fun element)) translates trivially to (mapcar #'my-fun list) but (loop for element in list collect (my-fun element my-constant)) doesn't. You'd either have to stick with the loop version or use something ugly and inefficient like (mapcar #'my-fun list (make-list (length list) :initial-element my-constant)) or write out the lambda: (mapcar (lambda (element) (my-fun element my-constant)) list) (my-constant could actually be an arbitrary form).

None of these solutions appealed to me. So I wrote fmask. With it, the example is simply rewritten as (mapcar (fmask #'my-fun ? (? my-constant)) list)

Another example:

  (mapcar (fmask #'list ? (? (1+ (length ?))))
  		 '(3 8 2) '(("some" "items") "A string" (nil "test")))

=>((3 3) (8 9) (2 3))

Here's the implementation:

  (defmacro fmask (function unknown (&rest args))
    (check-type unknown symbol)
    (macrolet ((autotraverse-let (bindings &body body)
	         (let ((gensyms (iter (repeat (length bindings))
  				    (collect (gensym "TRAVERSED")))))
  		 `(let (,@(iter (for (nil value) in bindings)
  				(for gensym in gensyms)
  				(collect `(,gensym ,value))))
  		    (symbol-macrolet (,@(iter (for (var nil) in bindings)
  					      (for gensym in gensyms)
  					      (collect `(,var (prog1 (car ,gensym)
  								(setf ,gensym (cdr ,gensym)))))))
  		      ,@body)))))
      (labels ((unknownp (arg)
  	       (etypecase arg
  		 (atom (eq arg unknown))
  		 (cons (some #'unknownp arg))))
  	     (literalp (arg)
  	       (or (typep arg '(or array keyword pathname number))
  		   (and (consp arg)
  			(or (and (eq (first arg) 'function)
  				 (cdr arg)
  				 (not (cddr arg)))
  			    (eq (first arg) 'quote))))))
        (let ((unknowns (iter (repeat (count-if #'unknownp args))
  			    (collect (gensym "UNKNOWN"))))
  	    (knowns (iter (for arg in args)
  			  (if (not (or (unknownp arg) (literalp arg)))
  			      (collect (list (gensym "KNOWN") arg)))))
  	    (known-function (and (consp function)
  				 (eq (first function) 'function)
  				 (symbolp (second function))
  				 (second function))))
  	(autotraverse-let ((an-unknown unknowns)
  			   (a-known (mapcar #'first knowns)))
  			  (let* ((unknown-function (gensym "UNKNOWN-FUNCTION"))
  				 (call-function (if known-function
  						    (list known-function)
  						    `(funcall ,unknown-function)))
  				 (main (let ((inner `(lambda (,@unknowns)
  						       (,@call-function
  							,@(iter (for arg in args)
  							     (collect (if (unknownp arg)
  									  (subst an-unknown
  										 unknown
  										 arg)
  									  (if (literalp arg)
  									      arg
  									      a-known))))))))
  					 (if knowns
  					     `(let (,@knowns)
  						,inner)
  					     inner))))
  			    (if known-function
  				main
  				`(let ((,unknown-function ,function))
  				   ,main))))))))

Please use Lisppaste to show gorgeous code :-)

http://paste.lisp.org/new

Good idea!

fmask's implementation with syntax coloring:

http://paste.lisp.org/display/81488

I also have a more extended example usage in an annotation at the bottom.

For the simpler cases you can just use curry or rcurry, eg:

(mapcar (rcurry #'parse-integer :junk-allowed t) '(" 1" "2xx" "3" "foo")) (1 2 3 NIL)

... whereas your more complex example doesn't strike me as an improvement over lambda + backquote:

(mapcar (fmask #'list ? (? (1+ (length ?)))))) vs.

(mapcar (lambda (n list) `(,n (1+ (length ,list)))) numbers lists)

... which might be longer than the fmask version but has the advantage of being immediately readable to any Lisp programmer, and of allowing arguments to be reused, or used in a different order than they appear.

Doug Hoyte has a nice read macro called #`:

(mapcar #2`(,a1 (1+ (length ,a2))) numbers lists)

You might be interested in what John Wiegley had to say on the subject (in particular, Haskell has no macros; is this bad?) a couple of months ago:

http://www.newartisans.com/2009/03/hello-haskell-goodbye-lis...

The comments below the article also have some interesting and relevant things to say.

For somebody with 15 years of Lisp experience he sure managed to mention the most trivial use of macros.

Public language changes are usually just trolling for the love of a community; it's like those public "converters" who go from one mainstream, often Abrahamic, religion to another.

The rest of us just rewrite the critical stuff in whatever that's the most efficient, and the non-critical stuff in whatever that's easiest.

Oh screw this! Guys, I just converted from Common Lisp to CSS! The cascade is like OOP inheritance, without the annoying CLOS multiple-inheritance and over-configurable multiple dispatch. It has keyword methods, :hover, :active, etc, I can even implement list processing with getElementById! Take that, you Lisp weenies!

Well, look at the comments. Things get a little deeper there.

Haskell can do almost everything people generally use macros for without them. But there's always some things only macros can do, so for that you have template haskell.

As I understand it, if Haskell didn't already have syntax for

  let vars
  in expr

you wouldn't be able to use Template Haskell to create it from scratch. Any domain-specific languages you make have to mimic existing Haskell syntax.

Most people think macros are for "compacting" code, or factoring out common parts.

Macros are for introducing new control primitives, syntax and evaluation models into the language.

It's both, no? You can always write macroexpanded code by hand, only with much repetition, so in that sense macros are always compacting.

Except, I suppose, for perverse macros which expand into shorter code :)

It's for both, but utility functions also serve the same purpose (which is why you see people claiming they could do whatever Lisp macros do with function/methods/classes/monads/templates/makefiles, etc.) :-P

Personally, I think compiler macros are far more interesting than 'regular' macros. I would love to see compiler macros even in mundane languages such as python.

Example, in pseudo code:

<code> defun regex_find(some_string, regex) return regex.compile().find(some_string)

def compilermacro regex_find(some_string, regex) if IS_STRING_LITERAL(some_string) and IS_STRING_LITERAL(regex) SUBSTITUTE exec(regex.compile().find(some_string)) elif IS_STRING_LITERAL(regex) SUBSTITUTE exec(regex.compile()).find(some_string) else DO_NOTHING return

</code>

regex_find(MYSTRING, '\w(.+)\b')

A language without compiler macros would compile the regex at runtime, even in those cases that it is a string literal, and could be compiled at compile time. And in the case that both the string and the regex are string literals, the result itself could be computed at compile time. Another alternative, used by lots of languages/libraries, is to maintain a cache of recently compiled regular expressions. Of course, this doesn't help when you have a loop that uses 15 regular expressions but it only caches the last 14, for example.

This is one reason why CL-PPCRE is faster than the C PCRE in benchmarks, it makes great use of compiler macros.

This is a very concrete real world thing, that isn't endlessly debatable, unlike regular macros, and it could be included in languages without any special macro syntax.

One example of a time I really wanted compiler macros in Python was when I was doing some parsing of binary files, using the struct module. The struct module lets you do stuff like struct.decode(data, 'UUUIUH'), where the UUUIUH is some code for Unsigned long, unsigned long, 32bit int, whatever. However, it parses the little code each time you want to use it, which ends up taking way longer than the actual decoding itself. In python 2.x, whatever I was using, they didn't have any way to make any sort of compiled decoder, you had to just use it as above, which is silly, since the code is almost always going to be a string literal and 99% of the work could be done once when the code is compiled.

> Another alternative, used by lots of languages/libraries, is to maintain a cache of recently compiled regular expressions. Of course, this doesn't help when you have a loop that uses 15 regular expressions but it only caches the last 14, for example.

Nitpick: it depends on the replacement policy whether it helps. If the cache is MRU, for example, it would work fine.

It doesn't matter if each one is used once, say in a loop.

If you have 14 slots in your cache, and you loop over 15 regular expressions, no cache policy (unless it also tracks regexes it has already lost from the cache, which effectively gives it more than 14 slots) will work, except maybe some kind of cache that just remembers the first 14 regexes it sees, and only forgets one randomly every 10 more regexes it sees. Such a cache policy would be really really stupid in most cases not specifically designed to thwomp, say, an LRU cache or a regular deque or something.

I used macros a lot in C to initialize structs and arrays with constant information, like keywords for a parser. It would still be useful for languages, like Java, which do not have literals for initializing objects, maps, and arrays. Another use would be DSL's on top of a language like Python. Sure you can do all of this within the languages, but macros would make it more expressive and concise. I really don't buy the argument that we need to be protected from idiots. They will piss in the soup regardless.

C "macros" != Lisp macros.

Why with the "sarcasm"?

Because C macros have no sense of context; they're replaced in the raw character stream while lexer is reading in a file. C macros are mostly stupid string-replacements of the s/foo/bar/g type.

Lisp macros operate on the abstract syntax of the code forms, after it's parsed. They're tree-transforms. They could also be nested, so one macro would operate on the result of another. All of them could also operate on the values read by reader macros, which are Lisp's C-like preprocessor macros: A lisp macro that frobs vectors could, for example, processes code that looks like any of the following: foo[], vector foo, (vector foo), (array 1 ..), etc. if the appropriate reader syntax is defined for them. That's right, ONE macro to do all that.

Ah, that makes sense. Thanks for the explanation.

It's not sarcasm, I'm sure he's implying that while the two techniques have identical names, they are totally different in essence

Although the context (being on news.ycombinator) tends to make me think of Lisp macros, there is no reference to any language in the post's text. I believe c macros still count as 'macros'.

they're preprocessor macros. editor input recorders are also "macros", but they're editor macros. Lisp macros are not generic "macros", unless you're in the company of Lispers, but lisp macros :-P

If you want to explore the highest level of code transformation, grab a nice little book called "Term-Rewriting and all That".

P.S. It's actually not a nice book. It's a theoretical mindfuck that will have you chasing abstract algebra down rabbit holes. It took a good year of my life and I still have no clue. </confession>

Sure, working with code in someone else's domain-specific language is hard. But not as hard as working with a lower-level translation from a domain-specific language which only exists in their head. Even if Clojure were only used for one project in the world, I'm still better off learning Clojure than trying to maintain the JVM bytecode it happens to compile to.

Note the reason discussion on programming languages.

http://gmarceau.qc.ca/blog/2009/05/speed-size-and-dependabil...

Lisp and scheme do pretty poorly on the power scale. Even going back to the original data neither finish in the top 10 and Lisp gets beaten by java.

Reply on news.ycombinator.com