Hacker News Reader

Comment by Todd | Hacker News Reader

That's it, in a nutshell. The macro facility is what differentiates Lisp from other dynamic languages that have, seemingly, all of the same attributes--dynamic/lexical scope, closures, etc.

They allow you to write a true DSL. Paul Graham pretty much wrote the book on the subject (On Lisp) which he graciously offers as a free download since it's out of print.

View on HN

As a data point, in the source code of Arc and its libraries + the web server + News.YC + all of YC's internal applications, there are 738 calls to def (= CL defun) and 187 calls to mac (= CL defmacro). This isn't a complete list, because there are other defining operators, but it gives you an idea how large a role macros play in a typical Lisp application.

If you measured by uses of each type of operator, the role of macros would be even larger. You often write a function that you only call once, but you wouldn't do that with a macro.

Here's the last piece of code I wrote. I wrote it because I was curious to see what colors users had chosen for the top bar.

  (defopa topcolors req
    (minipage "Topcolors"
      (tab
        (each user (sort (compare > [karma _])
                         (keep [aand (uvar _ topcolor)
                                     (isnt it (hexrep orange))] 
                               (keys profs*)))
          (tr (td (link user (user-url user)))
              (tdcolor (hex>color (uvar user topcolor)) (hspace 30)))))))

There are 10 macros used here: defopa, minipage, tab, each, karma, aand, uvar, tr, td, tdcolor.

Actually it's a hack that karma is a macro. I wanted it to be settable and was too lazy to define a setter, so I just defined it as a macro. It should be a function, and I should be able to say just (compare > karma).

> and I should be able to say just (compare > karma).

I suggest you implement a terse syntax for partial-apply. For my own pet lisps, I use {}. (You are welcome to steal it ;-)

So {+ 1} would expand to (in scheme) something like (lambda rest (apply + `(1 ,@rest)). You could keep partial-applying arguments

  (do
    (= x {+ 1})
    (= y {x 2})

then actually apply it with ()

  (y 3 4 5) ; => 15

Your accumulator generator in Arc:

  (def foo (n) [++ n _])

Could be shortened to:

  (def foo (n) {++ n})   ; even terser than regular Arc!

The only time you'd need _ is when you want to change the order of the arguments (e.g. you'd never need {+ _ 2}, but you might need {/ _ 3}, or even {/ _ _ _ 3} if you want to jam the first three arguments in front.

This has some other interesting properties:

- In a lisp-1 OR lisp-2, #'foo can be represented as {foo}.

- Your karma macro can fit in there as (compare > {karma})

- With the _ variables, it might even be a general case of your [] syntax.

- {} stands out visually. "Here be electric magic."

- If you ever implement generic functions dispatched left-to-right, you can implement partial dispatch for partial application.

I just entered this variant into the server's repl so everyone can see the range of choices:

  (defop topcolors req
    (minipage "Custom Colors"
      (tab 
        (each c (dedup (map downcase (trues [uvar _ topcolor] (keys profs*))))
          (tr (td c) (tdcolor (hex>color c) (hspace 30)))))))

Defopa means only admins can use something, but since this is defined with defop, everyone can see this page: http://news.ycombinator.com/topcolors.

(before I begin: I am familiar with macros & DSLs, code generation, building a custom language layer and then write your app in that custom language and so on, and I buy into almost all of it)

I am sure the code above is poetry for those deep into it, but for a person who is a bit rusty (or god forbid, new to the code base), this is basically indecipherable. Look at how many tokens are there in a line like

(each c (dedup (map downcase (trues [uvar _ topcolor] (keys profs*))))

How many people can answer the question of which ones are macros, functions, simple identifiers and so on? In that particular example I am stumped at [uvar _ topcolor] and can't get past it.

Yes, I have read Paul's thesis on Lisp code density (quick summary: each line is dense, but there is a lot less overall code so it evens out or better), but my counter would be that there is a density beyond which most mortal brains can't penetrate.

Without intending to cause offense, I am going to say that this is tending to write-only code. Sure, the equivalent Ruby or Python (leave alone blubs like Java or C#) would be pages long, but I suspect that lower density leads to better understandability.

Let me see if I can translate that in to Ruby. Note that the expression isn't closed, so I'm changing it to:

  (each c (dedup (map downcase (trues [uvar _ topcolor] (keys profs*))))
    (do-stuff-with c))

which I think translates roughly to:

  keys(profs*).find_all{|prof| uvar(prof, topcolor)}.map(lambda {|str| str.downcase}).uniq.each do |c|
    do_stuff_with(c)
  end

This could be slightly off, as I'm having to guess at what some of this stuff does. It's almost as dense and no more readable.

For me, your ruby version is more readable because the order is more sequential (like a pipeline). The arc version has to be read from the inside-out, vs left-to-right in ruby.

I always read s-expressions left-to-right. The outer expression tells you what you're getting. I usually don't have to go very far in before I know what the code is intended to do.

Look at how many tokens are there in a line like

There's not really a concept of lines in Lisp; you can break expressions wherever you like. E.g. you could also write:

  (defop topcolors req
    (minipage "Custom Colors"    
      (tab 
        (each c (dedup (map downcase 
                            (trues [uvar _ topcolor]    
                                   (keys profs*))))
          (tr (td c)
              (tdcolor (hex>color c) 
                (hspace 30)))))))

With dense code, you have to think about every word. With sparse code, you have to think along with every line/block/idiom. Both require the (same?) thought process, but with dense code you get two advantages:

- You can look ahead easier and get a bird's-eye-view - Once you learn an operator, recognizing it (and writing it correctly) becomes a single word instead of said line/block/idiom.

My guess is that [uvar _ topcolor] is a function similar to (lambda (x) (uvar x topcolor)). That's just a guess, but once I learn whatever it is, I'd wage that it's easier to recognize and debug the bracket syntax.

I wonder what makes the difference in person's perception of code. I think I have a good idea of what the pg's code fragment does at a first grance. (I do know what [ _ ] means in Arc, though; pg mentioned it somewhere.)

I doubt it's the density that's the problem. I also doubt macros are. It's more like whether you recognize "the way" in which you're supposed to write, in the given language. Any language, verbose or terse, is indecipherable until you get familiar with the way even you know the syntax and semantics of the language.

Granted. To be fair to Paul, I got about 60% of it based on a (rusty) idea of lisp. The _ got me (should've thought of Perl)... but now it is clear.

My broader point still holds though - there is a maximal density ...

And the reason I commented on it was that in posting that code fragment, the message was that it would be "kind of obvious" to any reasonably competent programmers how macros really help. I agree macros really do help, so no quarrel there.

In this case, we understand the "application" perfectly, and then we get to see the code. It still got (some of us) stumped, and (some of us again) actually know about macros and are moderately familiar with Lisp idioms. That is why I asked if density can go too far.

Compare that to something like LINQ, which I am coincidentally studying. It is a DSL for querying object collections (a similar use case to the code here). With no special background, you can go in, read code, and get a reasonable idea of what is going on.

LINQ actually lets you get rid of a lot of code (something I agree with in general), so it lives much higher on the density scale than a typical blub.

I don't think the point was that everyone was supposed to be able to understand every detail of the code. I think he was showing that by using a bunch of macros, he could add a new command to the web server that would create a new type of page with a very small amount of code. How would the code look to create the top color page if it was written in another language?

Less code = faster development, easier maintenance, fewer bugs

I see your point. So the question is actually whether a language designer can only care his "types"---those who have similar thinking habit and thus has less difficulty to understand the code---or should target at larger audiences.

Ultimately that's the choice of the designer; not all languages need to address to the broad range of programmers.

You're just confused by some syntactic sugar. [uvar _ topcolor] is shorthand for (lambda (x) (uvar x topcolor)), according to "Arc at 3 Weeks"

each = mutant foreach

dedup = remove duplicates

(trues [] (keys profs*)) would mean "map the lambda (predicate) over the list of keys to the hash of profiles, returning a list of elements for which it returned #t."

A few more specifics about the application and language would make this comprehensible.

Close, except trues returns the values returned by the function, not the elements for which the function returned true.

  (def keep (f seq) (rem [no (f _)] seq))

  (def trues (f seq) (rem nil (map f seq)))

Actually, the equivalent Ruby/Python would use a simple SQL query. I have long noticed how SQL is the poor-man's functional "DSL" for such uses ...

I don't think that's a problem with macros so much as obscure naming conventions and an obsession with brevity. In the original example, I had a pretty good idea what minipage, tab, each, dedup, map, keys, downcase, tr, td, and tdcolor mean. I can figure out c (color?) and req (request?) from context. I have no clue about defopa, uvar, trues, _, [], profs, topcolor, hex>color or hspace (or I have guesses, but wouldn't trust my guesses enough to pin down the semantics).

From PG's post above, it seems like those are fairly evenly distributed across macros and functions. I don't need to know whether td is a macro or function to guess that it outputs a table data cell. I do need to know that it outputs a table data cell, and I only know that because I'm familiar with HTML.

Agreed on the obsession with brevity.

Reminds me of mathematical proofs, where you work out everything offline, and then make the proof elegantly brief. There is a similar process going on here - writing macros really does boil down to that kind of offline work.

Just as with such elegant proofs, though, you do need a pencil and paper to trace through the steps. And if the programmer were anything but of the highest caliber, it is fairly easy to get tripped up.

The way blub languages handle (the absence of) macros for td, tr etc. is to invent templating languages (JSP etc) where you embed Java inside HTML. You basically have a DSL living in a separate syntactic space. That gets the job done, for the specific problem space at hand (web apps).

I have come to like that syntactic space separation: HTML lives as HTML, SQL lives as SQL and so on. With macros, you are getting everything to live in the same syntactic space, so you get the proliferation of tokens, without that structural separation that aids in comprehension.

That calls for the feature to enable color picking by clicking on the colors ;-)

Now you don't think pg's time would be better spent writing essays, do you? :-)

http://news.ycombinator.com/item?id=97830

Just trying to go with the flow ;-)

I thought that Arc is written in Scheme, not in Common Lisp? Are Scheme Macros equally powerful?

Edit: OK, perhaps Arc is written in Scheme, but Arc itself features CL like macros, that would explain it ;-)

Arc is written in Scheme in the sense that it compiles into Scheme, and uses the Scheme read, numbers, etc. But Arc macros are not Scheme macros.

Scheme also has "CL" like macros that are hygienic. Scheme 48 for instance, supports explicit renaming macros and MIT has syntactic closures. However, most schemes have a library that adds your traditional defmacro style macros, CL users are likely to want, but there use is almost always discouraged.

As far as I know Arc is written on top of MzScheme. Besides its 'clean' macros - MzScheme supports CL like macros, too. But your explanation could be correct as well.

Reply on news.ycombinator.com