Hacker News Reader

I'm really tired of hearing this:

syntactic abstractions dont scale up with the number of programmers

... as an objection to Lisp macros. It has become the invariable staple of these discussions and is accepted (by the people who accept it) without any question. Yet is there any evidence for it? I don't think I've ever even seen any evidence alleged for it.

The fact is, to justify this statement requires more than proving it's true. You'd have to prove that this failure of syntactic abstraction to "scale up with the number of programmers" is worse than the failure of abstraction in general to do so. Indeed, in my experience, it's programming that fails to scale up with the number of programmers.

I dont know what you count as evidence but here are two eminent authorities speaking about related topics - Turing award winner Barbara Liskov - http://www.infoq.com/presentations/liskov-power-of-abstracti... (Syntactic abstraction related talk starts at minute 17).

Heres what Dijkstra has to say about something that is tangentially related (he is talking about the importance of read time comprehensibility of code). "My second remark is that our intellectual powers are rather geared to master static relations and that our powers to visualize processes evolving in time are relatively poorly developed. For that reason we should do (as wise programmers aware of our limitations) our utmost to shorten the conceptual gap between the static program and the dynamic process, to make the correspondence between the program (spread out in text space) and the process (spread out in time) as trivial as possible." This is from the influential GOTO considered harmful paper.

"Lisp has jokingly been called "the most intelligent way to misuse a computer". I think that description is a great compliment because it transmits the full flavor of liberation: it has assisted a number of our most gifted fellow humans in thinking previously impossible thoughts."

- Edsger Dijkstra, CACM, 15:10

Dijkstra's quote sounds more like an argument for defaulting to immutability than anything about syntactic abstractions. (I happen to agree with him, there.)

"to make the correspondence between the program (spread out in text space) and the process (spread out in time) as trivial as possible". Most programmers read programs as plain source most of the time, ie not as macroexpanded source. If editors showed macroexpanded source code by default, things would be different. Also macroexpansion does in practice affect the ability of the compiler to highlight what line caused the error, or the ability to step through code intuitively (you are stepping through macroexpanded code and sometimes its tiresome to be able to intuitively understand how that relates to code that you wrote).

Not to take anything away from Dijkstra and Liskov, but of course I don't count their opinions as evidence; eminent authority opinion is what one resorts to in the absence of evidence.

I don't have time to watch the video, but it seems rather obvious that the Dijkstra quote has little to do with macros. What he's saying sounds to me like "lexical scoping is easier to understand than dynamic scoping". If anything, that makes macros more valuable rather than less, since macros are intrinsically lexical and macroexpansions don't typically change over time.

Let's assume for a moment that the critique is correct: macros don't scale. OK. The C++ templating system can be used to accomplish many of the same things as a good macro system (see http://matt.might.net/articles/lambda-style-anonymous-functi... for one example). Does the C++ templating system "scale"?

Or, to put it another way: lots of C++ shops ban the use of advanced templating metaprogramming. Has C++ suffered egregiously as a result?

How does assuming that macros don't scale help in proving that macros don't scale?

How does the complexity of C++ templates prove anything about syntactic abstraction in general or Lisp macros in particular?

C++ templates are both less powerful and more complex and fragile than lisp macros. But no one ever says "oh, the C++ templating system does not scale" or "oh, C++ will never be successful until it eliminates/replaces the templating system"...do they?

I think I misunderstood your original point. Thanks for the clarification.

Prolog also has syntactic abstractions, though it doesn't have a culture that uses them to the same extent because (overgeneralizing a bit) every expression is quoted by default, and it also has extremely powerful pattern-matching. That (or good support for lazy evaluation, in other languages) covers many of the same use cases as Lisp-style macros.

Also, K has a "whizbang sleek look" that makes Lisp look ridiculously verbose. (http://www.nsl.com/papers/kisntlisp.htm) Ditto J and APL, though K seems to take it furthest. Part of their density comes from being based on ideograms, not lexical tokens; the density is like comparing Chinese to English.

Unfortunately, K is closed source and limited to 32-bit platforms for non-commercial evaluation. It's also quite expensive, since its primary product is a mind-bogglingly fast database designed for real-time stock data analysis. I'm been fascinated with K for a while, though, albeit mostly at a distance. (J's evaluation isn't limited to noncommercial use, though.)

Thanks for pointing K out, hadnt seen it so far.

There's a lot of stuff about it at http://nsl.com, and you can get an evaluation version of Q (the newest K dialect) and associated documentation from http://www.kx.com.

K, J, Lisp, Prolog . . . I'm beginning to be worried about my own fascination with uncommon languages.

Is there a support group available? If the three books you read when the kids are asleep are The Reasoned Schemer, The Art of Prolog, and On Lisp, are you now officially beyond help on the language geek scale? I used to be a somewhat productive programmer. These days I think more about making sure I've picked the right language for the problem . . .

I spent a while chasing the weirdest languages I could find (relative to everything else I knew at the time), but feel like I've gotten to the point of diminishing returns. I've been studying types and semantics instead.

I do most of my actual programming in Lua + C and an in-house language. I really like Erlang (but haven't worked on the right kind of projects lately), and I also prototype stuff in Prolog.

I would love to have a more modern dialect of Prolog, designed primarily for embedded use as a C library (much like Lua, and ideally with a similar C FFI). It's a cool language, but it's also really strongly skewed towards certain problem domains. Lua has taught me that when a language can delegate its weak points to a symbiotic language, it can stay really small and focused on its strengths.

Well reasoned for the most part, but Lisp has occupied the high ground for development tools for a long time (REPL, SLIME, trace/untrace, step, describe/inspect, room, macroexpand). Other languages have been playing catch up in this regard as well. Whether Lisp still occupies the high ground today is unclear, but it is erroneous to say that Lisp programmers envy the development tools of other languages.

Good point about slime (its my development environment of choice). Lisp tools are built for dynamic development from the ground up, so on that aspect they are very good. However finding a lisp source stepper that works has been an exercise in frustration for me. The best one ive found so far has been the PLT scheme stepper that mostly works.

With regard to development tools, maybe he was referring to libraries and community support.

Perl, a slow language criticized for its syntax and bearing the stigma of being a mere scripting language, is often lauded as having huge community support and tons of helpful (even if often crappy) libraries via CPAN. C++ has similar support (no CPAN that I am aware of, but I have always quickly found solutions and libraries with the search engine).

Granted, CL can connect via U/CFFI to any external library, but it is harder to find the wrappers for CL than C/C++ header files.

Insert Clojure rant here :)

I think there's one other important lisp feature that hasn't been widely adopted: compile-time metaprogramming. A lot of lisp macro use boils down "allow developer to generate lots of new code based on tiny fragments of code he writes." The Python community has adopted this to good effect, but the Java/C++ community really hasn't. The funny thing is, the need for good code generation doesn't go away simply because the language provides no facilities for it. That's why C++'s broken template system is so often abused to provide compile-time metaprogramming. And that's why Java added annotation factory support. The preferred use for annotation factories is to register a factory class that the compiler will call when compiling code that uses your annotation. Your factory can then access a special compile time API that allows it to create new Java source files based on the AST that the compiler feeds it. More sketchily, your factory can modify the AST in place. This is an insanely baroque way of getting compile time metaprogramming but it solves a real problem.

I dunno what life inside other big companies is like, but Google uses compile-time metaprogramming all the time. There're tons (perhaps too many) "little languages" used inside Google. Some have even been open-sourced, eg. protobufs, google-ctemplates, Closure Compiler. And if their open-source output is any guide, FaceBook uses tons of little languages as well.

It's just that they're done the UNIX way, with a small miniature compiler or XML substrate, instead of the Lisp way, where you write everything in S-exprs. This tends to make the little languages easier to grok for users but harder to implement, which tends to be a good tradeoff, as there will be far more users than implementers of a programming language.

Sure, you can totally do compile-time metaprogramming without language support, but the barrier to entry is much higher. That means that, at the margin, you're less likely to try it, especially for small projects. And most projects start small.

I totally agree that good development shops do lots of compile-time metaprogramming. The issue is whether they do enough when dealing with problems (that start out as) too small to justify writing a full blown parser/compiler, and if they don't, how much of the shortage is attributable to a lack of language support in C++/Java.

I have been trying to understand the problem with syntactic abstraction in other languages -- or creating your own languages to fit the problem. Is it a real issue? or is the problem with it just same problem people have with any unfamiliar codebase where the developers made unclear or poor abstractions?

To clarify what I mean, I have presented lisp ideas to fellow coworkers and met with resistance along two lines:

(1) They want to understand what is really going on, and the abstraction can obscure that. This sounds to me like a general abstraction problem more than an issue with syntactic abstraction... and really, it seems like a problem with understanding how someone chose to abstract things. They want to look underneath the hood for answers.

(2) They suggest that new developers will have a hard time understanding and adding to the system. But new developers for any project have to learn the local language anyway. It may look like C++, python, or whatever, but it the framework other developers developed always has its own learning curve.

Surely syntactic abstraction is just another abstraction tool that can be used to help make those frameworks easier to dive into. There is always poor code and poor documentation; this is not an excuse not to declare functions or other pretty approaches. Why is it any scarier to use this tool than to use someone's library?

There is a small difference - syntax affects the read time comprehensibility of code in ways other abstractions dont. For eg in C++ an overridden "+" operator could be doing anything in the background and there would be no way for you to know by a casual read of the code. Lisp has macroexpand-l for this, but one needs to be dilligent enough to use it on new pieces of code.

Since in C++ the + operator is actually a class method in C++, it sounds like all abstraction suffers the same potential problem of "no way for you to know by a casual read of the code".

That is a good point on the macroexpand functions; I would hope that if syntactic abstraction was added to C++, there would be a similar way to view the resulting code (sounds like a great future gdb / MSVC debugger feature).

I have found that while I mostly understand C++ as a language and respect it for what it tries to accomplish, the development environments are absolutely atrocious in many ways, and I spend more time fighting them than writing code. This has made me conclude that even though I could in theory program in C++, it is waste of human resources to do so with the current generation of development tools.

Ocaml also allows syntactic abstractions nowadays. Of course it's more complicated to use than in Lisp, because the syntax of Ocaml is more complicated in the first place.

To some degree, I feel that syntactic abstraction is a premature optimization. Many (most?) benefits of macros can be replaced by a combination of lazy evaluation and functional programming. Most of the performance benefits can be obtained by a sufficiently smart compiler which does partial evaluation. I use macros in lisp a lot, and I often wish I had them in Python/Java/C++.

In Haskell, I can only think of one very narrow use for them which has never actually come up for me in practical use (automatically generating instances for combined monads).

Out of curiosity, can anyone provide a more common use of macros which can't easily be done in Haskell with HOF/Lazy evaluation?

The only things I can think of are things that wouldn't enter the head of a Haskell programmer to begin with, such as syntactic shortcuts for complicated side effects.

So you're right, lazy evaluation can do just about anything in a purely functional environment. But now count the number of programming languages that provide a purely functional environment...

The rest of us need macros.

...such as syntactic shortcuts for complicated side effects.

I can think of a couple of examples where I've done such things in lisp (mostly elisp, sometimes Clojure), but I don't see a good reason why I couldn't do them in Haskell. In fact, a function f: a -> M d (with M = IO, Writer or State if you want side effects) almost serves the same purpose: it turns a value into a set of actions to perform.

I'm really curious, could you give an example of such a task/macro?

(Note: I'm not trying to engage in a language war, I'm honestly curious. If I'm treating Haskell as Blub, I'd like to know about it.)

OBLIG

"Greenspun's Tenth Rule of Programming: any sufficiently complicated C or Fortran program contains an ad hoc informally-specified bug-ridden slow implementation of half of Common Lisp."

- Philip Greenspun

"Including Common Lisp."

- Robert Morris

"We were not out to win over the Lisp programmers; we were after the C++ programmers. We managed to drag a lot of them about halfway to Lisp."

- Guy Steele, Java spec co-author