The Failure of Lisp? A Reply To Brandon Werner | Hacker News Reader

The Failure of Lisp? A Reply To Brandon Werner

danweinreb.org

61 pointslackbeard52 comments

Lisp (any lisp, be it lisp-1 or lisp-2) are currently fighting a fight on multiple fronts. From one side, each iteration of Python or Ruby gains new ground as they incorporate more and more lisp-concepts, while being "Visually friendly" (ie, with common syntax). They also have the ability to lure many more users from, say Java or C#.

On the other side, you have any language with type theory governing the language design: haskell, ML, etc. These have long since gobbled up most people interested in type theory and even these are loosing out to the new batch of languages with richer type zoos.

Lisp is not a failure and it has had more influence than people think. However, I think Lisp has played its major part like IBM has played theirs.

While I hear many people complaining about the "awkward" and "hard-to-read" syntax, it's rare to hear someone compliment the syntax of lisp. I am a big fan of the Lisp syntax because it's so simple and consistent - probably a relic of its roots in lambda calculus. Not having to worry about operator precedence is a huge bonus in itself - I don't know why on one hand people cling to PEMDAS, logic, assignment, etc having a complicated hierarchy of precedence and on the other reject lisp because it's too "foreign".

I agree - I think the other strength of it is that macros are less messy then a language with infix operators/functions.

There is something nice about a language (lisp) that doesn't look like anything else, in an artistic sense (not sure if that counts for much).

I think when most people complain about the parens, they are really complaining about the endless ))))))) at the end of an expression.

Interlisp had the last part correct. [ was a special open paren and ] closed all open parens back to (and including) the most recent [.

Nice - I remember someone saying that that in emacs you could have it format it using indentation and now show the parens (or maybe it wasn't emacs).

It was Emacs: http://www.emacswiki.org/cgi-bin/wiki/UnParenMode

In practice, it's just not a problem, because your interactive development environment takes care of it for you. I think the idea that anyone should be expected to program by using something with the power of Microsoft Notepad is an obsolete view. GNU Emacs with Lisp mode makes it very easy to work with the parentheses.

Meanwhile, we have no "dangling else" problem. Also, we have no need to memorize C's 15 levels of operator precedence, so when you see an expression, you never have to puzzle over how the items are grouped. In simple cases of C expressions, of course, it's easy, but it can get complicated. See "Java Puzzlers" by Josh Bloch and Neal Gafter to see how C/Java-style syntax can get you into trouble and fool you. There are pros and cons to each approach (C/Java and Lisp). I've used all of them extensively, and I prefer the Lisp way.

I agree that it's not a problem, but for newcomers all that parenthesis accumulation is quite problematic. I speak out of my personal experience: I've abandoned lisp even before learning something about macros because I couln't stand the parenthesis. I gave a second chance to Lisp some time ago, years after my first attempt. I've walked through Seibel's book entirely and know I love Lisp, but the parenthesis give a really bad first impression, they only make sense after you get to know the language better.

Clarity and unambiguity aren't necessarily the same things. The human brain can handle huge amounts of ambiguity, and our spoken languages reflect this. A programming language is meant to be a bridge between our ambiguous thoughts and the unambiguous calculations of a computer. S-expressions constrain the format of the programmer's dialog with the computer, making it simple and monotonic. For some people this will add a new level of clarity to their work, but for other people it will just limit what they can easily express in code.

High-level languages like Ruby may contain lossy abstractions that obscure the low-level workings of the machine, but Lisp is another lossy abstraction that has the potential to obscure the high-level, abstruse workings of our own minds. Both of these abstractions can be valuable, but neither is without its shortcomings.

Bravo! People tend to boggle when I decry Lisp as being hopelessly low-level.

Sometimes ambiguity can be free, allowing indifference to how the machine works, without obscuring it at all. A good example is classical Hindley-Milner type inference (the algorithm that adds type annotations to plain System F code): it is fairly easy to construct oddball statements where annotating the types is undecidable or would cause resource exhaustion. But it turns out that for the known examples, it would be either impossible to decide the types in any form, or it would take O(c^n) bytes to enumerate the annotations.

So effectively there is no downside to the ambiguity -- the ambiguous statements are already infeasible for other reasons. Everybody Wins!

People tend to boggle when I decry Lisp as being hopelessly low-level.

Intriguing. I'm not really convinced by your example though. In the spirit of conciseness is power[1], can you provide a short program that should be formally expressible in a good enough language, but isn't concisely in Lisp?

Or maybe your definition of low-level is different from mine?

[1] That's the PG version, of course.

I don't buy that concision is power. This borders on a tautology, but expressiveness is power. Concision is part of that (a big part, even), but it's not everything.

My only problem with Lisp is that it's exactly the sum of its parts - no more, no less. Everything decomposes into self-similar pieces, which decompose into other pieces, and on and on, turtles almost all the way down. This is wonderfully elegant, but also feels like it precludes creating something which is more than the sum of its parts. This may not seem like a problem to a lot of people, or just sound like self-important nonsense, but the thing I enjoy most about programming is the tangible sense of gestalt you get when you've been working on a codebase for long enough. For me, Lisp doesn't provide that.

>I don't buy that concision is power. This borders on a tautology, but expressiveness is power. Concision is part of that (a big part, even), but it's not everything.

Yeah, I completely agree. I was going to write about it, but then figured it wasn't relevant to the question.

I'd love to discuss these old PG essays one day.

I'd like to understand what you mean. Is there a way you could provide an illustrative example?

You have an interesting point here that is not often discussed.

I am pretty new to programming, which means that I'm not stuck in conventions about how code syntax is supposed to be. On this background I find lisp pretty easy exactly because the syntax is so simple.

To me lisp is like chess: It's easy to grasp the basic rules, but hard to become a master.

Then Scheme is like Go. Interesting like Common Lisp, but more beautiful.

I agree. I wish ML used s-expressions.

Also, you might like this presentation, "The Swine Before Perl" by Shriram Krishnamurthi (of PLT). It's a great thing to show people who wonder how you could think Lisp has a good syntax: http://www.cs.brown.edu/~sk/Publications/Talks/SwineBeforePe...

(Also: SK's book is great.)

ML and it's successors do not want for s-exprs.

They force an inappropriate fundamental data type on everything, and are only of benefit if you want to fiddle with the insides of extant functions.

Use composition like a real FPer. Learn that there's so much more to data than lists + atoms.

> They force an inappropriate fundamental data type on everything, and are only of benefit if you want to fiddle with the insides of extant functions.

Please explain.

(And I know composition, etc. I also use Haskell and Forth.)

I think what is meant is that the list is the primary data structure for everything in lisps. Fortunately, it is not the only data structure in lisps. I dislike constructs where eg. a graph edge is built as:

'(GRAPH-EDGE <source-vertex> <target-vertex>)

whereas in ML you would write

type edge = { source: Vertex.t; target: Vertex.t }

which is not a list, but an entirely different representation. In Common Lisp the representation could of course be a CLOS object or a DEFSTRUCT (I think, my CL-fu is definitely not that good).

As for the composition, lisps are lacking (nice) currying which is one of the basic building blocks when constructing combinator-libraries.

Ah, I misread it as that ML "force[s] an inappropriate fundamental data type on everything", not s-expressions.

It's true that earlier Lisp dialects had trouble. But in Common Lisp, you can do currying just fine. At work (ITA Software), it's part of our standard utility library.

Just read the slides.

The two comparing s-expressions ("ugly, evil, and an insidious plot hatched by misbegotten academics") and XML ("a hip, cool, great new idea") I found to be particularly brilliant.

I once scribbled down some Scheme on a napkin during a conversion with my sister, a non-programmer. (God knows how it came up, but anyway, it did.) She looked at it, and said two things:

1. So many parentheses!

2. You made a mistake right here.

Lisp will have played its major part when other languages have real macro systems.

Why can't macro systems be part of an IDE? Is there anything macros do that can't be done at compile-time?

Startup idea: web-based language-specific IDE, with a simple interface for sharing/voting on macros between users.

The lisp macro system can more or less be done at compile time, yes, but that's not the problem with other languages. The format of lisp makes it especially easy to do macros - code is data!

Here's an example. Take, for example, this class declaration:

  (defclass dog (animal)
    ((size :accessor dog-size)
     (name :accessor dog-name :init-form "Fido")))

How do you make that into data? Easy. Just quote it:

  '(defclass dog (animal) ...)

Now take the equivalent in Java:

  public class Dog {
    int size;
    String name = "Fido";
  }

How do you get that into data? Well, I suppose the best you could do without modifications to the core language is just put it in a string. But then to do something useful you'd have to do so much tedious string parsing it wouldn't be worth it. I suppose you could invent a special ParseTree class or something, and a quote operator, and etc etc, but that wouldn't be nearly as easy as Lisp makes it.

What the previous poster was saying is that once other languages have a code representation that is easy to work with as data, and can manipulate that code as data at compile-time to produce complex macros, then Lisp might fade. (At least that's how I understood it, correct me if I'm wrong.)

Or:

1. Allow a factory for Class.

2. Make Method contain the full body contents as a list of statements, and give them a factory as well.

Then it's a matter of reasonable constructor syntax.

And a whole lot of "wait, what tree does this code map to again?"

With lisp, there's no difficulty mapping:

   int main(int x) {foo()}

  Method m = new Method(return:=new Type(int),
                        args  := new ArgList(
                                  new Arg(new Type(int),
                                          new Name("x"))),
                        body  := new Block(

.... I'm sure you get the point....

The syntax tree is apparent from the syntax.

  (+ 1 2 (* 3 4))

Maps to:

  '(+ 1 2 (* 3 4))

While people generally talk about using macros for code generation, macros are essentially for doing things at compile time. (Or read time, in the case of read macros, e.g. 'x -> (quote x).) Due to simplicity of Lisp syntax, the easiest way to extend the language is usually by generating code, though, so that's what they're associated with.

You can do things with the C preprocessor at compile time, it's just completely unaware of C, which really cripples it. The Lisp preprocessor (its macro system) is also running Lisp, and can easily process itself.

Other languages don't necessarily need 'real macro systems'

Lisp does, not just because it is so hopelessly low-level (for FP), but also because once you're using s-exprs it's hard to restrain anyone from mogrifying functions.

In a suitable language like Haskell where all functions are composed, using functional composition for everything instead of low-level fiddling with "statements as data" makes a shit-ton of sense. In Haskell you can also use supercompilation to resolve as much of the graph as possible at run time. No need for macros.

Don't think that other language communities want the warts from your language, just because you are so enthusiastic about them.

To be honest I wasn't thinking of Haskell when I mentioned "other languages". Haskell has less need for macros for the reasons you mentioned. Lazy evaluation also makes macros unnecessary for creating your own conditionals. For eager-evaluation languages, or languages without such concise syntax for functional composition, macros fill a real need.

How would you do this in Haskell? http://news.ycombinator.com/item?id=222189

By the way (this is a serious question), what software products have been produced that were written in Haskell?

Besides the Glasgow Haskell Compiler (which probably doesn't count, for your purposes, but it's a good example of a large and complex program), the best example is probably Darcs (http://darcs.net/), a distributed version control system with a powerful patch composition system.

There are further answers from the Haskell wiki here: http://haskell.org/haskellwiki/Haskell_in_practice

Macros are a double edged sword. Their advantages also drags in some disadvantages. If they were only advantageous, they would have flown into mainstream languages long ago(in the full Lisp-sense where the macro is a lisp program).

But they have not. This can be due to multiple reasons: their disadvantages outweigh the advantages or that the effort of getting them into the language simply drags so many other undesirable things with it. Perhaps the advantage of a macro is perceived to be so small compared to e.g., function composition and higher order functions, that it is left out of the language.

Macros haven't flown into mainstream languages, but they've been smuggled in through the back door. C++ templates were not intended to be as useful as they are, but now many of the most powerful C++ libraries use templates as a compile-time metalanguage. Java library developers have also discovered the advantages of the transformation of data into runtime code - except they use XML as the data medium instead of s-expressions, and they use Java reflection to translate the XML instead of using "read" to get a Lisp object.

Ruby and Javascript library authors use the metaprogramming capabilities of their respective languages in combination with XML or JSON or YAML data. Look at ruby on rails, or at jquery.

I disagree about that. It's very, very hard to add Lisp-style macros to something like C or Java. The best attempt to do it for Java is Jonathan Bachrach's Java Syntactic Extender, but as far as I know nobody has built any software with it. It's just too hard to use.

Until you have read a clear description (e.g. Seibel) or had some experience, it's hard to see what macros are really all about.

Lisp macros are extremely important, powerful, and beneficial. Space does not permit me to go into the full explanation; see Peter Siebel's book and the great lecture he gave at Google, available on the web.

Macros allowed us to add object-oriented programming to Common Lisp. CLOS (the Common Lisp Object System) fits in very smoothly into the Common Lisp language, with a nice, clear syntax and very powerful semantics. At work, we have built an object-relational mapping system that, similarly, would be impractical (ugly and verbose to the point of unusability) without macros.

I weren't clear enough, sorry.

Yes, it is extremely hard to add lisp-style macros to C or Java because they have no s-expression structure you can work on. And they do not want an s-expression structure because it is too alien for those programmers. Thus, they perceive that macros are not worth the effort.

I think macros are left out of so many languages because people do not know how to incorporate them while retaining other desirable properties of the language (type safety for instance).

Also, macros are not the only way to build ORM systems. Python does the same with metaclassing as an example. But we can agree on that trying to map it into a statically typed language with reflection/AOP as Java did is doomed to fail I think ;)

I am not trying to say that macros are useless. I am trying to say that some languages leave them out on purpose. Many problems on which macros look nice can be solved with a little extra thought on representation. But then again, many can not.

I see the discussion of macros/no-macros a bit like the discussion on static/dynamic typing. With static typing, we know we are leaving some valid programs outside the gates of the static world. But we think that this loss is negligible compared to the benefits a static check gives us. Yet, some people oppose this viewpoint and wants the valid programs as well, because they perceive the benefits of the static typing to the negligible. Both approaches are valid, but it currently looks like it is hard to get all of the cake in one munch.

Such a respectful reply. To judge by his writing, Weinreb is a genuinely nice man. I'm glad that he has taken on a (semi-) public role as Lisp advocate.

I'm a recent (last 3 years) adopter of Common Lisp who works in it every day. To me Lisp is hardly a failure, it's a secret weapon. For example, a couple of weeks ago I had occasion to take a complex chunk of CL code and make it compile into Javascript as well (I need it to run both on the server and in the browser, under different circumstances). The point isn't that I needed, say, Lisp macros to do this. The point is that I was able to do it pretty easily. As a working programmer, I'm surprised by how often something that feels like it'll be scary and hard turns out to be pretty easy in CL. In fact I often spend more time worrying about it than I do coding it (which makes me wonder how long it will take before certain habits that come from experience in less productive environments die off).

This isn't the sort of thing you can really analyze on the level of language features. I find that a lot of the discourse around language comparison is the activity of researchers and hobbyists (for want of a better term) who aren't immediately concerned with building major systems. They like to explore many different things and the programming tasks they take on tend to be conducive to that - coding exercises and such. That's great, but not the same perspective as when you throw yourself into making one big system over months or years. To the first sort of person, the various Lisps and startup annoyances and incompatibilities constitute a huge barrier, enough to say "why bother" and move on. To me they were an investment that paid off relatively quickly and has since produced big gains. (I mean technically. We'll see if they cross over economically.)

Lisp isn't going to have a major resurgence on language grounds alone. It's too old for that, and too suboptimal for blog posts. What would, though, make a big difference is seeing more successful products made with it. That may sound odd as there have been few so far, but that's where I think the untapped potential of Lisp is. And if that ever happens, an ecosystem might develop that has fewer of the dysfunctional aspects of the past. Successful companies that use Lisp would fund open source development, and so on.

Successful companies that use Lisp would fund open source development, and so on.

Companies like these? http://www.pchristensen.com/blog/lisp-companies/

Prominent examples being ITA developing a replacement for ASDF and Clozure developing, well, Clozure.

Thank you for the kind words!

I don't think that Brandon Werner said that Lisp was a failure as a language. As I read it, he's saying that Lisp has failed to capture a lot of new users, get very popular, and so on.

I agree with you that a lot of the really big advantages of Lisp show up better in large systems than in small coding exercises.

There have actually been lots of successful products made with Lisp. People just don't say so, sometimes because Lisp is unpopular and sometimes because it's just not important. InspireData is a great product, but there's no need for anyone to know what language it's in: in fact, it's in the LispWorks implementation of Common Lisp.

For links to success stories, see my survey paper at http://common-lisp.net/~dlw/LispSurvey.html and see the "Success Stories" section.

Lisp is the second oldest high level language, having barely been pipped by FORTRAN... and it's still around, and under active development. Long way from a failure.

Equally, with that kind of a trajectory and history, you'd also have to predict Lisp isn't going anywhere soon.

If anything, the rising interest and competition in functional languages is pushing Lisp back into the spotlight.

I don't know CL, but from the subset of scheme that i know, i cant imagine how can this language fail. It just feels too good and the code is so beautiful. I could say for certain that scheme has made me into a better programmer.

Common Lisp and (especially!) Scheme are really elegant languages, sure, but there's really something to be said for being able to install the one clear Python implementation* and getting a huge standard library with lots of documentation and a large community, upfront, versus trying to figure out whether you should set up SBCL or CLISP or PLT or Chicken or ..., then finding there are several libraries for what you're trying to do, but you're not sure they're compatible with the compiler/interpreter you're using, etc. And then, what editor do you use? ("What's this SLIME thing about?") And this, that, and the other thing. That is a lot to figure out, all at once and upfront, before you can try it out. Making it really hard for curious people to play around with Lisp means that few people will see its potential.

All of the above could just as well apply to Ruby or even Haskell. I think Python, in particular, has been really carefully designed to give a good first impression, though. I don't know if that was deliberate or not, but it's brilliant. (I know some people balk at the indentation-based syntax, but I bet many people coming from e.g. a Perl or C++ background think, "Whoa, this looks really clean." "Executable pseudocode.") Lisp doesn't really seem to focus on making a good first impression. (On the other hand, OCaml just says, "Hello!" and immediately backhands you. It makes up for it once you understand the type system, but I'm not at all surprised the OCaml community is small.)

Stuff like Lisp in a Box helps a lot, but it wouldn't hurt to have a link to it in 300-pixel-tall burning letters at the top of lisp.org, you know? You can't even go directly to a download page for any implementation from lisp.org. (When you click the download link at the top of Haskell.org, it takes you right to GHC's download page.)

I've also frequently heard the Lisp community can be rude or off-putting, but I haven't experienced this personally. I could see them being a bit gruff and frustrated about answering a lot of the same questions over and over again, though, and seeing languages like Python and Ruby getting so much more attention than Lisp can be heartbreaking.

So, basically: Marketing.

* I know about Stackless, etc., but which does python.org point you to? QED.

Yes, lisp.org needs to be (and will be) vastly improved. We definitely need the equivalent of a CPAN for Lisp, and several of us are working on producing such a thing. You're quite about marketing!

As for multiple implementations: CPython, Jython, IronPython, Stackless Python, PyPy. What editor do you use? Gnu Emacs with Slime; or, some of the commercial Lisp implementations come with IDE's; and there's work being done ("curl") to make Eclipse work well with Common Lisp. Java has several IDE's, such as IntelliJ and Eclipse. For Python, what I'm told by the Python experts I know is that they just don't use any IDE.

But much of what you're saying above is very much to the point. The comments on my blog posting go into these issues more.

> We definitely need the equivalent of a CPAN for Lisp

Agreed! I like the Eggs repository for Chicken (http://chicken.wiki.br/Eggs%20Unlimited%203), but individual Scheme implementations just aren't compatible enough for such things to really benefit them all -- I'm pretty sure the lack of a module system in the core for R5RS is to blame. There's one in R6RS, but I hope it's not too little, too late.

Some may consider Lisp a failure of a language, but I sure am glad my startup is written in Lisp! The issue of syntax baffles me, because in recently learning Javascript, the essential area of bugs for me were almost all syntax related. I never have to worry about syntax in lisp as its either ( or ). There are no ; or } or { or , etc to worry about what goes where when.

The logical structure reveals itself so clearly in the form of indentation so that the parenthesis become transparent for me. Having emacs slime automate the indenting is of course essential in enabling a clear focus on the function of the code instead of spending time thinking about the syntax.

Lisp's biggest problem is the simple fact that most other languages already ship with most nonwindows boxes. _Successful_Lisp_ is a good book (amongst others), but what do you type the code into?

Until today, I hadn't heard of SBCL. I've been fiddling in e-lisp, which is a tragedy of its own sorts. The only mention of SBCL in "google:lisp linux" is an ad. I don't think I'm the only one still afraid of a $5k entrance fee for lisp.

Soon, lisp.org will be redone to make it easy to find good free Lisp implementations. Meanwhile, check out my survey paper at http://common-lisp.net/~dlw/LispSurvey.html.

"the community is judgmental and unfriendly to newcomers and thorny and un-inspiring."

Brandon's reply agrees with this criticism, because "I have heard this same criticism from other people than you, and at this point I assume it must really be true."

The comments here focus on lisp. Lisp isn't the problem.

In my college days I was very interested in AI and learned Prolog and Lisp.

Reply on news.ycombinator.com