Lisp is Too Powerful
c2.com
c2.com
The faulty reasoning is this: If Hacker Hortense does something with powerful language L that Newbie Nathan finds confusing, we assume that language J won’t permit Hortense to do that, and therefore if we standardize on language J, fewer bad things will happen.
This reasoning is faulty. First off, if Hortense and Nathan don’t see eye to eye on how to write programs, no language will solve the problem, because you have a disparity of experience and/or education. If what you want is code Nathan will like, and you don’t want to raise Nathan to Hortense’s level, you have to drag Hortense down to Nathan’s level with coding standards. And you could have done that in Language L just as easily as J. If you ban L, what will you do when Hortense implements Parser Combinators and Monads in J?
The first flaw is the assumption that language features introduce complexity, when in reality it isn’t the features, it’s the solution to the problem that confuses.
This indirectly leads into the second fault with the reasoning. There is a hidden assumption that Hortense is going to write so much code, and there will be so many “problems” Nathan will have with his code, and every problem we eliminate is one fewer problem in the result. This is not how software works.
Let’s say Hortense writes some code in L, and Nathan complains that there are seventeen things he doesn’t understand. Ten of them rely on features of L, seven are independent of the language. If we tell Hortense to rewrite things in J, we cannot assume that the result will have seven things Nathan doesn’t understand. It might have even more as Hortsense works around J’s deficiencies.
For a demonstration of this, look at any modern Java Dependency Injection Framework. These complex and opaque software machines are made up of of XML, interfaces, and classes. They exist in no little part because Java lacks the reflection and meta-programming of languages like Lisp or even Ruby. Are these thing simple by virtue of being written in Java instead of Lisp?
Imagine for a moment that DI frameworks don’t exist, and that Hortense had built something in Lisp with macros for dependency injection. Nathan is nonplussed, so Hortense rewrites it in Java and rolls her own DI infrastructure. Will the result really have fewer things that Nathan doesn’t understand? Or will it be even more inscrutable thanks to the accidental complexity of working around Java's limits??
The “problem” is not that some languages are too powerful, it is that people imagine that teams of programmers can work together on complex problems without communicating, and some people assume that when there is a disparity of experience of knowledge, that people can work together without educating or learning from each other.
A former boss of mine said that programmers are like hairdressers: whenever she changes to a new stylist, the first thing they say is, "Oh my god, what did that last person do?" That's certainly what I say when I have to examine an existing codebase. More expressive languages allow for more stylistic variation, and one's style is deeply rooted in one's values and experiences. One person's brilliance is another person's WTF.
I think the right solution to this involves developing a house style, which for my team involves zero code ownership and a lot of collaboration. Eventually the team converges on a set of idioms that work for everybody, so style issues are minimal.
But that's not always the working reality. If I have to inherit a bunch of code from a random developer, I'd much rather it be in Java than, say, Ruby. Java definitely limits brilliance, but it also limits too-clever-by-half idiocy. The downside risk with a Java code base is smaller.
If you want to say that the kinds of companies that hire Java developers don’t hire clever idiots, that’s a different discussion. But I’ve seen so many convoluted architectures built out of Java that it feels to me like too-clever-by-half is a constant across all languages, and the very best thing for me is discouraging cleverness, with second best being let people do their clever things in a language that helps them, rather than making them write even more inscrutable clever code just to get away with being clever in Java.
This is my point above, and perhaps it doesn’t match your experience, namely that some clever things will be done, and Java will not prevent them from being done, but merely prevent them from being done simply and cleanly. So why not have this other “pro” do what she is going to do, but do it simply and cleanly thanks to a powerful language, rather than resorting to XML and other shenanigans?
But honestly, I fear we are now having a weird discussion where our working assumption is inheriting code from people we don’t respect. It’s far, far easier to simply work with reasonable, sane people, than to make all your choices predicated on the idea that your colleagues are psychopaths.
It makes for witty sayings, but terrible careers.
Broadly, though, I think you're right: it's irrelevant to the real problem. Java is the new COBOL because most large software development organizations are functionally insane. They can't tell good code from bad, and rig things in a way that bad code is almost guaranteed anyhow. So naturally they'd prefer a language in which bad code is merely painful, not fatal. Because that lets them cover up their failings for much longer periods.
My personal solution is yours: stop working with other people's shitty code. I'd like to see the problem solved across the industry, but focusing on language choice doesn't seem like it's getting us anywhere.
In Java there is a sort of ceiling, one might call it an upper bound on the mindfuckery per square inch of code. In 100 dense lines of lisp, there is no a priori reason to assume that no one has written a recursive descent macro-expanding s-expression handler for arbitrary routing of network packets in a rules-based DSL. In the same amount of dense java code there are rarely more than 200 method calls, all statically checked, and rarely more than 100 variables, typed and scoped.
From John Carmack's recent article on static checking: "If you have a large enough codebase, any class of error that is syntactically legal probably exists there." Now, he is concerned with actual defects, but the same rules apply even more strongly for stylistic rules. As the size of a codebase increases, the probability that any particular language feature is absent approaches zero. And in a language like lisp, where you can write your own language features, this is moderately horrifying.
The fact that you can replicate java's dependency injection in a few orders of magnitude less code in lisp is not a comfort to me. Because the 10,000 lines of code in Java's Dependency Injection Framework is a red flag to me. The chances that someone who writes the same thing in lisp has drastically simplified the implementation are not so high.
We agree.
The chances that someone who writes the same thing in lisp has drastically simplified the implementation are not so high.
We disagree.
First, when I roll my own, I scratch only my own itch. I don’t need to build something that works for everyone, everywhere. It’s like Microsoft Word: MSFT boasts that most users only need 5% of what it does, but every niche of users uses a different 5% of the whole thing.
But can I roll my own? Well, I suggest that the answer is more likely to be “yes” in Lisp than in Java. First, folklore suggests that defects are constant per line of code. Therefore, if I need fewer lines of Lisp than of Java, I should have fewer defects to contend with. I assume three things. First, I need much less than the full framework’s functionality. Second, Lisp is more expressive than Java, so I need fewer lines of Lisp for any functionality than of Java. Third, I suggest that languages with meta-programming support are particularly well suited for tasks like dependency injection, reducing the amount of code I need to write even further.
Now, the big DI framework is written by someone else. So is it free? No. I need time to learn it, time to use it, I can make mistakes using it, I can get an XML configuration wrong, I can implement an interface when I am supposed to extend an abstract class, I am not immune from defects just because I am using a library.
So the net question for me is whether the chances of successfully rolling my own feature in Lisp for my project’s specific needs are greater than the chances of successfully using an existing framework in the Java world=, and parallel to that, the question is whether someone else working with my code will find it easier to decipher the XML configuration and interfaces and classes I have written to work with a Java DI framework or will find it easier to work with the smaller, simpler and more compact Lisp code written for this specific project.
Reasonable people can go either way on this, I find it hard to believe that Java is “obviously” a win, especially if they’ve used one of these big frameworks with the many gotchas (as I have).
p.s. Of course, the wild card is that there are plenty of libraries in Lisp as well. I reject the notion that every Lisp programmer reinvents everything from scratch: https://github.com/lprefontaine/Boing
Will it turn out to be "Lisp and a handful of super-programmers"? I don't know. What we need is an age of experimentation. The great thing is that startup costs are now so low that we are beginning to see that happen. Emphasis on beginning.
Keep in mind that I'm only suggesting that a crossover point exists, I don't pretend to know where exactly it is. In order for me to be wrong, a single superior programmer would always have to be better than two slightly inferior programmers working with a slightly less expressive language. I strongly doubt that this is true. The simplest explanation for what we observe is that in fact a team of inferior programmers working in parallel can be more efficient than a single superior programmer working alone. Not always, but often enough to prevent more expressive but less comprehensible languages from becoming dominant. What constitutes "expressive" and "comprehensible" will evolve over time, as you suggest (maybe Lisp will someday become tomorrow's Java!), but the underlying scaling law will remain.
But I wonder if this is wishful thinking, if this isn't just another case of the prisoner's dilemma. Perhaps like how cities with mostly poor people would collaborate many times in history to conquer neighboring barbarians, even though the barbarians had more freedom and were thus richer. (See http://en.wikipedia.org/wiki/Fates_of_Nations.)
Then again, there's reason for hope. Perhaps the parallelizable sort of programming is more menial. It certainly seems that way with the way communication costs overtake large teams. It's almost like Vernor Vinge's zones of thought (http://en.wikipedia.org/wiki/A_Fire_Upon_the_Deep, http://www.youtube.com/watch?v=xcPcpF2M27c) - as your team grows bigger you can just watch the members grow dumber in front of your eyes as more and more of their cognitive effort is eaten up by internal communication, leaving less and less for externally-useful work. If this is true, there's hope that advances in programming will automate the low-cognition tasks and allow programmers to focus on the high-cognition ones, leveling the playing field for small, high-cohesion teams.
---
Me, I've been obsessed with something raganwald said when he spawned this tendril of conversation: exercising explicit control over the space of inputs my program cares about. My current hypothesis: eliminate fixed interfaces, version numbers, and notions of backwards compatibility. All these are like petridishes of sugar syrup for code to breed more code. Replace them with with unit tests. Lots of them[1]. If I rely on some code you wrote, and I want to pull in some of your recent changes, I need to rerun my tests to ensure you didn't change an interface. Programming this way is less reassuring, but I think it empowers programmers where abstraction boundaries impose mental blocks. Great programmers take charge of their entire stack, so let's do more of that. I'm hoping this is the way to prove small teams can outdo large ones.
[1] Including tests for performance, throughput, availability. This is the hard part. But I spent a lot of time building microprocessor simulators in grad school. I think it's doable.
I think you're wrong (sorry!) because it's impossible to talk about superior programmers without talking about teams. Building complex systems is a team sport. There's no way around this. But you can't have good teams without good programmers.
The phrase "scaling complexity" has at least two axes built into it: the abstraction axis -- how to get better at telling the program to the computer -- and the collaboration axis -- how to get better at telling the program to each other. Most of this thread has been about whether we suck at the former. But I say we really suck at the latter, and the reason is that we haven't fully assimilated what software is yet. Software doesn't live in the code, it lives in the minds of the people who make it. The code is just a (lossy) written representation.
We can argue about how much more productive the best individual working solo with the best tool can be- but there's no way that that model will scale arbitrarily, no matter how good the individual/tool pairing. At some point the single machine (the solo genius) hits a wall and you have to go to distributed systems (teams). One thing we know from programming is that when you shift to distributed systems, everything changes. I think that's true on the human level as well. (Just to be redundant, by "distribution" here I don't mean distributed teams, I mean knowledge of the program being distributed across multiple brains.)
Maybe you wouldn't have trouble working in teams if we'd actually figured out how to make great teams. So far, it's entirely hit and miss. But I think anyone who's had the good fortune to experience the spontaneous occurrence of a great team knows what a beautiful and powerful thing it is. Most of us who've had that experience go through the rest of our careers craving it again. Indeed, it has converted many a solo type into an ardent collaborator. Like me.
I was originally going to write about this and then decided not to go there, but you forced my hand. :) Just as long as it's clear that when I say "team" I mean nothing like how software organizations are formally built nowadays. It's not about being in an org chart. It's about being in a band.
Ah, you're right. I was conflating the two axes.
I'd like to be part of a 'band'. I've had few opportunities, but I've caught the occasional glimpse of how good things can be.
Since that whole aspect is outside my ken, I focus on expression. Hopefully there's no destructive interference. I would argue that what you call abstraction is about 'communication with each other' more than anything (even though it breaks the awesome symmetry of your paragraph above :)
Also, "less skilled programmers" are not always "incompetent programmers" so more of the former who already understands the system may be able to complete the project faster than fewer "more skilled programmers"
Have you really read the mythical man month?
I've witnessed this directly many times over the years.
Why couldn't you solve the second problem with the first? As you've already pointed out, Lisp code is smaller. Thus, you don't have to worry about the "large code base" problem like you would with java.
At the very least, we can say "the LOC-bug relationship doesn't necessarily hold across languages". It's a useful rule-of-thumb, not a universal law.
"Lisp is almost an interpreter builder rather than just an interpreter/compiler"
So most languages have a fixed grammar, and programs are always written in that grammar. Java functions, whatever problems they have, are at least always declared using a fixed grammar -- something like
function ::= returnType Identifier parameterList;
"This means that one can make it be just about anything they want. Thus, its culture is almost a lack of culture."
The fear is that you'll end up having to join a project using "Dave's Object System" or "Nora's Conditionals Library". And that's where Lisp has the Aw(esome|ful) power to write in wildly divergent styles -- in fact, written in whole new grammars.
I suppose there's an idea that code can be written in a 'natural' way for the language. So you have this idea in python programming of something being 'pythonic' -- not just corresponding to the the python grammar, but also in a particular, respected programming idiom. So the question is, when you have the power to write your own grammar, can you then still write in a good, predicatable style that makes the code a good form of communication for other programmers? Is there a 'lispish' way to write code? (I suppose the same question applies to things like internal Ruby DSLs.)
Honest questions; although I've given Lisp a good shot, I've never written anything longer than a few hundred lines...
So, you could argue that a large, complex Java program can’t be understood just by reading it a line at a time, you have to figure out how it is all wired together via the XML files.
I think what I’m trying to say is that large and complex systems are, well, large and complex whether in Lisp or java. But Lisp has standard libraries just like Java, e.g. CLOS. So it’s true Dave could write his own objects, and Nora her own conditionals, but I wonder whether that is more folklore than practical reality on commercial projects.
Yes. If you read enough Clojure, Scheme and Common Lisp code you'll see that they have their own different styles. On a few occasions I read Common Lisp code written by schemers, it was somewhat ugly and inelegant, although the same code would have been elegant if written in scheme. Here's an example from an Erik Naggum article( http://www.xach.com/naggum/articles/3195982690560792@naggum.... ). First the Scheme in common lisp:
(defun make-account (balance)
(labels ((withdraw (amount)
(if (>= balance amount)
(progn
(setf balance (- balance amount))
balance)
(format t "Insufficient funds")))
(deposit (amount)
(setf balance (+ balance amount))
balance)
(dispatch (m)
(cond
((equal m 'withdraw) withdraw)
((equal m 'deposit) deposit)
(t (format t "Unknown request -- MAKE-ACCOUNT")))))
dispatch))
(defclass account ()
((balance :initarg :deposit :initform 0 :accessor balance)))
(defmethod withdraw ((account account) amount)
(decf (balance account) amount))
(defmethod deposit ((account account) amount)
(incf (balance account) amount))
(defmethod withdraw :before ((account account) amount)
(unless (<= amount (balance account))
(error "~A has insufficient funds to withdraw ~A." account amount)))
Lisp allows you to turn code like the first example into code like the second.
edit: formatting
The main point about lisp is 'there is no accounting for taste'.
Lisp leaves most things up to the programmer's taste.
You can write Lisp that looks like Fortran, or C++/Java, or Scheme.
You can make a DSL that directly models the problem.
You can use objects or no objects, do everything in CLOS or do everything with structs.
You can write your own object system.
You can make it fast or slow, you can use correct data structures or you can do everything with lists.
You can use macros for everything or you can never touch macros ever.
You can make your program one big macro.
None of these things are 'bad taste.'
Most people have different taste from you.
Most people only end up learning the part of the language consistent with the paradigm that they like.
If you only know that piece of the language, you will have difficulty working on someone else's project when they are working in a different paradigm.
None of the individual pieces of the language are particularly difficult.
People complain macros are difficult to understand. Macros are easy. If you can understand a program that concatenates lists to make a new list, you can understand a macro. Macros are quite literally 'just lisp code'.
Are some macros written in a way that you can not personally understand? Most likely, but that is not an issue inherent to macros. You have to be careful about checking inputs, and creating the proper debugging and type checking at macro-expansion time. Just like any other program. There does seem to be a stupid tendency for people to cram an entire macro into a single function. This is foolish, the whole point is that you have the entire power of this lisp runtime. There is no reason to write it like c pre-processor garbage.
So, how does one write good lisp code? Well, one way is to pick some standards. This is as difficult has having someone in charge willing to gently say 'this doesn't really match up with the style of the code around it'. If I've inherited some code, when I do a bug fix, I'm going to do my damnedest to stick with the style that it is written in (unless it is truly tagbody/go awful, in which case I might rewrite).
This kind of turned into a rant, I apologize. I guess my point is that sure, lisp is powerful, but the real issue is the number of options that it provides. At some point, you have to pick a subset and a style, and go with it. And then you have to be comfortable learning if you inherit something that you don't know yet.
I don't entirely disagree, but I think there's more to it than that. While the language certainly does offer some free choices, more often there are advantages and disadvantages to each, so that in any particular situation, some choices are better than others. Becoming an expert Lisp programmer requires learning about these tradeoffs, which takes experience and, usually, guidance from existing experts.
That's true of any language, of course, but some of the facilities Lisp offers are rare among other languages, so that people coming to Lisp from some other language are unlikely to have experience with them.
Oh, one point about macros in particular. If you have to resort to reading the implementation of a macro to understand what it does, the person who wrote it screwed up. Macros should always have documentation strings explaning their syntax and semantics. If you find yourself in that situation, the best thing to do is to go to the REPL and use `macroexpand' interactively to see the expansions of the macro calls you're interested in.
There is a saying 'perfection is the enemy of done.' In any particular situation, there is probably not a solution that is both optimally efficient and also optimally elegant. (This is true in any programming language). If you work towards that goal too much, you will likely miss your deadline.
But the point is, the little advantages and disadvantages don't matter until they do. There isn't going to be a big difference in most programs between using a loop to iterate a sequence, and using map nil with a lambda, and using do* (for example). In fact, whether there is any difference at all in the resultant assembly or byte code will depend entirely on the compiler implementation. It is a style thing... so try to be consistent, and work within what you are comfortable with.
That's true of any language, of course, but some of the facilities Lisp offers are rare among other languages, so that people coming to Lisp from some other language are unlikely to have experience with them.
Currently, the only thing I can really think of that is actually really unique is the macro facility (and that is only because, as soon as a language adopts the macro facility, it becomes a lisp.
As a programmer, the focus of my job is learning new things. I am basically a mechanism for translating the new things that I have learned into computer code. If I can't learn a few measly language features, what good am I going to be as a 'thing I just learned to computer' translator. And like I said, use what you are familiar with, until you are faced with something someone else wrote, or you have the time to learn new things. But don't punt.
If someone doesn't have experience with a given piece of the language that has been used, I expect them to pick up a book or online resource about it (and then, possibly most importantly, play with it). It is not hard, but it does take effort. There is nothing in common lisp that requires genius level intellect. (God knows, I'm certainly not that bright). No one requires that fresh-faced C interns be pointer arithmetic gods, but i'm sure they are expected to learn it if it is part of the job.
Oh, one point about macros in particular. If you have to resort to reading the implementation of a macro to understand what it does, the person who wrote it screwed up. Macros should always have documentation strings explaning their syntax and semantics. If you find yourself in that situation, the best thing to do is to go to the REPL and use `macroexpand' interactively to see the expansions of the macro calls you're interested in.
I'll add that in addition to doc strings for syntax and semantics, there should also be assertions written into the macro about the syntax and semantics. If I am passing a number or list where the macro is expecting a symbol, an error should get thrown during the macro-expansion phase. Macros are programs like anything else. Validating inputs and throwing an error at the earliest possible time is a good rule to go by.
So, macroexpand-1 is a good start, if the macro is implemented correctly, does what its documentation says, and you are simply flubbing the syntax. (Of course, it should be yelling at you for flubbing the syntax).
However, when there is a bug in a macro, the only thing that macroexpand-1 will tell you is that the macro doesn't work. You'll macro-expand it and say 'yup that's the wrong generated code.' It doesn't really tell you anything about how to actually fix the macro unless you are already familiar with the macro's code. Having examples of inputs with bad outputs will aid in pinpointing the problem, but not unless I already understand how the program works.
Macros are lisp programs, and can be as complicated as any arbitrary lisp program. Writing a more complicated macro is not screwing up (I think this is an important distinction to make)... inadequately documenting, explaining, and bulletproofing it is. Someone might have to debug it later, so strive to write readable macro code. It isn't hard as you are just constructing lists and writing normal lisp code with minimal efficiency requirements.
So I guess that was a roundabout way of me saying "I agree, mostly."
Leaving that aside though, other people smarter and more experienced with Lisp than I have suggested that this "problem" is not unique to Lisp, and may not be a much of a real problem. I really think it's just a question of having good documentation and sensible style which explains what you need to know to use the magic even if it doesn't explain what's underneath. It's true that a lot of "lone hackers" aren't going to be writing good (or any!) documentation, but that's not a Lisp problem.
Ruby's metamagic is a common example. I already know Ruby and recently I've been learning Rails. I know there's a lot of magic going on behind the scenes, and occasionally I'll read some example or something and know that underneath there's magic that I do not entirely understand. I can continue studying the framework as a user and probably even get away using it for a while without completely understanding the magic underneath. I doubt that every person who has used Rails commercially completely understood every part of the framework that they use, since that would be a huge drag on their ability to ship their thing.
Another interesting example: This past week I talked to someone at a nearly pure Scala company and he described this DSL that they wrote (or just use? I can't remember) to interface with SQL databases. It exploited the fact that infix operators are really just methods and that you can define implicit conversions on existing library types that allow you to sort of extend them (I don't actually know Scala, so this may be slightly off base). The snippet of code he showed me, while it looked like Scala, would be turned into an SQL clause. Programmers who use this don't necessarily need to understand all of the gritty details so much as they need to understand what the designer intends and understand the semantics of SQL and of the DSL.
In Common Lisp, "lambda" is a macro. I don't really know how it works, but I know that it expands into lower level code defining a function and a closure. It's the same way with standard macros (in some Lisps) like let, letrec, cond, if, and, or etc... Most Lisp programmers who are not experts can use them understanding that they are macros but maybe not knowing their complete implementation because they are very well defined (and because they are pretty simple).
More complicated macros like "with-open-file" can be used by programmers who probably have some idea of how the macro works, but not a complete understanding. As long as your own macros and Lispy magic are documented sufficiently and designed sensibly programmers should be able to use code in your "world" without understanding it completely. At least to start.
N.B. I haven't used Lisp in a commercial setting, so it's entirely possible all this is bunk and I'm just being naive. Wouldn't be the first time.
edit Upon re-reading this monster comment I realize it might come across that I'm advocating some kind of "cargo cult" or copy-and-paste style of ignorant programming. I'm really not. My point is that while understanding your tools is important, so is knowing when you don't need to peel back the curtain and instead need to address the problem at hand.
Because it is being used by a generation of programmers who cut their teeth on Rails and were frustrated by J2EE verbosity, popular Clojure code tends to be written such that APIs are quite readable and bereft of too much cleverness (the cleverness is usually hidden in the implementation of the API, rather than the interface).
Examples: Compujre, Ring, ClojureQL, Encanter, etc.
S-Expressions don't befuddle people. People befuddle people.
We're not even close to understanding how to reliably make good programs, so naturally we don't understand how to do that with Lisp either. Lisp is just a particularly pure medium for programming. Most of the things people are saying about Lisp ("it allows people to invent their own little worlds") are really just statements about programming.
Oh and one other thing. A pox on the terms "readable" and "unreadable" floating around as free variables. They are hopelessly relative and their prominence in any discussion of programming (or should I say every discussion of programming) renders said discussion pointless. We literally don't know what we're talking about.
As pnathan brilliantly put it in another thread (http://news.ycombinator.com/item?id=3387432): "Intuitive is just what you've seen before." Bravo, pnathan.
Actually, I can perfectly reasonably imagine a physicist complaining that a theory is too powerful. If a theory has the capability to explain any possible observation, it similarly does not have the capability of being falsified...
I can see refusing an approach on the grounds that it doesn't work, but to refuse it on the grounds that it works too well?
For what it's worth, one can easily call a theory "too powerful" for practical purposes. E.g., one often uses Newtonian mechanics to analyze something rather than Einstein's more powerful approach. Human brainpower is a sadly limited resource, so "easy" often trumps "powerful".
"Human brainpower is limited, so we should use weaker languages" seems a strange argument. The whole point of using more powerful languages is that they can express more with less, allowing us to fit more into limited human RAM. It's because our brains can't manage millions of details that we don't build systems directly out of machine instructions in the first place and resort to cleverer strategies like higher-level languages. At what point along the way does that process stop being valid?
Deeper abstractions are only useful in coding to the extent that you don't have to understand the details. But abstractions leak:
http://www.joelonsoftware.com/articles/LeakyAbstractions.htm...
As Martin Fowler writes, "Any fool can write code that a computer can understand. Good programmers write code that humans can understand." You might be able to turn 100 lines of Java into 1 line of Lisp, but if it takes your colleague more time to understand and modify the 1 line of Lisp than the 100 lines of Java, then you've made things worse.
That implies that humans must mentally compile the code. By definition that's not true of a higher-level language.
Ditto for your invocation of leaky abstraction: by definition, a program written in a language can be understood by learning the language and reading the program. If that's not true, then the program must not really be (fully) written in that language.
The whole reason why we prefer a program in, say, C to a 100x larger program written in assembly language is that it's easier to understand. Given that available evidence points to source code size as the best measure of complexity and best predictor of errors -- there was a recent thread on HN about this with links to several papers -- I think the burden is on you to explain how the benefit of writing smaller programs in more powerful languages holds good up to, say, Java, but breaks down when you move to, say, Lisp or K.
The higher the language, the more abstract the abstractions. Each symbol represents more, and therefore hides more. There are manipulations on the symbols that are plausible in context but idiotic when one looks at the hidden details. I see that regularly with developers who have only worked in higher-level languages. From their perspective, they will see two approaches to the problem that will be equivalent. However, there will be 3 orders of magnitude difference in performance. By definition (well, your definition) that doesn't matter in a higher level language. In practice, it does.
I don't think I've said that Java is some sort of magic universal middle point in languages (and indeed, I don't think that exists), so I don't accept the burden you're pushing on me. You asked how a language could be too powerful, and I tried to answer your question. If you have more questions, feel free to ask.
So this is about performance? But that is a separate issue. The claims in the OP are that a language can be "too powerful" because programs written in it, although shorter, are too hard to understand. I'm looking for evidence of this that doesn't simply reduce to "intuitive is just what you've seen before". The interesting thing is that there doesn't seem to be any.
(Performance is a separate issue because it has to do with the gap between abstractions and the machines they have to execute on. I think this may be the Achilles heel of the highly mathematical FP approaches. Still, it's a change of subject to interpret "too powerful" as "too powerful for the machine" rather than "too powerful for the brain".)
The underlying problem seems to be that abstraction is hard. That is, for any complex system, it's hard to come up with a set of concepts that can express the system with less overall complexity. This is true of programming in all languages. The problem is masked in lower-level languages because the abstractive weakness of the tool provides an excuse to produce the usual reams of code and thus the usual runaway complexity. But that doesn't mean the task is easy in a more powerful language. The hardest part -- coming up with a good set of concepts in the first place -- remains. Faced with that, people do sometimes rush to apply the technical devices offered by, say, a Lisp to a not-clear-enough set of concepts. The resulting programs can indeed be hard to understand. I've done it myself.
In that sense, some abstractions are indeed too "powerful" for the problem and the people at hand. When training somebody to work the register at a corner store, one does not start by deriving number theory from Peano's axioms and then telling them to work it out from there. You make sure they understand basic arithmetic through simple practice and then you train them on the specific operations for money, mainly with further practice. For the novice cashier, "power" in the sense you use comes at too high a price with too low a payoff to be worth it.
By the way, what came across as pushy was you insisting I had a burden to justify something I had never said (and incidentally don't believe).
Brooks said that a "programming product" (meaning one that can be used by other people) takes x3 the work of a "program" that works. He talks about documentation, testing, generalization and "can be run, tested, repaired by anyone". I think this means careful API design for usability, discoverability and understandability - not just efficiency - is important.
So, inventing new worlds (DSLs) is not a problem; inventing your own little worlds that are hard to understand and use is a problem. But it takes x3 as much work to do it right, and it usually isn't worth it unless it is explicitly intended to be used by others (e.g. it's a library for sale; or a utility for use within a large organization; or a web API).
Secondly, an example from the history of relational databases. Codd had the idea of relations plus a high level language. He designed a couple of high level languages, but no one liked them. Instead, Boyce and Camberlin came up with SQL (originally "SEQEL"), that was usable by mere mortals. "Since Codd was originally a mathematician (and previously worked on cellular automata), his DML proposals were rigorous and formal, but not necessarily easy for mere mortals to understand." http://webcache.googleusercontent.com/search?oe=utf-8&rl... Sometimes, designing a DSL for others is so hard that it takes a different person to do it.
This isn't specific to Lisp. It's an issue for designing DSLs and APIs (and languages in general), which can be done in any language. Lispers may invent more often and with more variation, because lisp is more powerful. Power --> Responsibility
In Java, the "DSLs" you tend to get are at the class level, in different files. Like any abstraction, these can be well- or ill-designed. There's some differences to lisp: (1). the abstractions are less flexible/powerful, so there's less to understand; (2). having them in different files makes it harder to grasp the whole than if all in one file (or one screen); (3). the syntax is fixed, so you can at least understand the symbols without understanding anything else.
I think that inventing a new language has the best chance of making something that is a genuinely better solution. But if you want it to be understandable, it's helpful to link it to existing concepts that are already known, understood, and with known modes of use and application, perhaps via metaphor - i.e. adoption through familiarity.
But the general case for adoption seems to be that something must be x10 better (or compelling in some way) for people to go through the pain of adoption, of learning new syntax, new concepts, new ways of working, new infrastructure, new tradeoffs, new gotchas, new shortcuts, new consequences, new policies, new standards, new training, new suppliers, and so on.
If you can reduce the pain of adoption, adoption is more likely.
Put another way: there are two kinds of pain: the pain your solution addresses, and the pain your solution creates. The reason to adopt your solution is to reduce pain, but if the solution itself brings too much pain, it's simply not worth it.
Relational databases are an example of this. The relational concept solved the pain of storage change, but also created pain (difficult to use; x10-x100 slower). As those secondary pains were solved (with SQL; with optimization stategies and Moore's Law), its adoption accelerated.
So, sometimes a fundamental improvement needs to place power and flexibility over ease-of-use - but to be adopted, sufficient ease-of-use is essential. And your abstraction has to be, not just good or better, but x10 better.
Does anyone have examples of crazy unmaintainable Lisp code we could look at?
On the other side, what examples of powerful / elegant Lisp code do you feel best make the case for Lisp?
I fully realize this is subjective ("no accounting for taste"), and that a handful of anecdotes doesn't really settle anything. Nevertheless, I'm interested in what the failure modes might be for Lisp, and whether they have analogs in the languages I'm more familiar with.
I'm also bothered by a tendency in the Lisp community to say, "Lisp is the best, and I have all this awesome Lisp code, but no, I'm not going to show it to you." If someone asked me for great C, C++, Python, or Perl code, I have favorite examples I'd point them to without hesitation. What gives Lispers? Is your Lisp code so personalized or specific to the problem that you fear it really wouldn't make any sense to an outsider? If so, how come this doesn't translate into maintainability problems?
Here's some impressively awesome Lisp code, that leverages features that might be considered "too powerful". It's some Common Lisp from Peter Norvig's Paradigms of AI Programming: http://norvig.com/paip/auxfns.lisp
Checkout any of the macros in that file (search for "defmacro"). If you need a primer on Common Lisp macros I think Peter Seibel's intro is sufficient: http://gigamonkeys.com/book/macros-defining-your-own.html
The memoization stuff is pretty cool and so is the stuff for managing resources (defresource and with-resource).
That's one of the best books on programming I've ever read (I mean started and set aside then picked up again and again for many months and haven't finished), and there's a lot of really nice and elegant code in it. The way he describes searching graphs and trees is really nice but that code doesn't use any of the really Lispy stuff. It would look pretty similar in Python or Ruby. The rest of the code is here: http://norvig.com/paip/README.html
I can still show you examples of beautiful and horrible 20 line lisp programs, but I'd rather show you examples of large scale design. The most popular might be Emacs. Emacs has a million lines of elisp, imagine if it was written in java or C++, scary thought :)
In a large system like emacs you'll find many examples of beautiful and ugly code, but the overall system is still beautiful and maintainable. This talk by Stuart Halloway might explain what i mean by that: http://vimeo.com/1013263
In a nutshell, emacs is big, but small for its size, meaning that those 1000000 lines of lisp do much more than a million lines of java will ever be able to do. That property comes in part by using a lisp as an implementation language(and not a very good lisp at that :)
Yep, came here to post that too.
I don't think you can have it both ways. (I'm not saying that any one person is making both of these points, but person B's anti-Lisp argument is arguing against person A's anti-Lisp point.) If you can't use Lisp in a team because somebody might write a macro that you don't understand, how can you deal with Python or Ruby or C# code that inevitably tries to fake it by taking as parameters functions-that-returns-other-functions?
(The other common alternative I see these days is to make your DSLs in XML. That means they're more verbose, you can't step through them in your debugger, and so on -- plus you still have the problem that allowing arbitrary DSLs is too powerful! I suppose there's also a third alternative: don't even try to use higher-level abstraction, and build giant apps out of low-level spaghetti code.)
I've seen perfectly readable code in Lisp, even making extensive use of macros. (There are conventions, and good programmers do follow them.) I've also seen perfectly undecipherable code in every language I've ever seen. When I've watched individuals write code in multiple languages, those that write bad Lisp tend to write bad anything. Nothing that I've seen leads me to believe that there is any significant set of programmers who can only write bad Lisp code, but can write great code in lower-level languages. We can give them Lisp, though, and they will at least write less of it!
There's a lot of horrible Java code on the internet, mostly because there are tons of people writing Java. Nevertheless, in good hands, Java can be quite elegant and succinct.
Could you elaborate, perhaps with an example? Not trying to start a flamewar or anything - but I honestly can't imagine a case where Java is succinct relative to other languages. And as far as elegance goes, Java is so overly verbose and fixated on classes ('too many classes? there's a class for that!') that I have a hard time thinking of it as elegant.
But then again, maybe I'm just used to reading bad Java code everywhere, and you've been lucky enough to find the good stuff!
runOnUiThread(new Runnable() {
@Optional
public void run() {
// ...
}
});
And once you start writing something significant or that needs to work cross platform say hello to design patterns to work around the straight jacket.
I'll take Lisp, JavaScript, Ruby, or Python any day of the week. All languages have warts but to me it seems that Java has warts by design.
public void saveAll(Object[] objects) {
File file = new File(filename + ".xml");
try {
BufferedOutputStream bos = new BufferedOutputStream(new FileOutputStream(file));
XMLEncoder xenc = new XMLEncoder(bos);
for (Object o : objects) {
xenc.writeObject(o);
}
xenc.close();
} catch (FileNotFoundException e) {
System.err.println("File not found");
} catch (IOException e) {
System.err.println("Some other error");
}
}An example from my library[1]:
SqlCommandEx sql = "select * from person";
foreach (dynamic p in sql)
{
var full_name = p.first_name + " " + p.last_name;
var age = ((DateTime)p.dob).YearsAgo();
//etc
}
//automatically disposes the connection
I once took one of Peter Norvig's Python programs (a spelling corrector) and converted it into VB.NET, just to see whether VB was as succinct as Python. Turns out it is:
http://www.stevecooper.org/index.php/2010/11/10/vb-net-and-p...
Or finding out that C# anonymous functions are more concise than Haskell. Thing is, sometimes these things surprise you. Languages do get reputations that are hard to shake.
Also browse some of the core Java source code to see really elegant implementations by him.
For the curious reader, here's how to do Java entirely from the command line:
1) install the JDK and add its "bin" directory to your PATH. For me this was "jdk1.7.0_02/bin"
2) copy and paste http://pastebin.com/R47bwDZx and save it into a new file called "test.xml"
3) copy and paste http://pastebin.com/Ut5dKBkn and save it into a new file called "ReadXMLFile.java"
- note, this is not my code. It's just a snippet I found from the 'net. Feel free to clean it up. =) My goal was "to get up and running as quickly as possible". Production code would be cleaner than this.
4) run these commands:
javac ReadXMLFile.java
java ReadXMLFile test.xml
So why go through all this trouble, when you could just write it in Python?
Deployment comes to mind as one reason to go with this approach; fewer systems have Python installed than Java runtime.
So therefore when you deploy your app, you'd have to either 1) make customers install Python, or 2) (more likely) ship the contents of the "Python27" folder along with your Python app, and invoke your app via a shell script, or something. I'm not sure.
I would like to see proof of this. Perhaps Windows has fewer Python installs, but I would find it hard to believe that there are more Linux systems out there with Java (which most distros don't install by default) than python (which more systems do install by default).
(It's been a while since I used either of those tools, so there may very well be other alternatives around now.)
Though working on a big project or just simply wanting to get up and running fast a good text editor or an IDE is so nice to have. I hate having write my own getters and setters, or just simple the psvm method.
Sure, there are languages that don't let you get at the meta, but just because the ones that let you do can be abused does not invalidate the usefulness of the notion.
However, it's at least partly taking advantage of the fact that normal Haskell actually looks (in shape, anyhow) vaguely like basic as is. At the very least, Haskell doesn't force you to use parenthesis or braces everywhere.
Here's an example: http://augustss.blogspot.com/2009/02/is-haskell-fast-lets-do...
Yep, it's called Baysick - https://github.com/fogus/baysick
I think calling this a straw man would be generous. Hackers use lisp because they feel it makes code more easily readable. You don't have to agree with them, but at least take time to understand their arguments before you refute them.
"This is why Lisp is a HackerLanguage instead of a commercial language: hackers are generally loners who don't care if others can figure out their code (at least while they are in the mode or role of hacking)."
I don't know many hackers who this applies to. Rather, the DRY ethos seems to extend into readability and accessibility - most acknowledge they'll one day pass the code to someone else to maintain and too much complexity makes that near impossible.
Lisp requires a new way of thinking - in recursion, lambdas, mapcars etc. - to write good code which reflects the awesome abilities of Lisp. Unfortunately many people don't grasp it. They don't want to think or to learn superior ways if they just can use a language which makes them able to solve their problem. The way to the solution doesn't matter a lot if the solution itself works.
Btw the same phenomenon happened with Ada. The Ada 95 language is awesome. I admired it, it was real fun to use it. But average programmers are simply overwhelmed. That's the reason why Ada died.
Many people also complain about Unix and Linux but if you take the effort and learn it seriously you will love it.
http://www.apl.jhu.edu/~hall/Lisp-Notes/Macros.html
And here an insightful statement of someone who was sceptical but became a Lisp convert:
http://www.defmacro.org/ramblings/lisp.html
Lisp is so different from all other languages that you have to use it to understand it. Just reading about it is not enough.
Note that there are many variants of Lisp. Older Lisp versions are suitable for experts only. For beginners I would recommend Scheme which is a well defined successor of Lisp. I would recommend Racket (http://racket-lang.org/) as SDK. It is suitable for beginners as well as for professional development.
As I pointed out in a comment at the beginning of the thread, scheme teaches you some habits that don't translate well to other lisps, so to those thinking of picking it up, be mindful of the assumptions of the language, and when you decide to look into other lisps, don't assume they will be the same there as well.
In fact i would actually recommend learning clojure or common lisp before scheme. I consider both of them to be better languages, but i have my own set of assumptions that might not be shared by others :).
I do note the PG quote at the bottom of the page -- "the power of Lisp cannot be traced to any single one of them. It is the combination which makes Lisp programming what it is" -- and I can see that unity being very elegant.
But the real standout item is macros. I suppose it's the only unstealable feature. Or rather, those attempts to steal it, like .net DLR Expression Trees, make really ugly code and are never going to be standard programming techniques.
Anyway -- thought-provoking. Thanks.
[1] http://www.amazon.com/Designing-Programming-Personal-Expert-... [2] http://pastebin.com/q9hMS1rE (warning: WIP)
The standard answer to If-Lisp-Is-So-Great used to be "It isn't."