Beating the Averages (2001)
paulgraham.com
paulgraham.com
Oh, right. That was Facebook. They ALSO ran circles around their competitors and crushed them. I hear they are worth /even more/ than Viaweb!
Great PHP story:
> Dustin was like, 'Hey, I want to help out"
> Zuckerberg told Moskovitz, "That's pretty cool. But you don't know any PHP."
> "That weekend he went home, bought the book 'PERL for Dummies,' came back, and was like, 'OK, I'm ready to go,'" Zuck said, and told him, "The site's written in PHP, not PERL, dude."
"The Blub Paradox" is mistaken. It's a misunderstanding.
To see why, think about Haskell. Haskell users are sure they're at the top of the power curve. When they look at Lisp, they're sure that they're looking down. "How can you get anything done in Lisp? It doesn't even have a decent type system."
But Lisp users, when they look at Haskell, they also are sure that they're looking down. "How can you get anything done in Haskell? It doesn't even have macros."
This situation - two languages, each certain that they're looking down at the other - tells us that there's something wrong. And what's wrong is the idea that languages can be ranked on a one-dimensional axis labeled "power".
So, what is "power"? Or perhaps better put, power to do what? Well, to write programs. What programs? General programs? I've never written a general program in my life. I've written a bunch of specific ones, though.
When I'm choosing a language, I care about its power to write a specific program. (Because why do I care about how good a language is for writing a program that I'm not trying to write? Why would I care about that?)
So, ask what technical issue makes it hard to write the program you're trying to write. Then look at what language goes the furthest in the direction of making that problem easier. (You could go further, to think of the set of technical issues as a vector in a multi-dimensional space, and languages as a tree in that multi-dimensional space. Pick the language that reaches furthest in the direction of the vector of the technical problems.)
But you have to have a somewhat broad idea of "language". It includes libraries and ecosystem. It includes the ability to write code that can be maintained over multiple decades (if that's part of what you need).
TL;DR: The Blub Paradox is mistaken. For it to work, you have to be able to rank languages in one dimension (power). You can't. It's much messier than that.
If you’re planning to pick a language de novo for each program, you don’t care about this. That’s probably not the common case that a dev intends/is willing to learn a new language for each new program.
And then I have to think about, well, X sounds like a useful tool. Is it useful enough that I want to make the goal be "learn X, and write this program" instead of just "write this program"?
I went back and read it again, and unsurprisingly pg addressed this point too:
> But I don't expect to convince anyone (over 25) to go out and learn Lisp. The purpose of this article is not to change anyone's mind, but to reassure people already interested in using Lisp-- people who know that Lisp is a powerful language, but worry because it isn't widely used. In a competitive situation, that's an advantage. Lisp's power is multiplied by the fact that your competitors don't get it.
Every time you feel like discounting Lisp, just remember “The world wouldn’t have Hacker News.”
There is still an astonishing dearth of languages that take both dynamicism and performance seriously[1], but you're even starting to see bits of metaprogramming become more mainstream.
Tooling is still mostly terrible outside of the lisp and smalltalk worlds ; I have mixed feelings about LSP, because while it greatly improved things, it appears to have done so by establishing a local maximum that few development environments will escape from. Jupyter is really cool, but it's the exception that proofs the rule.
So compared to the '90s, your backend language will probably support garbage collection, anonymous functions, higher order functions, and maybe symbolic computation and/or a rich type system. It's harder to beat the averages when the average has gotten closer to where you are.
1: While performance is a property of implementations (not languages), performance is not just about implementation. The Lisp standard has several places where limits are placed on allowed dynamic modifications specifically for performance.
1. Dynamic Dispatch
2. Multiple Dispatch
3. Auxiliary Methods
4. Namespace independent of specialization parameter(s)
I find it interesting that you focus on #4. People coming to Lisp complain about #4 a lot, and people leaving lisp complain about the lack of #4. If you aren't going to use multiple dispatch (which, empirically is rare in CL code), then #4 seems to be more a matter of what one is used to. Heck, when I program in a Lisp-1, I struggle with the semantic overhead of avoiding accidentally binding a variable that happens to be the name of a function.
Also, the fact that many Lispers end up writing GFs named things like widget-frobnicate and sprocket-frobnicate might indicate that coupling the namespace with the primary operand might not be completely unwanted.
Also, Python is almost certainly the worst offender with consistency here -- e.g. "len(x)" vs "x.len()".
P.S. C++, Javascript, and Python have libraries available implementing multiple dispatch.
If they do that, they're doing it wrong. Probably because they are bringing in ideas from the non-Lisp world, not realizing that these techniques are not actually good programming practice, they are strategies for dealing with deficiencies in mainstream languages.
IMHO languages should be made to serve programmers. Programmers should not have to work to serve the language. So if I want to frobnicate something I want to be able to write (frobnicate the-thing) and have it Just Work. Common Lisp gives me that (when it's is used properly). No other language does.
> when I program in a Lisp-1, I struggle with the semantic overhead of avoiding accidentally binding a variable that happens to be the name of a function
Yes, that is why Lisp-2 is the Right Answer.
There is a certain sleight of hand that every lisp advocate pulls when this conversation comes : claiming that every single inspiration or language feature that is remotely related to lisp is "re-invention", with the additional implication that it's an inferior imitation.
APL, Algol60, ML, Smalltalk, Simula, Prolog, Perl, Ada, Erlang and Eiffel are all extremly influential languages that all modern languages (including Lisp, ironically) are imitating and re-inventing all the time. But it's always the lispers who are obsessed with who took ideas from them, you never see smalltalkers raving about how the rest of the world took JITs, MVC and most OOP from them, or MLers about how the rest of the world learned proper static typing and module systems from them. Only lisp is that special snowflake that the PL community, countless researchers and language implementers, have been stuck helplessly imitating since 1960 and failing.
"The thing about re-inventing the wheel is : you can actually do it better the second time"
- Paraphrased, allegedly from one of JSON creators in response to somebody telling them they are re-inventing XML.
-------
Among the endless "exclusive" features claimed for lisp are :
- Homoiconity: Which Mathematica does better, with its extremly expressive and diverse syntax that nonetheless all boils down to Lists in the end. Unlike Lisp which tries to force its inferior syntax on me as a "feature", Mathematica understands that concrete syntax is not abstract syntax, the former can be (indeed, needs to be in a high level language) rich and expressive, the latter can be spartan and pure. Also Elixir.
- Macros: Which countless languages does with a myriad of forms and functions. Some are : Elixir, Rust, Julia, Scala, and those are just the ones that call them "macros", AST manipulation and compile-time code reification are very general concepts that are called a lot of things in different languages. ("Templates" in Haskell, "Annotations and Annotation Processors" in Java and Groovy.)
- Dynamicity: Come on, Ruby can redefine the base language from underneath you.
- Image-Based Development: Fair enough, but I will politely point to Smalltalk and Erlang, and beg lispers to read some PL history before claiming that their programming languages is the O N L Y one to do so and so.
- The Condition System: Also fair, and this is the only one that is truly without analog in any other programming languages I know, but it's also only in Common Lisp, not scheme or Clojure or Arc or Racket or any other lisp dialect, and it doesn't meaningfully (as far as I know) depend on "Lisp" the language, what prevents anyone from implementing it in any language they like ? What's so "Lispy" about it exactly?
Lispers need to wake up to the fact that Lisp is not special in 2020. They needed to wake up to that fact since 2000, but better late than never I guess.
>the cognitive burden of figuring out whether to write x.f(z) or f(x,z) every single time they want to invoke f
huh ?
Of course. The reason I like Common Lisp is not because it has been influential. I like it because by my personal quality metric it got more things right than any other language currently in use.
> you can actually do it better the second time
Theoretically. But in practice all modern languages suck compared to CL with respect to my personal quality metric, which is that I want a language that imposes the minimal cognitive burden on me as a programmer. To me, making programming as easy as possible is the whole point of high level languages. If I were willing to work hard, I'd just write in assembly language.
There have been a few improvement on CL: T. Oaklisp. Dylan. But they have all sunk into obscurity.
> every single inspiration or language feature that is remotely related to lisp is "re-invention", with the additional implication that it's an inferior imitation.
Yeah, well, if the shoe fits. The only real innovation in programming languages since Common Lisp is static type inference.
> Ruby can redefine the base language from underneath you.
Yeah, that is not necessarily a good thing. Also, Ruby can't change its surface syntax. Common Lisp can.
But that's not what I was saying, I wasn't saying that you should like those languages, I was saying that Common Lisp is not special compared to them and nobody is specifically "re-inventing" it, it's a pretty vanilla (by 2000s standards) language descended from a 1960 grandpa.
>my personal quality metric
How would that convince anyone who is not you ?
>To me, making programming as easy as possible is the whole point of high level languages
Lisp syntax fails spectacularly on this one, it's painful and tedious, it forces you think about the explicit nesting between structures, like how assembly forces you to explicitly think about data flow between instructions. In any sane programming language, you just write a+b, in lisp, it's (+ a b). A bizarre 1980s-calculator world where you have to think carefully about every operation and read it outloud. It can be gotten used to with experience off course, but it's not effortless, it's far from the language making it easy for you.
The entire point of conrete syntax and grammars is that the compiler builds the syntax tree for you, you just lay out a nice and readable specification of it in the mathematics\english mixed notation that is most programming languages, and the parser infers all the dependencies in the implicit tree structure and builds it for you. Lisp inverts this relationship, making you into a human parser that translates your own ideas into its simplistic notation. S-Expression is a 1960s hack to simplifiy parsing.
>The only real innovation in programming languages since Common Lisp is static type inference.
This dismissal shows an astonishing ignorance of PL design and history. Haskell lets you design your own control structures with lazy evaluation. Perl6\Raku has first class support for BNF grammars built right into the language. Both languages doesn't treat operators as special and let you define arbitary unicode symbols as operators and control their precedence and associativity, enabling unparalled DSL development. OCaml has a module system that represents code as first class structs and runs arbitary functions to transform them at compile time.
Ironically for the community that invented the "Blub Paradox", CL programmers have an unfortunate syndrom of the Blub Programmer. "CL is all I know, therefore CL is all I need and the most powerful language in existence".
>Yeah, that is not necessarily a good thing
I wasn't saying it's necessarily a good thing, I was saying it (and Python, and Groovy) beats Lisps in dynamicity, which was once (and sometimes still is) a common selling point that lisp advocates repeat.
>Also, Ruby can't change its surface syntax. Common Lisp can.
Plenty of others can, some with systems that are not inspired from lisp. I mentioned some of them in a point that you skipped right over.
S-Expressions were invented as a way to write (externalize), read (internalize) and represent list like data structures. The original Lisp design had two different syntaxes for code (m-expressions) and data (s-expressions).
It turned out that programs can be represented on top of s-expressions and that simple execution engines and compilers can be written for it.
Further it turned out that one can write user interfaces which in simple ways can read, write, execute s-expressions. Thus s-expressions seem particular useful for interactive programming.
> you just write a+b, in lisp, it's (+ a b)
Lisp does not care much about that, since much of it was to implement tools on top of itself. Thus some of the first systems who could manipulate mathematical formulas were written in Lisp. Much of Lisp is concerned with all kinds of symbolic notations - some use s-expressions, some are not. For example, the first ML was implemented in Lisp and did not use s-expressions. Scheme OTOH also was first implemented in Lisp and did use s-expressions.
Lisp macros and special forms implement syntax on top of s-expressions - what makes it convenient, is that the syntactic extensions are transformation programs written and executed with the same language they extend. That makes it a very powerful and relatively easy to use syntactic extension mechanism - which is widely used in Lisp.
> but it's not effortless, it's far from the language making it easy for you
Some tools need some learning. Once learned, the Lisp user has access to a language which provides a simple and well-integrated syntactic meta-level level.
To the contrary, Lisp syntax is much simpler than traditional syntax. It does not draw an arbitrary distinction between operators and functions. Operators are a notational hack. We write sin(x), sqrt(x), log(x), so why not sum(x,y), expt(x,y), xor(x,y)?
> it forces you think about the explicit nesting between structures
Traditional syntax forces you to think about operator precedence. It's also much harder to parse.
> Haskell lets you design your own control structures with lazy evaluation.
Lisp lets you do the same thing with macros and closures.
> Perl6\Raku has first class support for BNF grammars built right into the language.
Why would you want that? What benefit is there over having parsers available as library code?
> beats Lisps in dynamicity
> Plenty of others can
I think you're missing the point. The reason Lisp is cool is not that it has the most powerful version of any given feature (except for CLOS -- that is indeed still unmatched). The reason Lisp is cool is that it gives you the right combination of all of these things packaged in a way that is, at its core, very simple and easy to learn. Nothing else comes close in terms of power-to-learning-curve ratio.
Because in Raku, grammars are really classes. So you can actually use roles and inheritance to build your grammar. And at the lower level in Raku, rules, tokens and regexes are just executable code (method on the grammar), so you can wrap them if need be. And they have all of the optimization opportunities (such as inlining) as all of the other executable code.
> Mathematica understands that concrete syntax is not abstract syntax, the former can be (indeed, needs to be in a high level language) rich and expressive, the latter can be spartan and pure.
While there are plenty of novice Lispers who don't understand this about syntax, the balance of the lisp community does, and most of them are still perfectly happy with the (relatively) tight coupling between the concrete and abstract syntax. People who are unhappy with it tend to either write reader macros or invent Julia [edit] I neglected to mention Racket language modules here as well.
> - Dynamicity:
> - Image-Based Development:
Dynamicity and image based development go somewhat hand-in-hand. Being able to e.g. safely redefine a class at runtime is certainly less important with out it. There is also a certain je ne sais quoi of the combination of dynamicity, native compiler, and tooling. I can redefine a function and collect assembly-level profiling of it without restarting the program. As far as I know there's nothing preventing someone from implementing Ruby in such a way as to allow this, but here we are.
Smalltalk and Lisp developed in parallel, with lots of cross-pollination, so there is a lot of overlap there; in turn Erlang was heavily inspired by Smalltalk.
> - The Condition System: ...it's also only in Common Lisp,
- Dylan
- Scheme: prior to R6RS this was left to the implementation; the R6RS exception specification is certainly similar to CL's in feel (but it uses explicit continuations, which is more scheme like)
> Lispers need to wake up to the fact that Lisp is not special in 2020. They needed to wake up to that fact since 2000, but better late than never I guess.
I think Lisp is at least special still in the sense that its particular combination of features are in some ways unique. I also think that in 2000 (and even more so in 1995), it conformed to some much stronger definition of special in terms of having some features that were either rare, or not present at all, in other non-experimental languages.
What gets me is just the smug attitude of "Hehe, look at all those uncultured masses developing in a language other than our lord and savior. <shouts loudly> We were there since 1960, swines <shakes fists>."
When I say that lisp is "not special", I don't mean it's bland or uninteresting in the sense of Java-The-Language (not the VM), that would be a contradiction since I already acknowledged the uniqueness of the condition system and the Image-Based tools, and the interestingness (but not uniqueness) of macros and CLOS. I just mean it in the "not specially special" sense, i.e. it's not any more special than Erlang, Smalltalk, Haskell, OCaml, Perl, or any of the dozen or so languages that led the way in the development of new abstractions and ways of expression.
It's a bit like saying "Greece isn't special" or "India isn't special" to a modern nationalist of those countries who think that their civilizations are unmatched. Yes, Greece and India are very special cultures who invented fantastic things and led the way in many ways, but so is Egypt, Iraq, China, Western Europe, Russia, US, etc...
While you're perfectly fine being proud of things you love ("Smalltalk invented OOP!", "Erlang invented concurrency!"), denigrating other things and implying that they're posers who can't possibly match the thing you like despite their "reinvention" is off course something that will get people riled up and understandably aggressive.
That is not correct. Joe Armstrong in his paper The Development of Erlang says this;
The design of Erlang was heavily influenced by ideas from the logic and functional programming communities. Other sources of inspiration came from languages such as Chill and Ada which are used in industry for programming control systems.
We concluded that we would like something like Prolog with added facilities for concurrency and improved error handling. No such language existed at the time.
By about 1988 is was clear that Erlang as it was now called was a good language for prototyping telephone exchanges [10]. It was a strange mixture, with declarative features (inherited from Prolog), multi-tasking and concurrency (inherited from EriPascal and Ada) and an original combination of error handling mechanisms.
Great insight because Lisp is the only general purpose programming language.
https://web.archive.org/web/20051207043012/http://reddit.com...
This seems a rather extreme claim. Can you back that up?
It would have been possible with CGI scripts. Or PHP. Or Perl. Or a myriad other things.
It could be that Lisp because forces one to think more than just hacking out code, the end result is better designed.
HN is, primarily, the community it developed, not the code running on the server. Crediting Lisp seems like you're reaching.
The next time I visited web apps, Jetty was around and seemed way smoother in comparison.
Microsoft's IIS used to have an outrageous footprint (if memory serves, you needed a whole computer to host a domain, or something like that). So Apache took off. Then virtual machines. But even they were too bloated, so Docker took off. And the BSD people
Likewise with Spring. It was huge and super-powered! By the time I got the hang of it, it had already become Spring Boot, shedding a bunch of the Enterprise baggage (especially xml) and trying to become leaner and meaner.
I was completely blown away when I saw https://sparkjava.com/ and once again when I saw https://learnbchs.org/easy.html
Now when I code web apps (or "microservices" these days), my mindset is that http is just a function from request to response, and I feel so much better for it!
> we are so far beyond the discussion if its a good idea to write your own tools for doing http request
In short, I've become skeptical of the value-adds of gigantic software artifacts, as well as existing ways of doing things. I can't wait for more shitty little side-projects to displace bloated incumbents.
> And large scale adoption of the modern open source ecosystem was only possible after 2013 .
What happened in 2013? It looks like CPAN's been doing its thing since 1995.
Compared to what I could be using, my ideal stack does not have: class loaders, virtual machines, dependency injection containers, web application servers, Log4J, etc.
Interesting how this advice really doesn't apply anymore. If you want to build fast, it's less about the language itself, and more about the community and packages. Trying to write a web app using Lisp will most likely set you far behind any startup that just uses NextJS.
Is but one single data point, but it cost me quite some months to learn (I tried to talk them into simpler solutions for a while and then gave up and bailed).