Hacker News Reader

Comment by maliker | Hacker News Reader

I've read this story a couple times as it's popped up on the internet in different places. I've enjoyed it each time because it promises that lisp is an uber-powerful and easy (?) language.

I'm always curious though: what's the other side of the story? Why is lisp always rejected?

I started programming seriously (for money) in 2009, and when we needed a language we went with python because it had better math/science libraries than Java and much more libraries in general than clojure/lisp/ocaml. I confess I like the functional languages and like using some of their features in python.

Was the no-libraries dealbreaker for the "pure" functional languages also true in 1999, which led google to java? Or maybe there are complex meta-programming bugs you can generate in the functional languages (and elsewhere) that are the dealbreaker? Or what?

View on HN

Languages like Lisp, APL, and Haskell reward extraordinary skill with extraordinary power. If you're a researcher or an early-stage startup founder, that's great. You want your little band of geniuses to have the maximum possible leverage.

But if you're an engineering leader at a large company, the last thing you want is an asset in your portfolio that only extraordinarily skilled people can work on. What if they quit? What if your company's stature declines and you can't attract such people anymore? What if a more pressing problem comes along and you want to deploy them elsewhere?

So we get languages with short learning curves and low ceilings (Go!) instead.

Except in performance critical contexts, where it can’t be helped, but C++ already won that one.

This seems correct to me. The last two things I've learned were javscript/node/react and Haskell. I had never touched js or html before, but within two days I had a pretty decent web app up and running. It was something that could (and did) provided value for a business.

Two days after starting Haskell I had essentially nothing to show for it except a feeling that I has a huge amount to learn before I could be productive. Even a year later I still have a huge amount to learn.

No, that's not it. The world is not run by functional programming languages, despite their "leverage". The word you are looking for is "quirky".

Of course, the same is true for the programmers. They are not extraordinarily smart, they make quirky technical decisions. Interest in functional programming solves technical problems as much as your choice of editor.

I don't think that extraordinary skill and power are linked at the language level. Within a rounding error, an "extraordinarily skilled" developer can probably be just as much of an asset to a company with an "extraordinarily powerful" language as they could with an "ordinarily powerful" (read: mainstream) one. Now, super-skilled folks might like playing with Lisp or Haskell or what-have-you (and if it affects your ability to retain them and get value from them, you might want to let them), but their skilled-ness is usually transferable to other languages if the need arises.

Hm. Do you think that the expressiveness of the language you’re working in doesn’t actually influence the scale of what one person can build with it?

Or that mainstream languages already have the maximum expressive power and these are just lateral moves?

I think Lisp doesn't belong in this set in many cases (eg Clojure, Racket), it's pretty easy to learn and use compared to eg JavaScript.

> Why is lisp always rejected?

It isn't always. I have a consulting gig for a chip design company that uses Common Lisp internally. It's one of their principal competitive advantages.

But mainly it's a vicious cycle: no one uses it because no one uses it. Employers don't use it because they think (mostly correctly) that they can't find CL coders, and no one learns CL because no one hires CL coders.

Always great to hear there are companies out there thriving with these more powerful languages. The tired example I always turn to is Ocaml being the language for all the code at janestreet (~900 employees).

I've always been annoyed by the "there are no X coders" argument. Coding skills are broadly transferrable. My team's code base is all python, and we've had successful team members who'd previously worked only in javascript, Java, C#, etc. When we tell recruiters that any of these languages will do, they hate that. Oof.

From my point of view, Lisp gives programmers too much power. No company wants giving a programmer so much concentrated power, they want it distributed around the workforce, and people disposable. Replace them whenever is needed.

With lisp you can modify your development platform to suit your needs, In practice, over time you create your own programming language.

You create your DSP(Domain specific language) with your own systems. This is not standard. You are dependent on the programmer.

Companies prefer to buy "solutions"(packages,environments) from companies and depend on those solutions. They are standard and you can hire people trained on them. Replace them if needed and make people continue the work that someone else left.

That is because "solutions" have open documentation, designed by people that are able to draw beautiful diagrams, lots of books, videos on the Internet, but lisp gurus have all the info in their heads.

Bugs are way easier(orders of magnitude) to find with meta programming. In fact you can create a subsystem that finds all bugs so you don't have to.

The parent comment got downvoted, but I have definitely experienced this "too much power" issue at work in other languages. E.g. one programmer on the team chooses to use powerful abstractions, but then other folks have trouble modifying the code. Usually we give the respect that is due and ask for a simpler rewrite. When the abstractions or tools are powerful and also simpler, and we can all pick them up and use them, that's the happiest outcome.

> one programmer on the team chooses to use powerful abstractions, but then other folks have trouble modifying the code

doesn't this kind of mean they were bad abstractions though?

(am guilty of this myself of course...)

> Bugs are way easier(orders of magnitude) to find with meta programming. In fact you can create a subsystem that finds all bugs so you don't have to.

This needs some explanation.

In homoiconic languages like Lisp or Mathematica the program is just "data" and can be transformed arbitrarily, even at runtime.

This allows you to write utility functions that transform programs in all sorts of useful ways to find bugs.

A simple example is to just transform the program so that it traces whatever you want to a log, much like Visual Studio Intellitrace, but completely flexible.

You can do things like replace all floating point numbers with a "dual number" (a 2-number tuple) where the first number is always rounded down when operated on, and the second number is always rounded up. This then lets you robustly determine if your numerical accuracy is guaranteed to be within a certain range (due to floating point error, at any rate).

You can symbolically evaluate programs involving random variables using their distributions instead, symbolically working out the products of functions instead of scalar values. This allows things like video poker or lottery programs to be robustly evaluated for average payout even if some of the plays are rare, such as jackpots. A friend of mine did this using Mathematica back in the early 2000s.

It goes on and on...

> gurus have all the info in their heads.

Whichever word you put before "gurus" in that sentence, I don't want to work somewhere which values that dynamic. Programming is very occasionally a solitary and purely technical task. The rest of the time it's a collaborative and human task. Eschewing documentation and presentation in favor of "fits in one genuis's head" is not a good thing if the task at hand is collaborative/involves multiple humans.

Not to be pedantic, but since Clojure is a hosted JVM language (as one of a few platforms it's hosted on) and has full interoperability with Java, by definition it has more libraries than Java as it has all the Java libraries plus all the clojure libraries. So definitely not library support in recent years but maybe back in the days. Performance was an issue early on as well, but that's certainly not the case now. Clojure is extremely performant and outperforms pretty much all the popular interpreted languages easily.

Yup, it can also now use all of the Python libraries as well: https://github.com/cnuernber/libpython-clj

So it now has all of Java, JavaScript and Python ;)

This is really a non-argument since Graal [0] can do anything JVM plus JS, R, Python, Ruby, and LLVM.

[0] https://www.graalvm.org/

Polyglot programming with Graal will eventually be awesome, but it isn't yet. If it was, I'd use Graal to use Python libraries from Clojure, but I'm not, because it still has some ways to go.

So for now, I use Pandas, Numpy, matplotlib, and all that from Clojure using libpython-clj. Which is the biggest issue with graal-python currently, how to leverage all the C-based libraries. That's where libpython-clj excels over Graal.

Yep and even then Clojure has Erlang and .net runtimes and interop built syntactically into the language, GraalVM has to reimplement those languages in it's tooling so they'll always be playing catch-up with the real implementations (unless they merge)

That said we do like GraalVM it has a cool ability to create native binaries out of Clojure apps, which is currently powering a resurgence in Clojure for scripting the last wave had to use Clojure's JavaScript target (cljs)

Yeah binding with C is a general JVM problem. Hopefully project Panama will solve that soon :)

What is your experience with python interop in clojure? Does it work sufficiently well that it is really usable or is it rather experimental?

Or clojure, java, scala, kotlin, python, ruby, JavaScript, c, cpp, r all together with graal vm.

https://www.graalvm.org/

>Not to be pedantic, but since Clojure is a hosted JVM language (as one of a few platforms it's hosted on) and has full interoperability with Java,

Armed Bear Common Lisp is an ANSI Common Lisp, thus can use all CL libs, runs on the JVM, and it can easily call Java libraries. In fact as easy as calling regular lisp functions.

Interesting. We didn't look at this closely since we liked the numpy/scipy/sagemath tools, but we should have realized this put clojure ahead of java. Especially java's 2009 features and syntax.

Due to the fact that Clojure has its own classloader hierarchy, quite a few libraries that require AOP like mechanisms (or even OSGi components) cannot be as easily used from Clojure.

I was at JPL on this Lisp project, and left for Silicon Valley. First place I interviewed was Peter Norvig's startup (that's how I think of it, though it had actual cofounders and he was an employee). He told me if they'd known they could hire Lispers, they wouldn't be using Java.

Just one datapoint on how things were trending then.

Sounds like the lisp programmer skillset doesn't have a strong overlap with the java/python/javascript programmer skillset. Makes me want to learn more lisp. :)

There is some overlap with javascript because it's the only one of the three that offers uncrippled lambdas. Or I may be wrong about that; did Java finally add them?

ES6 offers sane lambdas via => and lexical scoping via let.

I can actually program in it without wanting to stab myself, which has previously mostly been limited to perl and various lisps ;)

Java has had lambdas since Java 8. (I don’t know if you’d consider them “crippled” or not; they are less crippled than Python’s.)

Even before Java 8, Java had inner classes that if you squinted at them right looked like very syntactically awkward lambdas (indeed, Java 8 lambdas compile down to something that looks an awfully lot like an inner class).

One of the key selling points of lisp is that it allows you to easily build a language that more closely maps to the problem you're trying to solve. In the 1970s, this was powerful because parsing was an active area of research.

Today, parsing is one of the least interesting problems in language design (unless you're C++ or based on natural language). Most of the effort of spinning up a new language is in the backend (optimization and error checking), where lisp doesn't offer a compelling advantage (well, writing folding rules in C is a total pain, but this is a recursive problem... I can just write a code generator).

> Most of the effort of spinning up a new language is in the backend (optimization and error checking), where lisp doesn't offer a compelling advantage

SBCL literally allows writing compiler intrinsics (and I'm talking about native machine code, not bytecode) at runtime out of the box: https://www.pvk.ca/Blog/2014/03/15/sbcl-the-ultimate-assembl...

So maybe other languages have finally beat lisp at its own game? Makes sense from the wave of functional language features becoming popular in the 2010s.

It's not necessarily that they beat lisp at its own game. I still think the tight integration story of lisp is unmatched nearly anywhere else, but, when viewed holistically, lisp's advantages have become less compelling over time for people who have already written software outside of the ecosystem. It suffers from a lot of the same problems as smalltalk.

I think the key metric for programming language popularity is something like “when someone makes a thing work, how easy is it for someone else a) to find out about it b) understand what’s happening, and c) shoehorn the solution into their own janky application”.

Some combination of easy integration and a big ecosystem of active development? Makes a lot of sense to me.

In the Clojure community we just got access to all of python http://gigasquidsoftware.com/blog/2020/01/10/hugging-face-gp...

I think learning Clojure will continue to pay off the reach you can get with it in one language is insane

I think it was on HN that I read recently, but I heard some complaints that it's a language that you develop on, but not a language that you develop. Rich Hickey develops it and lets everyone use it. May be good or bad but that seems to have turned away some good developers. So anyway I don't know if that has any impact on its long-term future.

What led Google to Java is a good question - I'd guess they wanted to standardize on something likely have longevity, because they knew they couldn't predict the success of good languages, so the same old reason why less all the other corporations usually do it.

Good that JVM eventually started to get nice languages like Clojure, Scala, etc - despite being the platform of choice of corporate Vorlons.

http://winestockwebdesign.com/Essays/Lisp_Curse.html

> Why is lisp always rejected?

My best guess is performance.

In the 70s, computers were slow. When the decade started, the competing languages of the time were Fortran, Algol and Cobol. All were imperative languages with manual memory management that would compile directly to hardware and had semantics very close to that of the hardware. On the other hand, at that time, Lisp was this high level thing, garbage collected, dynamic types, semantics more removed from the hardware, etc. Meaning Lisp was much less performant.

Then in 1972, came C. A language inspired by Algol and Fortran, using very similar imperative semantics and constructs, but changing the syntax slightly to what is now the common syntax most programming languages use.

The whole point of C was to be crazy fast, and for compilers to be easy to write so it could be portable as well. C's biggest move was that Unix was re-written in it. Before that, operating system kernels were implemented in assembly language! Unix was implemented in assembly language! Think about that, performance was critical in the 70s for a kernel.

Once Unix was re-written in C, and C proved to be performant enough to run a Kernel, its momentum started growing. Unix won the OS war, and slowly everyone was using computers running Unix, meaning that it had a C API. What happens when the most popular OS has binding in a specific language, more people choose that language to write programs for that OS. Thus C became ever so popular. Lisp was there, but not leveraging Unix as much, and still much too slow.

And this was the beginning of the momentum for C, it only accelerated since then. Linux came around, also in C, Windows, also in C, all major OS were in C. So C like syntax, semantics and programming constructs became the norm, and was what got thought in school and uni, etc.

Fast forward to today, and Lisp is finally "fast enough" to be used. We finally reached a point where you can have useful programs in slower languages, like Python, Java, JavaScript, C# all demonstrate. But how can you beat 50 years of momentum in C like languages? That's what most everyone knows. That's why imperative, mutable, procedural statement based, with C like semantics and syntax is still the norm, and all other languages that aren't struggle to get a foothold.

I agree with lot what you say.

In the 70s speed wasn't really the big problem, but performance. Sounds similar, but it wasn't. The performance of Lisp on a mini computer seriously degraded with the number of users on the machine and the lack of available RAM (one megabyte?). A single Lisp on the machine could be relatively fast, but Lisp had high memory demands and a simple GC could walk through all allocated (virtual) memory of a Lisp process. Thus the machine got slow when 50 users suddenly worked next to one or two Lisp users. For Lisp users this was especially a problem, since they were often meaning to use interactive Lisp systems, not batch computation.

The idea of Lisp computers was not so much about faster CPUs, but a performant machine for a single user with high memory demands.

90s memory sizes started to get bigger, but Lisp was then hit by the AI winter. Many projects at that time were either moving to C++ for applications (because that was the new efficient industrial language) or tried to develop new languages, which are simpler, cheaper and more deployment friendly (Dylan, Java, C#, Scripting languages in a C runtime, reimplementations of special languages in C/C++, ...).

Cost of development systems and the cost of deployment - we are talking about cost of both software and hardware here - was a problem for a long time.

There were always also people who tried to provide solutions, Scott Fahlman for example directed the CMUCL project and his goal was to have an optimizing CL system for free - providing CMUCL as no-cost public domain software was a major goal for him.

C succeeded because of Unix, not the other way around. Unix was given away and ran on small, inexpensive minicomputers (originally the PDP-11). Lisp typically ran on more expensive equipment.

High level programming languages have been used to write operating systems, including the kernel: Lisp (on Lisp machines), Algol 60 (MCP on Burroughs mainframes) and PL/1 (MULTICS on General Electric and Honeywell machines). MCP and MULTICS (from which Unix gets its name) preceded Unix.

One difference between C and Lisp is that C was designed to run on the PDP-11, and is a close match for its instruction set. Lisp didn't have a native hardware architecture (the closest it had was the PDP-10) until Lisp machines were built, and there were economic reasons - largely economies of scale - why they're no longer built.

> High level programming languages have been used to write operating systems, including the kernel: Lisp (on Lisp machines), Algol 60 (MCP on Burroughs mainframes) and PL/1 (MULTICS on General Electric and Honeywell machines). MCP and MULTICS (from which Unix gets its name) preceded Unix.

IBM also has a long history of using (non-standard and semi-secret) PL/I dialects for operating system development. Much of MVS was written in a PL/I variant optimised for systems programming PL/S (which later evolved into PL/X) – older versions were mostly assembly, with successive versions more and more written in PL/S – and that remains so with contemporary z/OS, although brand new development has largely switched to other languages, especially C.

Similarly, most of the software for the AS/400 was originally written in a pair of PL/I dialects, PL/MP (for code beneath the virtual machine layer) and PL/MI (for code above it.) (Starting with the CISC to RISC transition, PL/MP and PL/MI were progressively replaced by C++.)

Apparently, AIX 1.x and 2.x were written in a mixture of C and PL/I, but IBM got rid of all the PL/I code in AIX 3.x onwards which are pure C.

> C succeeded because of Unix, not the other way around. Unix was given away and ran on small, inexpensive minicomputers (originally the PDP-11). Lisp typically ran on more expensive equipment.

Yes I agree with that, I thought I had made that point clear.

C won over the other C-like due to Unix. But Lisp-likes lost because they weren't fast enough to run on commodity hardware. That meant that the hardware that became popular did not run Lisp, and didn't have an OS written in Lisp. Instead it did C, this has created a massive momentum that still dictates to this day what language is popular and which one isn't, in my opinion. At least it is a strong contributor I'd say.

How then to explain Python's popularity? Modern Common Lisp is dramatically faster than pure Python. CL is compiled down to the metal; Python is usually interpreted.

C functions called from Python (like Numpy) are faster than Common Lisp but pure Python is much slower.

I'm not too sure, but Python has always had great integration with C. It also follows the familiar statement based, imperative procedural model pretty closely. And it has been bundled on unix-like systems for the last 25 years. In that sense, its tottally riding the same wave, leveraging the momentum behind those languages.

That's the crux of my argument. Not that Lisp is too slow today, it's quite fast considering today's programming language landscape. But that it was when it mattered, in the 70s and 80s.

Had Lisp been able to overcome that, we might be in a world where the most popular OS is implemented in Lisp with a Lisp API. Where the commonly thought paradigms in school would be expression based lambda calculus, functional programming, dynamic runtime object systems, meta-programming and the all mighty parenthesis syntax ;)

Already in the early 80s, Lisp was used to program massively parallel supercomputers.

Lisp stopped being slow a long, long time ago.

> Already in the early 80s

The 80s is already 10 years later than my timeline, C had already gained massive momentum by then.

> Lisp was used to program massively parallel supercomputers

I'm curious to learn more about this, can you be more specific or provide links?

But anyhow, a massively parallel computer is a very different beast. Most computers used by people and enterprise were not massively parallel, and Lisps had poor performance on them and their architectures compared to C.

> Lisp stopped being slow a long, long time ago.

How long ago is that? I'd bet it was too little too late already. In my opinion, it would be somewhere along when Java came around, in the mid 90s. And even then, I'd say it was still too slow for most user application, but started being good enough for server work. At that point though, the momentum of C was already unstoppable, and Java became dominant, not CL.

It's the same story. Lisp ran on expensive early massively-parallel hardware, like the Connection Machine. But, again: exotic, expensive, few commercial users at the time, etc. Also killed by the AI winter.

> How long ago is that?

1973.

From http://www.softwarepreservation.org/projects/LISP/maclisp_fa... , "This reports the results of a test in which a compiled MacLisp floating-point program was faster than equivalent Fortran code. The numerical portion of the code was identical and McLisp used a faster subroutine-call protocol."

Original article: https://www.deepdyve.com/lp/association-for-computing-machin...

That's a great article, thanks for sharing.

I think this reinforces my thesis here. You can tell from the article that at the time, Lisp needed to prove itself in the mind of people as "fast enough". That seems to be the intent of the article. Given a good compiler, it may be just as fast.

I'm not trying to bash Lisp, I love lisp, I use it all the time. And when I say "fast enough", I know lisp is manageably fast, maybe custom hardware could have made it equally viable, maybe it only needed a more sophisticated compiler more widely available at an affordable price, but in practicality, it had more hurdle to overcome than other languages to meet the bar of the time.

Imagine an alternate reality where Lisp was clearly fastest. Where its semantics mapped very closely to commodity hardware of the time, and where Fortran, Algol, C, etc. semantics are all further removed from those hardware instructions, making them require more sophisticated compilers or very expensive specialty hardware to have them perform as well as Lisp. I feel it in such a reality, I can't see why C would have become the language of choice and not Lisp.

> > How long ago is that?

> 1973.

> From http://www.softwarepreservation.org/projects/LISP/maclisp_fa.... , "This reports the results of a test in which a compiled MacLisp floating-point program was faster than equivalent Fortran code. The numerical portion of the code was identical and McLisp used a faster subroutine-call protocol."

> Original article: https://www.deepdyve.com/lp/association-for-computing-machin....

To add some more detail, the public description MACLISP floating point code being faster than Fortran, was published about 7 months before public unveiling of UNIX and when C was just taking more solid shape in it.

Interesting. Choosing languages to get work done in the 2010s has insulated my team from a lot of these historical forces; interpreted and garbage collected has always been more than fast enough. I can see how earlier businesses would choose a different route.

I actually worked at Sun as a Java architect back in this era, and consulted with the JPL as they were adopting Java.

In the end, the JPL isn't a software shop. The glory is in the hardware end of things. As a consequence, you want something that has a broad pool of talent, tooling, vendors, etc. to draw upon for support. It also was a more natural transition from their extant Fortran & C code.

LISP's marketshare had already declined significantly at that point, and it's primary remaining user base didn't have very similar needs to the JPL. When you're an outlier for a language with small marketshare, that means the tools, skills, vendors, etc. aren't going to be as well suited to what you're trying to achieve.

tl;dr: when you're in a domain where you aren't trying to be quite so innovative, it helps to run with the pack.

Reply on news.ycombinator.com