How I lost my faith in Lisp (2002) | Hacker News Reader

How I lost my faith in Lisp (2002)

groups.google.com

96 pointsScarbutt60 comments

After 35 years of programming, I discovered Common Lisp and programming is enjoyable again. I wish I'd discovered it earlier.

With what I've learned in the last five years I've (1) written a Common Lisp compiler that interoperates with C++ and uses LLVM as the backend (github.com/cando-developers/clasp); (2) I've used it as the basis of a programming environment for designing new molecules and materials; (3) we've developed a Jupyterlab kernel and ported Jupyter widgets to Common Lisp as the graphical user interface for the programming environment.

I say "I've" and then "we've" because while I started this myself - several people have joined me and we are now turning it into products.

Common Lisp is great - no other language is as rich and expressive and fun to program in. Real macros (code that writes code!), multiple dispatch, conditions and restarts - it goes on and on. There are lots of features in Common Lisp that haven't made it into other languages.

[edit]

Oh - and then there is this: http://greenlab.di.uminho.pt/wp-content/uploads/2017/09/pape... Of all the dynamic languages (and who doesn't like dynamic languages?) Common Lisp is the most energy efficient by a long shot.

Thanks. I really enjoyed your Google Techtalk about CLASP https://www.youtube.com/watch?v=8X69_42Mj-g

One thing I struggle to understand is how common lisp deals with the "modern" multi-core world. Not just "async i/o" - that's just the tip of the iceberg - but concurrency in general.

I know that common lisp (the spec) explicitly doesn't mention any of this - it was the early 90's so who cared anyway - but how would you fit common lisp into a modern server-side app that has to process many things at once? Or is that just not an area that it's concerned with?

I implemented multithreading in Clasp using pthreads and added support for the 'bordeaux-threads' library (https://github.com/sionescu/bordeaux-threads). Now any program that does multi-threading using the bordeaux-threads library works in Clasp. This is how Common Lisp adds support for new capabilities.

A few questions for the Lisp fans out there:

1. The summary here seems to be "modern high-level languages, used, well, are just as productive as Lisp." Does that seem right?

2. One of the tensions for me in technology is between love of simplicity and love of complexity. An example of the latter is Enterprise Java, where the tendency toward a FactoryProxyBeanMutatorFactoryInterfaceImplementation is well known. Is the perceived superiority of Lisp perhaps because it attracts people with strong simplicity bias?

3. Every "Lisp is super-productive" story I hear is about a lone individual. Is Lisp's suite spot that of the solo programmer? Since the rise of the Internet, I think most software has shifted to be about teams, and I'm wondering if Lisp is less suited to team collective ownership or projects where many teams are involved.

re: 3.

I can't speak for anyone else, but some of the most productive research coding I have done was in lisp.

I do think it is difficult to work in large teams with lisps, their flexibility becomes a barrier to communication. The enterprise model of "100s of programmmers, all fungible" isn't quite right anywhere, but it really spectacularly falls apart in an environment like lisp.

I do think lisps are amongst, if not the, most productive languages for one programmer (or a very small team that works well together) to do novel, exploratory programming. This fits really well in some types of research.

On the other hand, if a lot of what you want to do is basically glue - lisp doesn't buy you too much.

For example, I don't thing python is a very productive language, but it is a very productive ecosystem - if that makes sense. Far more so that lisp(s) today for many problem domains.

If you really want to start something basically from scratch, it's still a very good match. But if you are going to spend a lot of time building 70-80% solutions for problems that in another environment would be one include away ... maybe you won't be so fast.

My answers only reflect my own opinions as a long time Lisper and Scheme user. Other's mileage may differ.

1. Yes, more or less. LISP used to be superior to everything else but modern languages have caught up in most areas. LISP still has a few special tricks like MOP in CL, macros, dynamic loading of code, resumable exceptions, but these are rarely needed in everyday programming.

2. No, not at all. Neither CL nor Racket nor any other mature LISP and Scheme dialects are simple to use and easy to learn. They have long-steep learning curves, especially idiomatic CL. The syntax is somewhat simple if you ignore special reader extensions, but that's irrelevant. CL is a large language, you need to learn both imperative and functional programming techniques, and then there are tons of libraries and conventions to learn.

3. Yes, I would say. LISP is incredibly powerful in the hand of one hacker who really knows the system. You will get awesome results very fast and can do things that would be really hard in other languages. It also encourages high level abstractions like no other language I've ever seen. But the main disadvantage of LISP is maintainability. It may be hard to even understand your own code a year or two later, let alone that by other people. Everybody has his own stile and every larger program will create a DSL you have to learn in order to understand what it does.

> But the main disadvantage of LISP is maintainability.

LISP allows programmers to use abstraction so powerful that sky is the limit. Problem is, humans have their limits, too. And different humans have different limits. Abstractions are powerful, but then you have to think abstractly in order to use them properly.

When you are good at abstract thinking, LISP will liberate you. When your colleague is better than you, or just as good as you but has more LISP experience, reading their code will make your head hurt. To make you two cooperate, your colleague would have to give up some of their powers... but the ability to use those powers to their maximum was the thing that made them love LISP. So now one of you is going to suffer.

Other languages often put artificial limits to your abstract thinking. For example, you have a few parts of code you realize are somehow just different instances of the same pattern; the pattern could be extracted and reused... but doing it in this language would either be impossible, or the outcome would be really ugly, or it would require writing so much boilerplate code that the more abstract version would end up being longer, less legible, and not really nice at all. So you sigh and don't do it. And the same thing happens to your colleagues in similar situations, so now you are all writing approximately on the same level of abstraction.

In other words, with lesser languages, the frustrating thing is the language. With LISP, the frustrating thing is other humans (and that may include yourself on a different day). But the frustration is always there. Unless you are really good and working alone, I guess. But that puts the company that employs you in a dangerous situation, so it is unlikely to happen.

When your friend is a faster runner, and is wearing running shoes, he will leave you in the dust. Solution: everyone wears combat boots, and carries a 30 pound load on their back. Now the field is level; and you even have to cooperate to go forward.

That's the sad fact about human nature: when you have a project that requires everyone to move at the same speed, chaining people together and putting unnecessary weight on their back works better (more reliably) that asking the fastest ones to please slow down a little.

(Then of course there are also other problems, such as companies wanting to keep all their employees completely replaceable, which is incompatible with people using their unique skills.)

Ah, thanks, this makes a lot of sense to me.

For me, one of the key determinants in how I write most code is how approachable it will be to the next person. Because anything significant is going to involve more people, and I don't want to have to be chained to my past successes.

I think there's an analogy with writing. If I'm writing something for myself, or for a narrow, specialist audience, I can indulge my desire for all sorts of things: abstruse ideas, obscure words, Dickensian sentences, punny wordplay, tangents galore. But when I'm writing here I try very hard to rein those desires in. Because here the writing isn't about me.

So I think LISP's ability to turn my ideolect into a tower and creeup up into it is exactly what I don't want in a tool.

How "1. ((...)) LISP used to be superior to everything else but modern languages have caught up in most areas. LISP still has a few special tricks ((...)) but these are rarely needed in everyday programming." and "3. ((...)) You will get awesome results very fast and can do things that would be really hard in other languages." can both be true?

1. The gap is smaller, and shrinking every year but still exists, and it's not just macros; the condition system and the tooling is way better.

The tooling alone is sufficient reason for me to use lisp. Even commercial IDEs for python (e.g. pycharms) are nowhere near as good as slime, plus sbcl is a compiled language. I can change a function, run a test and get instruction level profiling information. Pretty much no modern dynamic language lets you do that, and the non-modern static languages (e.g. C++) take longer to link (much less compile) then it takes me to do all of those steps for incremental changes in lisp code.

2. I have a strong simplicity bias, so I can't argue there. I will say that lisp is good at "getting out of your way" which helps remove a lot of accidental complexity. Java (particularly the Java from 15 years ago), is notoriously bad at "getting out of your way" so they are near opposite ends of the spectrum there.

3. Lisp works fine in teams; most languages with small communities have fewer team projects just because the set of possible teams grows quadratically with the size of the community. This is exacerbated in the lisp community which historically has several silos. For open source projects, SBCL (which is written in lisp) is the first team project that comes to mind.

For commercial software, QPX comes to mind; ITA was listed as having 400ish employees, I'm not sure how many were working on lisp code though.

I feel like Lisp didn't win out for the same reason that HP calculators never won out: people don't like the mental burden of flipping their calculations around by hand. Which is another way of saying that functional programming (FP) doesn't follow any of the imperative programming (IP) conventions they're used to (by design, generally).

Everything about FP is conceptually good except for the human element. I find other people's FP examples to be exceptionally hard to read. But in fairness not as hard as object-oriented programming (OOP) IP languages like Ruby or Angular. I picture them as two extremes: abstract formalism with FP on the one hand and syntactic sugar/convention/patterns with IP/OOP on the other. I'm classically trained so to speak, so I can make the mental leap to FP when I need to. But I continuously have trouble with IP languages because for the most part I don't understand what problem they're trying to solve.

I think Elixir, Clojure and F# are making inroads on the FP<->IP/OOP front but they still aren't easy enough for mainstream adoption IMHO. Haskell, Scala etc are too fringe and have probably already reached saturation.

Also I do agree about the ignorance aspect. Most programmers I've met don't understand the difference between simple and easy, as examined in depth by Rich Hickey:

https://www.youtube.com/watch?v=34_L7t7fD_U

This is "famous". Look, here is a Japanese translation:

http://web.archive.org/web/20021113063322/http://www.gembook...

Link to pure article headers+body:

https://groups.google.com/forum/message/raw?msg=comp.lang.li...

"At this point I was seriously considering the possibility that there really was a thriving Lisp economy out there somewhere, and that I was being excluded from it for some reason, like maybe my reputation for being obnoxious. (But even that theory came unraveled when you showed up, Erik.)"

Ouch.

And in further posts, Erik responds,

"Sigh. Will you _ever_ get a clue? (For full credit, your answer must be 2000 words or longer.)

"A free hint for you: Cut your losses and just _move_on_. If Python is so great, enjoy it for what it is, and make it your new community. Pursue your happiness -- if it is to be found elsewhere, move accordingly. Hanging around here apparently makes you progressively more unhappy."

I saw that too...ouch. What's the beef about here?

Erik was (he has passed away) fairly obnoxious. Like makes Linux Torvalds look like a diplomat obnoxious.

e.g. Naggum on XML: http://harmful.cat-v.org/software/xml/s-exp_vs_XML

Erik was rather hostile online, but his technical abilities were quite high and his opinions well-researched. I met him once IRL (as did many Lispers when Franz paid his way to the Lisp conference in Berkeley in (can't remember which year. PG was a keynote speaker and McCarthy was holding court in the hallway.) Erik seemed perfectly polite in person. To me anyway.

My summary: he worked at Google in the early 2000's (when there were indeed many astonishingly productive progammers), and he saw them being super productive in languages like Python. And he himself became productive in such non-Lisp languages.

He uses the hash table as an example, and I think it's apt. Lisp does feel "old" to me with respect to having 10 different choices for hash tables in 10 different dialects. (I briefly worked with Julia's femtolisp a couple years ago and felt this.)

I also felt it when trying OCaml like 5 years ago. I loved the language, but having to "choose" which hash table to use felt odd.

Newer languages like Go and Rust (I think) both have Python/Perl-like hash tables built-in, or at least close to the core where all libraries can interchangeably use them. I think this is here to stay. Hash tables won :)

Also, standard libraries that make good choices have won.

These are some of the most annoying part of any language: In Perl the key of a hash must be a string, in Haskell, String is a linked list. And so on.

FWIW, in Perl 6, keys to so-called "object hashes" can be objects without them being stringified. See https://docs.perl6.org/language/hashmap#index-entry-object_h...

You have four different choices for hash-tables. You choose how the key hashing should work with four levels - pointer hashing (which is very fast) to hashing the contents of arrays and list keys (which is slow). If you only get one choice then hash-tables wouldn't be as powerful.

And in many implementations you have several options for weakness on hash tables. While not required by the standard, weakness is a very powerful enhancement to hash tables. (Weakness is the kind of thing programmers could add themselves if CL had a MOP for the garbage collector.)

Did Google have a lot of Python in the early 2000s?

In 2002, C++ had a plethora of non-standard hash-tables: http://www.drdobbs.com/the-standard-librarianhash-tables-for... Today there are a plethora of standard hash tables.

Yet C++ isn't dying like Lisp.

Clojure places a lot of emphasis on its built in hashmaps.

Previous threads:

https://news.ycombinator.com/item?id=2308370

https://news.ycombinator.com/item?id=107683

On one of these I like the comment "You sound like an English speaker claiming that English is easier for people to understand than other languages. "

well yeah of course english is easier, else there wouldn't be these kind of tables : https://www.effectivelanguagelearning.com/language-guide/lan...

There's no equality amongst languages. Japanese students have to get up and do one hour of kanji learning every year from 8 years old to 18 years old in addition to normal language classes for instance - there's nothing comparable in english.

Adults don’t have to go that route though. I learned all the kanji from daily use through college level (some 3,500 or so) in a few months of study. You might as well point to the massive number of root vocabulary English has and make the same argument that English would be crazy difficult to learn (and you’d be equally right).

That has nothing to do with learning English.

It’s based on the premise ‘as an English speaker’ these languages are easy to learn. But, if your native language where say Dutch you get a different list than if your native language is Japanese.

That table is ranking languages by how easy they are for an English speaker to learn -- that is, how much commonality they have with English. It doesn't make any claim about English having some fundamental property that makes it easier to learn.

From the first paragraph:

"The Foreign Service Institute (FSI) has created a list to show the approximate time you need to learn a specific language as an English speaker."

That table is expressly from the point of view of an English speaker; I'm sure it would look very different if written for Arabic speakers. Since it's largely a function of similarity to your native tongue, how would we even measure if a language is easy or hard to learn?

And what counts as "normal language classes"? Until at least high school in the US, every class other than math focuses on reading and writing.

Dutch and German are classified differently (cat 1 vs 2). But it is a well-known language gimmick of these 2 specific languages that when a native speaker of one of these languages speaks slow enough, a native speaker of the other one can understand him reasonably well. So where do these 6 weeks of extra work come from?

(2002) I believe. I wonder where the author stands now? 16 years is plenty of time to get your faith back and lose it again.

you could ask him yourself. these days he goes by the name "ron garret," and he commented right here on this site only an hour ago.

https://news.ycombinator.com/user?id=lisper

Thanks for the tip!

He wrote a followup to that post on his site: http://blog.rongarret.info/2008/02/what-are-programming-lang...

Main quote addressing your question:

"One thing that many people found unsatisfying about my how-I-lost-my-faith posting is that I never really got around to explaining why I lost my faith other than saying that I saw people being productive in other languages. Sorry to disappoint, but that was basically it. What I think needs clarification is exactly what faith I lost. I did not lose faith in Lisp in the sense that it stopped being my favorite programming language. It didn't (notwithstanding that I switched to Python for certain things -- more on that in a moment). What I lost faith in was that Lisp was the best programming language for everyone (and everything), and that the only reason that people didn't use Lisp is that they were basically ignorant. My faith was that once people discovered Lisp then they would flock to it. Some people (hi Kenny!) still believe that. I don't."

If you are interested in LISP but want to have a much lower risk of having an aneurysm or becoming a hermit, I'd recommend Clojure/Clojurescript

I first met Lisp in university and I fell in love immediately. Over the years though, I went through the same thing as the author here and now I'm happy with the mainstream languages.

I still think it's beauty is unrivalled but in the OO world, Typescript comes pretty close.

Clarification: He only lost his faith into Enterprise Common Lisp. He didn't make his Lisp into Space (the Mars Rover), only as simulator, and then he had to write the Google Adwords program in Java.

There are still plenty of successful but very small Enterprise Common Lisp companies, many of them being bought later and converted into something worse: Viaweb, Ithaka, Nintendo, NaughtyDog, ... KTI, Bentley or Grammarly still doing strong.

https://lisp-lang.org/success/

> What's more, once I got knocked off my high horse (they had to knock me more than once -- if anyone from Google is reading this, I'm sorry) and actually bothered to really study some of these other languges I found myself suddenly becoming more productive in other languages than I was in Lisp. For example, my language of choice for doing Web development now is Python.

Wonder what critical features of Python are particularly hard to replace with Lisp extensions. Interface with other runtime environments?..

I had tons of fun with Lisp in college and have wanted to use it various times in my career but never had any success convincing my colleagues.

I am also very fond of Perl and I've been able to use it many times because it was already there (I didn't need to install anything) and I could find excellent modules on CPAN.

I bet Lisp would be way more popular if every Linux installation came fitted with it.

Ron Garret did a talk at Google closely related to this post in 2012.

https://www.youtube.com/watch?v=_gZK0tW8EhQ

How do I view this without logging into Google?

I just clicked the link without logging in and it worked.

Google Groups often gets into a weird halfway-authenticated state that won't let you see public posts. It's visible if you're actually anonymous. Try it in an incognito window.

You dismount your high horse and log in.

> BTW, ignorant is not a pejorative term. If you think it is you are ignorant and you need to go look the word up in a dictionary.

Did that. The Merriam-Webster, in fact. Here's what they said:

> There are several meanings of ignorant, all of which are concerned with a lack of knowledge in some sense; some of these are more insulting than others, and care should be exercised before applying this word to people who you do not wish to offend. Saying “They were ignorant of most of the laws of physics” means that the people in question did not have a specific body of learning. Saying “You are an ignorant person” is possibly describing someone as primitive, crude, or uncivilized.

https://www.merriam-webster.com/dictionary/ignorant

I'm going to resist the temptation to call the original author ignorant since I don't want to spawn some sort of infinitely recursive ignorance-accusation chain. Instead, I'll call him unintentionally rude.

> some of these are more insulting than others, and care should be exercised before applying this word to people who you do not wish to offend

This is not in the definition. Please try to stay honest when chastising someone for not being accurate.

> a : destitute of knowledge or education

> also : lacking knowledge or comprehension of the thing specified

> b : resulting from or showing lack of knowledge or intelligence

The claim that a term is not pejorative is a claim about usage and social reception, not definition.

Far too often people confuse dictionary definitions and colloquial usage. Dictionary definitions always lag colloquial usage. The meaning of words comes from their usage and the quote you referred to seems like someone being intentionally obdurate to linguistic changes. The person acknowledges that there are people who use the word ignorant in a pejorative way.

OK, I'll bite. Do you live or work in an environment where you could call someone ignorant and be understood to mean only that there is some specific bit of knowledge they don't posses, without making a larger accusation of intellectual incompetence?

Where?

I try to be sensitive to issues of usage like this because civility does not come easily to me, and I have to be careful of what I say lest I inadvertently offend. And I am confident that when I have heard the word used, it is almost invariably in the larger sense. Use in the narrow sense tends to be by foreigners, who seem to be looking for a translation into English, and could easily not know the broader implied meaning.

I think you misunderstand my post. The person who was quoted as saying that “ignorant” isn’t pejorative since the dictionary definition of the word isn’t pejorative is being an ass. Colloquial usage trumps dictionary definitions when it comes to interpreting speech.

Depends on the context, too. Especially in non-casual contexts, where more rigor is expected, you take greater care to use and define terms more consistently in order to make communication more accurate and effective.

Oops. My bad. Sorry.

This is a great article that sums out the direction these threads typically go in and why it's generally a waste of time: https://www.lesswrong.com/posts/7X2j8HAkWdmMoS8PE/disputing-...

Meanwhile, I ported Arc to JS: https://imgur.com/0Ba0NZN

If LN turns into anything, it'll be because of Lisp, not in spite of it. It really doesn't matter that the world is phobic to it when you alone are the blacksmith.

One could argue "See? It's running in JS. Doesn't that mean Lisp is useless?"

Maybe. But macros are a thing. And when you can generate React on the fly, without having to make a class for every single thing you want to do, the power disparity starts becoming very apparent.

There are interesting Lisp codebases, but you have to dig for them. Abuse (a game engine) comes to mind. http://abuse.zoy.org/browser/abuse/trunk/data/lisp

And what other language could let you add type inference with relatively little effort? https://web.archive.org/web/20070610012057/http://www.cs.ind...

https://web.archive.org/web/20070615124421fw_/http://www.cs....

FWIW, React is a built-in feature of the new Arc-in-JS port. Here's how it looks:

  (<a> href: "https://news.ycombinator.com" "Hacker News")

That lets you merge s-expression syntax with React syntax quite nicely.

  (<html>
    (<body>
      (<div> width: "100%" "Hello, world")))

Here's the output of a REPL session.

  $ rlwrap bin/lumen-node

  > (load "arc.l")

  > (print:compile:expand '(<a> href: "https://news.ycombinator.com" "Hacker News"))
  React.createElement("a", {href: "https://news.ycombinator.com"}, "Hacker News")

  > (print:html (<a> href: "https://news.ycombinator.com" "Hacker News"))
  <a href="https://news.ycombinator.com">Hacker News</a>

  > (print:html (<html> (<body> (<div> width: "100%" "Hello, world"))))
  <html><body><div width="100%">Hello, world</div></body></html>

  > (print:html (whitepage "Look ma, no Racket"))
  Warning: Each child in an array or iterator should have a unique "key" prop.

  Check the top-level render call using <html>. See https://fb.me/react-warning-keys for more information.
    in body

  <html><body bgcolor="white" alink="blue">Look ma, no Racket</body></html>

You can see it's actually JS, since you get all the same warnings that you'd normally get in a node repl. (It's literally running on Node.)

And (print:html (msgpage 'shawn toofast*)) spits out a page saying "You're submitting too fast. Please slow down. Thanks."

I find it much easier to generate HTML than traditional methods, and much more maintainable. But it's not fair to claim that something new is inherently better. Time will tell.

I'm most interested in any unexpected warts, but there don't seem to be any so far.

Current code is at https://github.com/lumen-language/lumen/blob/f7bfd4dca71ed1e... if anyone's curious. It's basically a copy-paste from the official arc3.1 release.

(You can diff with the official sources at https://github.com/arclanguage/anarki/blob/f01d3f9c661eed055...)

Apologies for the nested replies; wasn't able to edit.

Reply on news.ycombinator.com