Hacker News Reader

Comment by ducharmdev | Hacker News Reader

I hear sentiments like this often with lisp, but if this were the case, why isn't everyone using lisp?

Largely because of myths surrounding it, for example parenthesis syntax and lack of editor support. With paredit, you get meta level direct AST manipulation support in emacs that is still light years ahead of any programming IDE out there. I think the other fear factor is you have to think a lot more about how to approach the problem when writing Common Lisp and it requires a more complete engineer due to a smaller library ecosystem. Usually there is one and sometimes two libraries for doing X and if not you’re up for making a library yourself.

I’m attaching an example of how quickly you can write TLS 1.3 by a single person in Common Lisp. At the time I wrote this library most websites still struggled with TLS 1.3 support. The point is that most software creators don’t have the breath and depth of knowledge required to be proficient in LISP if they only posses siloed domain knowledge.

Links:

https://github.com/mateuszb/tls1.3

If you want an example of expressivity you can take a look at EC crypto with a full NIST test vector set in about 300 lines of code

https://github.com/mateuszb/tls1.3/blob/master/elliptic-curv...

> With paredit, you get meta level direct AST manipulation support in emacs that is still light years ahead of any programming IDE out there.

Do you happen to know of a video or article with visual examples that demonstrates this? I always hear about these things with Common Lisp, but I have personally never seen it. I’d love to understand how this works or enables people.

There's the Animated Guide to Paredit:

http://danmidwood.com/content/2014/11/21/animated-paredit.ht...

It really feels more like you're working on a tree of code than just editing text.

That’s helpful. I’ll read it more in depth when I’m off mobile as that site is hard to read on mobile.

Although, I think there’s still something missing between those demonstrations and the quoted claims.

I have never used Paredit when doing Scheme or Racket, so maybe I should just hunker down with it at some point (I have looked into it before). I don’t use Emacs (tried it a few times), but there are several Paredit-based VSCode extensions.

Paredit is the editor part, but then there is also SLIME REPL. In lisp, unlike in other programming languages, you don't start writing your code with a blank slate and an empty source file.

A common approach is to start with an initial core. The idea is that all programs start the same, with an initial core, that contains runtime and standard library. You can test things out in a REPL, that you run side by side with a text editor. As you work things out in REPL, you can start moving them to the text file and the program starts taking form and shape. It's a much more exploration driven design and with fast structure editing of paredit you can really transplant or reorganize pieces of code really quickly. The idea is to grow the program from the initial core into something else inside-out style. It's akin to being dropped into a running program, and modifying it as it is running but that is not possible with most languages easily and as interactively as in Common Lisp for example.

With SLIME the REPL and text editor integration, you can for example write a buggy function, test it in REPL and find out about the error. Modify it in the text editor, press Ctrl-c twice, and the function is hot reloaded on the fly. No reloads, no re-imports, no-caching, no program restarts. Next time the function is called, it is replaced with a new stuff you just wrote. It really is pretty magical. It just works, and works for classes, objects, meta classes, functions, variables, etc.

The language is unlike other languages, because it offers object introspection (often mistakenly referred to as reflection) and object intercession; introspection and intercession combined give you a true reflection. It is the latter technique that supports all of the above, for an imho superior development experience. It really just works, and it works marvelously well. No yak shaving for hours on end trying to get some dependency chain to work, compiler errors, package version mismatches, etc. No need for build team, no need for testing/qa team, no need for most of your usual dev overhead we are so used to these days :)

Is there any source to learn this way of development?

You could start with a copy of the book „the land of Lisp” or „practical common Lisp”

People seem to recommend portacle https://portacle.github.io/ for a first dev environment but I usually set everything up myself. Portacle seems to have all the essentials. You can follow SLIME installation instructions from the quicklisp page of portacle doesn’t come with it.

Then I recommend slime manual at https://slime.common-lisp.dev/doc/slime.pdf if you’re going to be using Emacs.

I’m not familiar with other tools.

Next Advent of Code, grab yourself a download of Portacle and see how far you can get with Lisp.

By day 7 you should have a feel for what people mean when they say you are directly editing the AST.

Basically: there is the same amount of friction to moving constructs around at a high level as there is at a low level, and this is due to Lisp’s uniform syntax, and paredit leverages this a lot.

Contrast this to a language like Java, where the friction between modifying a single expression and modifying class-level interactions are totally different, syntactically. Now you find yourself leaning on highly specific IDE-level support or Lombok for things like setting up Builders for every POJO, to get around these constraints.

From a Lisper’s perspective, these syntactic constraints are unnecessary, and should at most require a library that you can trivially port to any editor. Which is exactly what paredit achieves.

I have used Scheme and Racket a fair amount and know bits of Common Lisp.

What I don’t understand is what I stated and what I quoted.

Without seeing it demonstrated, I just don’t understand the quoted claims. When I’ve done Scheme and Racket, I never really needed to be copying and moving ASTs (i.e., parenthetic groups) all over the place. Maybe that’s just me and my style, for better or worse.

I think "structural editing" is a better way to talk about it. You manipulate code as units of S-expressions. You don't type parentheses so much as you create and modify (trees of) S-expressions. So things like unbalanced parentheses or improperly indented forms are not possible since there's no way to create them in the first place.

In a somewhat vague sense, the "units of editing" are not characters.

There are some videos on YouTube demonstrating ParEdit and SLIME.

That’s fine, but I don’t see where one gets "light years ahead of any programming IDE out there" from that.

I like Lisps/Schemes as much as the next person, but other languages don’t have parentheses to deal with, so by that same token, you’re getting that for free, such as an ML like F#.

There isn’t a single language IDE that I know of outside of Lisp that allows you to hoist a sub block of a loop (a sub tree in the AST) and remove the parent of that code and paste the hoisted expression while simultaneously removing the loop construction with a single key. Please show me an IDE like that and I will buy it immediately. Paredit is not about parenthesis editing and matching despite its name.

Okay, but my original question remains.

with lisp structural edditing you are editing blocks of code, as opposed to characters. in fact it is the use of s-expressions that makes this possible. here is a simple example. immagine you have a list, two elements just outside it, and your cursor is just after the second element:

  Python: [x,y,z] 1,2
  Lisp: '(x y z) 1 2

and lets say you want everything in the list st

  Python: [x,y,z,1,2]
  Lisp: '(x y z 1 2)

in python it would take me about 5 keystrokes to achieve that, while with paredit in lisp it takes 2. this is just a very simple example of the most basic paredit operation. extending this approach to your whole source file and treating whole blocks of code similarly to the above examlle i find really handy

I think the paredit stuff is a bit overblown but apart from managing parens for you, another simple example is editing single expressions. e.g. in Java you might have a line: "int a = blah.bar(something, thing, whatever);" If you realize you need to actually pass "whatever" first, not last, unless you know an IDE shortcut that can make the edit for you, you're going to have to type stuff. I would probably just move my cursor to the start, type "whatever, ", move my cursor to the comma after "thing" and highlight to the end then delete. If "whatever" was a longer variable, or even more interestingly an entire sub-function call like "whatever(x, y, z)", I might instead highlight it all, cut, backspace the comma, move cursor to the start, paste, type a comma. Moving the cursor to the start might be character by character, or using the operating system wide ctrl+left/right to move in 'word' chunks, or possibly it's faster to use the 'home' key first or god forbid the mouse. Oh no, I might miss a comma or somehow mess up a paren/semicolon or typo a name?! Whatever, it's rare for me, and for most mistakes I'd get a red squiggly alerting me to it immediately. I like typing, and prefer most 'helpful' plugins get out of my way for most things, so such a process isn't that annoying to me.

But I do at least see there's a nicer process if you have something like paredit: you just move you cursor to the "whatever" (even if it's instead "whatever(a,b,c)") and a command will move it to the left/right/etc. and fix up anything that needs fixing up. In Lisp though the base syntax is so simple and uniform that there's not usually much needing "fixing up" -- there's no pesky commas to deal with for instance, and having the opening paren come in front of the function name instead of after simplifies a lot of things. The worst is adding/removing/moving a form that's at the end of a let binding, or sometimes introducing a let binding, or sometimes adding something to the end of a function that previously ended with ))).

I like to use vim (which does have paredit though I have it disabled) and just having the ability to jump between open/close parens by pressing "%" and to cut jumps as a whole, or the insides, without having to move my cursor character by character, is good enough for me. I still use some paredit-like commands in some instances like moving forms around or in those "worst case issues" I mentioned but I use them with these mappings: https://github.com/tpope/vim-sexp-mappings-for-regular-peopl...

There are more advanced things but how much I care about them varies; I don't tend to need them for Lisp, though every so often I'll miss something from Eclipse that I suspect not even emacs does (or does well). e.g. I know emacs can do a "templateized" completion just like a Java IDE where you type a function name and it completes it and inserts its required arguments as placeholder variables to later define/type over, I don't know though whether emacs can then let you place the cursor over each one in turn and with something as easy as 'ctrl+1' hoist that var to an assignment form just above (I did this all the time in Eclipse to avoid having to choose a name, type it, and type its correct type). (In Lisp it's complicated by needing to introduce a let binding if it doesn't exist or append to one if it does. It wouldn't surprise me if paredit can do this, it's just that I'm aware of some refactoring tools in Slime but they don't tend to approach what Eclipse or IntelliJ users expect even if in theory they could.)

you could take a look at this in action here:

https://www.youtube.com/watch?v=D6h5dFyyUX0

Lots of reasons.

1) Lisp acquired a reputation of being associated with AI, back when that was a bad thing, lots of government-funded "expert system" boondoggles and the like. It is not considered a general purpose language, despite being one.

2) Lisp is thought to have poor library support. Unlike Java or JavaScript, which have standard libraries for every task, Lisp is thought to lack this. Quicklisp helps, but it's not as comprehensive as Maven or npm.

3) Lisp is associated with a certain programmer type: highly intelligent, persnickety, only interested in solving interesting problems. Companies prefer programmers who have only a moderate amount of technical skill but who are diligent, who will take orders and get the job done without complaint.

4) Not many programmers actually know Lisp, because not many bother to learn it. If CS students are exposed to Lisp in college, they will complain about it and won't touch it again after passing required classes that use it. This means Lisp has a much smaller qualified candidate pool and it's much harder to hire Lisp programmers.

5) Software projects, especially enterprise software projects, are optimized to require lots of programmers each of whom develop one component without stepping on each other's toes. In the enterprise, not breaking existing things is more important than building new things rapidly. This is why OO languages like Java are so well loved... they help encapsulate programmers from each other as well as encapsulating units of functionality. Each programmer poses more risk in a small team where every programmer's responsibility is larger (and every programmer could theoretically rebuild the whole system themself given enough time) than in a large team where none of the programmers have a complete understanding of the system and everybody stays in their lane.

7) Larger programming teams are just more impressive. As a manager if you have 100 direct reports instead of 10, you are a bigger deal. You can command a bigger budget, which also makes you seem more important.

So the odds are pretty stacked against Lisp. It doesn't scale well to large teams, but it lets a small team get more done. The incentives in most software development reward large teams of programmers just barely smart enough to get the job done, not small surgical teams of wizardly Lisp hackers. So Lisp remains niche, perhaps rightly so.

You might be able to hire 100 Lisp engineers. It's just super expensive, they're going for really high prices right now, and it might be way overkill.

To expand on this, JPL probably has 100 Lisp engineers because it has intricate astrophysics problems that warrant 100 Lisp engineers. Most companies will never have such difficult problems. They may think they can, but they cannot. They can have crappy problems, however, and crappy problems can tie up those engineers provided they have very little agency in solving the underlying problems that give rise to those crappy problems. And you have to be careful to manage them very adversarially to keep it that way because they will totally try to actually fix things without your knowledge or permission. Not always, though, sometimes they get stuck in a mesh of very nerdy concepts and then there's no risk.

Really good observations

> It doesn't scale well to large teams

Doesn't saying stuff like this just contribute to your reasons 1-3 that boil down to perception issues, and not anything wrong fundamentally with the language itself or even its modern day practitioners? I don't disagree that Lisp doesn't scale in the sense that there's not enough Lisp programmers available to actually scale to such grand sizes of even 1000 developers right away, but as I'm sure you already know I want to say for the benefit of passersby that Common Lisp itself is object oriented, has namespaces and can support a modular or component style of development, its global memory model isn't too unlike that of the JVM prior to Java 9 (i.e. everything dynamically loaded and being introspectable and to some degree changeable (more effort in JVM)), it's been used for multi-million line projects (not sure how large the teams for those are/were though), and nothing of the recent microservices trend to scale in a different way is forbidden to Lisp...

Besides, it's not like enterprise companies that have 1000+ devs actually have a 1000+ size team with a single standup. No, things are divided into smaller teams of around 2-10 with maybe 6 being average.

I disagree somewhat with your reasons 3 and 5 and 7, having worked at an enterprise company with thousands of developers and been involved in hiring processes. Maybe I was just lucky and other enterprises are different? In my case, the implicit incentives might sometimes align in the direction you indicate, but explicitly, no, they don't. Hiring selects for stronger technical skills over weaker technical skills, interviewers prefer candidates who have a story of leadership/taking action/dealing with people socially (i.e. how do you handle such and such kind of a disagreement) over heads down and take orders and pretends there are no disagreements, no participation. Each engineer is pressured into career growth and expected to reach a certain role level after N years. Getting promoted to such a level typically involves taking on some sort of leadership role that impacts more than just your own team, and often more than one such role or a secondary role in another's project because things fall through. Despite encapsulation, things very often broke in the monolith (note only in development, production breakages were rare), and many new hires had to be trained to not sync every day if they wanted a more peaceful life. The teams that end up with a domineering management who allow no input from the development side typically don't last long because the developers switch teams (easy to do in an enterprise org where there are tons of teams with headcount every quarter) and with no team left area ownership will be dispersed. No manager has 100 direct reports. They might have 100+ indirect reports, but after a certain direct size, they have to become a higher level manager. They may still retain a couple ICs as direct reports but most of their directs change into a small number of other managers or directors, in classic org chart hierarchy.

This is yet another “non-Lisp programmers are not as smart as Lisp programmers” comment. Seriously? I know Lisp well and have implemented my own variations for fun. But I prefer other languages. Lisp has been over hyped for 50+ years and have still very little to show for it. The world runs 99.999% on non-Lisp code. So perhaps consider that there might be a reason why? Unless you actually want to suggest that 99.999% of developers out there are less intelligent than Lisp programmers? Really?

What did you implement? :)

A console game generating $500 million+ in revenues, a high performance optimising compiler used in production, an embedded web server used by companies in the field, Android software used for warehouse management, commercial 3D engines etc. And much more I can’t remember right now. Why?

lol. ok. but do tell (or better show!) what lisp you implemented

So what have you implemented in your magical 10x language? I told you some of what I have done not using Lisp. Here is your chance to brag about all the cool production stuff you have written in Lisp. Let me get my popcorn. I am waiting :)

yes you made some claims but i am a little bit skeptical :) tbh i don't think you have a handle on this subject at all. also i specifically asked you about your implementations of lisp and you mentioned everything except that. it's not interesting. as for me i work in machine learning and we are starting a project in which lisp is used as a sort of glue language between various components. i only wish more of the stack consisted of lisp components, but who knows maybe it will eventually :)

So you don't have any experience using Lisp on large scale projects. And the project you are starting now is only using Lisp as a glue language. Count me not surprised.

lol

I don't disagree with you about whether Lisp magically makes people more productive: It doesn't. But you've asked for this kind of example a few times in the evident expectation that no one can supply an answer, and I can, so, just so that there is such an answer in evidence, here's mine.

I won't distinguish between projects that I wrote 100 percent myself, and ones where I was a contributor on a team. I include examples of both. Similarly, I won't distinguish between pure Lisp projects and ones where Lisp was one important tool among several, or between projects that I conceived from the start and ones where I joined someone else's.

- a knowledge-based application compatibility testing system used in production by Apple to identify and report thousands of compatibility bugs in its system software

- an experimental operating system for an Apple mobile device

- a network security appliance acquired and shipped by Cisco

- a direct-manipulation-based rapid application-development system

- an embedded system controlling a forensic fingerprint-scanning device

- a consumer-focused simple list-management database

- a commercial graph database

- a framework and compiler for building mobile simulation games

- a knowledge-based control system for laser-sintering manufacturing machines

- a strategy and control system for networks of real-time sensing devices

I've omitted hobby projects and things I built only for my own use. All those above were for employers, clients, or direct sales to customers.

Although, like you, I don't believe that Lisp magically transforms anyone into a 10x programmer, I have actually been a 10x programmer (according to DVCS statistics or stakeholder reports) on some projects, but I've only managed to do that using Lisp.

To be fair, I have also been a 0.1x programmer, but never for very long. It's hard to be happy or to prosper in such a situation.

Again, I don't disagree with your main point. Lisp won't magically make someone amazing. I do think, though, that it's a powerful force multiplier for programmers who get along with it well, and that for certain people (myself among them) it's the best kind of tool.

I also don't think much of anything is proven by failing to find anyone who's done substantial work in Lisp, or who did things in Lisp that they couldn't do as easily without it, but if that did prove anything, then I guess I would be the disproof.

thank you for this! it is always so nice when someone uses 'argument from authority'[0] to pounce their preconceived notions onto everyone else, then someone else from even more authority proves them wrong. i also appreciated the following piece of honesty:

> I won't distinguish between projects that I wrote 100 percent myself, and ones where I was a contributor on a team. I include examples of both. Similarly, I won't distinguish between pure Lisp projects and ones where Lisp was one important tool among several, or between projects that I conceived from the start and ones where I joined someone else's.

on the other hand the person you are responding to claimed to have "implemented" a console game of great commercial success. this is somewhat disingenuous because at least according to the credits of said game, he was NOT the chief person behind its development. moreover, the one game i managed to find in which he was chief turned out to be a flop. in any event, this is still far more programming experience than i have right now, but since he likes games maybe he would appreciate knowing about the use of lisp in that domain and look up 'Game Oriented Assembly Lisp' [1]

being responsible for this 10x mess (in this thread) i want to once again explain that it was just a playful joke. the parent claimed that lisp is not useful for large teams and in response i wrote that if he employs lisp he will have no use of large teams. i think this was a funny throwback to the 10x myth. but then again tastes in humor vary more widely than preferences for computer languages!

anyway, i definitely do not think that people must use lisp, or that if they do they will magically be a super-dooper-programmer, or opposite: if they don't use lisp that they are somehow less worthy as programmers than lispers. i am definitely not a 10x programmer (nor do i try to be), but i really enjoy programming in lisp and i feel like i am more productive in it. however, what i think is important in my experience is that for quite a while i wanted to learn lisp but certain mistaken preconceived notions about it always diminished my motivation. so whenever i advocate use of lisp it is aimed at people who are of similar state of mind as i was then and i try to address those issues that held me back

[0] https://en.wikipedia.org/wiki/Argument_from_authority

[1] https://en.wikipedia.org/wiki/Game_Oriented_Assembly_Lisp

Because you need to find those 10 lisp programmers. The ratio of mediocre Java to good lisp programmers is probably more than 100:1. With only 10 specialists, attrition can become a problem quickly. It's a different optimization.

On the other hand there are virtually no mediocre Lisp programmers.

Why not?

Probably the same in APL, Coq, Prolog and Haskell etc. Weird but mathematically oriented languages that don’t easily get you jobs are persued by passionate programmers. It is like there are no mediocre hikers on the north pole!

You should see some Haskell I've written in production and you'll change your mind quick enough ;)

I've met quite a few lisp programmers over the years, about half of them mediocre -- particularly those who only knew lisp. Reason being that they learned lisp for a job, and that was just the language the job was in. I might believe that only good programmers pick up lisp on their own, but mediocre programmers follow the money.

The excellent lisp programmers I've known have been polyglots -- folks who can dig deep, know how memory works, know how cpus work, know how their lisp works under the hood. The mediocre ones just bash stuff together, oblivious of the details, and the result is no better than javascript.

I haven’t met anyone who started with Lisp, only those graybeard types who ended with Lisp after 20 years. Where do people work with Lisp at an entry level? I can imagine that code base would be a shitshow unless they have a ridiculously competent architect.

Lots of people had their first, and possibly only, programming exposure to Lisp as the scripting system for something else (i.e. Emacs, AutoCAD etc.) I'd go so far as to speculate that orders of magnitude more people learned it that way than have ever learned Lisp for the sake of learning Lisp.

Totally forgot about those, fair play!

I’m intrigued that you’ve experienced a world where people learned Lisp just to get a job. Would you please elaborate? I confess that I can’t imagine an employer that desperate for Lispers today.

My previous employer is on this list: https://github.com/azzamsa/awesome-lisp-companies

The driver is actually that the company needs people with a specific skillset, and their tooling is in lisp. As I saw elsewhere on HN today: "you can teach a biologist to program, but you can't teach a programmer biology". I don't think anybody is "desperate for lispers" unless their senior guru is recently departed.

Because such simple one-factor sentiments, even if they are true, are not how people evaluate using one thing over another. At best they might consider it in a multi-factor analysis. But with this particular sentiment it still probably won't be considered much because it's too hard to evaluate the truthiness of such a general claim -- I suspect it's more true in some cases but not so true in others. You'd be more interested in considering the specifics of language+ecosystem (not language alone) and how they relate to your problem domain.

Because MPAI.

Reply on news.ycombinator.com