Running Lisp in Production (2020)
grammarly.com
grammarly.com
A key to running Lisp in production is to step away from thinking of it as a lone-wolf bionic-suit hacker language, and to invest in developing a style guide, code review guidelines, CI/CD, and a culture of training new programmers to write Lisp. I've found that you typically don't have to hire Lisp specialists. As long as you find good programmers, they pick up Lisp just fine in just a few short weeks.
The sibling comments are right though: Lisp's reputation is unfortunately affected by so, so many myths that it actually stunts the ecosystem in certain ways. The most rampant myth these days is that (somehow) macros automatically imply doom of a codebase after a certain size or contributor count. Classic myths from decades ago, like "Lisp is interpreted and slow", still echo from time to time too. Lisp's extensive, half-century history is a strength but also clearly a (social) weakness.
A lot of myths come from good-intentioned speculation of the supposed consequences of certain features or aspects of Lisp, and not from actual experience using it.
Programming in Lisp is very different than software engineering in Lisp. Of course that's true in any language, but Lisp has a lot more "shoot your team in the foot" possibilities than C, and you absolutely need to learn to avoid those patterns or formally communicate your use of them in order to make Lisp effective on a software engineering team. Rather than depending on the language itself to enforce good software engineering discipline, with Lisp you must create out-of-band agreements with your team about how to write good software. This is not a bad thing; it's something all SWE teams should do anyway. But Lisp is especially intolerant of neglecting this aspect of SWE.
I love Lisp for its adventurous and fun nature, but the above statement is a huge negative for real engineering work on a team. Any time you can rely on your tools or language prevent these types of issues, that's a win. This became especially evident to me when my team started using Rust. Code reviews are now almost purely about logic and design.
Unlike C, the powerful (and safe) language features are a boon to software engineering projects, as long as you're doing was dreamcompiler said: software engineering and not "just programming".
I disagree with the thesis that the more restrictive one is, the better it is for software engineering. We've empirically disproved this time and time again with every supposedly "simple" language evolving way, way past its original design goals. Python, Go, and even Scheme are great examples.
Lisp: With great power comes great responsibility.
What I am talking about is having to rely on people to do the "right" things when your toolset could be enforcing many of those things. We've empirically shown time and time again that people are poor at consistently avoiding pitfalls. This is especially true as the size and complexity of a project grows.
Pretty funny, considering how much faster CL is than, say, Python -- never mind Ruby.
Lol, with the new JIT compilers and TruffleRuby, Ruby is blowing Python out of the water performance-wise. TruffleRuby approaching Nodejs speed for some tasks.
But yes, CL is fast. Faster than any other dynamic language, approaching Java speed (and Java is shockingly quick considering it has a GC).
All of that said, I would love for Ruby to usurp Python from it's thrown!
Almost as fast as Java, that's pretty speedy! It's not LuaJIT fast but it ain't half bad!
Lua is also a dynamic language, but who's counting?
One could use it as a dynamically typed interactive language with nice support for nd array manipulation, excellent support for calling C functions. Once satisfied with correctness of the implementation one could then compile it down to a statically typed C for speed. It ran circles around byte compiled Python in speed. Two of Python's relative advantages in the data-science domain were (i) nicer array manipulation syntax and (ii) one could build the non data-sciency part of the application in Python as well thus avoiding a proliferation of tooling and languages necessary.
Regarding Ruby, even TCL was faster than non JITed Ruby. That was funny because TCL used to be considered the slowest possible runtime (among professionally used runtimes)
For some languages, I could not find much of a reason to use them for such tasks. When in Java I used StringBuilder instead of simply concatenating the values. If I had to add something to the existing string in C, I used memcpy and memset. It's easy enough to understand why. So why would I do the same thing again in Lisp? For the same reasons I did in C?
These days my biggest issue is that I kind of lost the need for anything more than Bash for one-off tools, and I do it rarely enough that the incremental benefit faces big hurdle of dealing with development-antagonistic environment (mac)
The biggest hurdle has been difficulties with deployment. Open-source Lisp implementations mostly[1] only let you build an executable, and not things like a C ABI-compatible shared library. Commercial Lisp implementations are a lot better in this regard.
Even if your OK with building an executable, there are limited options in terms of "enterprise integration", especially in the arena of databases and RPC. Google only very recently released support for gRPC [2], for example, and it's not a first-class citizen in Google's gRPC offering. Most hobbyist Lisp hackers don't exactly care to build (and more importantly maintain) a bunch of open source stuff for enterprise use.
[1] There is a Lisp implementation called "ECL" that compiles Lisp to C, but it's lacking in many other dimensions.
I had a top-level pattern language with a dataflow-like visual representation, using the LispWorks UI lib to make the editor. That pattern language compiled down to a Measurement Calculus representation (one-way QC, so not the usual circuit model) in s-expr form, which I could both execute with my QVM and do optimizing transformations on. I had a CL QVM was initially CL, but it was quickly replaced with a C + OpenMP version using the same s-expr input.
[1] MPI version: https://github.com/quil-lang/qvm/tree/master/dqvm
That's a real, non-facetious answer. Another facet to the answer is that many people see value in learning a different language to help give them perspective on the existing ones they already know.
I generally don't consider it a net loss to ever learn a new language.
What are your thoughts here?
I’ve found lisps to be my favorite languages to read other people’s code in.
And the types are not just for information: the compilers can and do optimise heavily if you ask for it with `(optimise 3)`! You can get nearly as fast as C.
[1] There is something called a SATISFIES type which lets you turn any globally defined predicate function into a type, but I don't find that it's used widely in practice, because the compiler can't reason about it.
If you do use a library like Coalton, then you get a type system pretty close to Haskell's.
https://coalton-lang.github.io/20211010-introducing-coalton/
If the platform-standard tooling still works -- expected logging libs, tracing, etc -- you can use what you like to do the actual work.
For example, I worked on a Clojure project that would leverage this supposed “10x” power of lisp. Despite hiring the even key Clojure contributors, this hypothesis was falsified. The project went from around 35 to over 400 people and it is moving at a the speed of glacial ice.
Once you have a project with more than a few people a better language does not have much impact: it is all dominated by people factors and how to fit the project into its larger context.
On a fun note: The old guard of Common Lisp (like Norvig IIRC) was very fond of Python when it hit mainstream because for them it had a whole lot of the feature set that lisp had which other popular languages lacked.
[disclaimer: I am a die-hard Lisp and Clojure fan but the other fans just make me cringe at times]
on the other hand i think people can be more productive in lisp provided certain criteria are met: they are motivated to solve the problem at hand; and that they enjoy and are capable to use the tooling that is offered in lisp to its full extent. but this is not surprising because in order to understand lisp to this extent you have to understand far more about computing than just knowing lisp syntax, which is trivial btw
I am not saying that at all. If anything I said the opposite of that. I think, given how simple Lisp's syntax is, being able to write/use Lisp is almost trivial. However being able to use Lisp is a way that has given it an almost mythical status (one being discussed in this thread is the so-called "x10 programmer") requires far more than just being able to write Lisp
EDIT: see the problem i have with your post is its tone and refusal to read what i actually wrote. if you read it properly i am not saying anything different to what you seem to be saying: if you are a good programmer you can be productive in any language, including lisp. if you are a crappy programmer, knowing how to write lisp is not going to save you. knowing how to write lisp is trivial. even implementing some (minimal) dialect of lisp is not too difficult
on the other hand, i think that repl-driven development that is available to many (all?) common lisp implementations is unmatched in any language, and i think that this is unique to common lisp. please see the post by mikelevins in this thread that details this process. i think this can be a great productivity booster for production as well as exploratory programming. in any event i enjoy it a lot
… crickets …
Still waiting :)
To confess, I am probably, on a good day, 0.8x programmer. (I have known a handful of 10x programmers. And more than a few 0.1x programmers.)
My first big project was in FORTRAN II. The way that IF statements worked was really horrific, what with three statement labels. Next up was a massive project in Sigma5 assembler with interrupt driven tasks. Once we implemented co-routines, co-operative inter-process communication was a breeze, with very few multi-thread errors. I cringe when I see folks struggling with async-await hell. Prior to go, the only language that I saw that did co-routines well was Bliss-36.
Later I did a lot of C (pre-web). When I picked up C++, I ended up with fewer crashes, as I found it easier to deal with pointers to objects.
I came to Lisp later on, and am still working with it. It is a lot better than other languages for me in several ways. First, it is quite expressive. Secondly, with a slime, it is trivial to debug using break statements, ability to instantly navigate to the source of a function, be it yours or deep in the heart of e.g. SBCL. I use Lisp exclusively now (with an embarrassing amount of bash mixed in), but not because it is 10x for me. I am likely several X more productive, but in part due to the debug ability plus the repl.
Also, sbcl is really fast, even if you don't resort to hackish tricks like was done for sbcl to intimidate other languages in the mandelbrot benchmark game. Or log4cl compared with the speed of log4j.
Where some real leverage comes in with Lisp is the ability to express things that are significantly more difficult in other languages. There are things done in Lisp that are just plain impractical in other languages.
But there are some things that the programs are just about as annoying as in any other language. Look at the code dealing with http in either of the libraries drakma or dexador. Lisp ain't much help making that awful protocol go away.
And often in larger projects, minor advantages of a language compound to end up with a 2x or better result.
So this argument about a language being a miracle productivity tool is a pretty old one. I had a boss, a PhD fellow much smarter than I claim that higher-level assembler languages would make writing an operating system trivially easy. I was a bit suspicious of that then, and am quite certain of it now.
as regards this 10x business, i actually wasn't being very serious. it was just a playful response to the parent's claim that lisp is not suitable for large teams
I have toyed with the idea of using Lisp to take apart binaries.
Also, Lisp wouldn't be very good with encryption, as it doesn't do constant time very well.
I would have thought that's implementation dependent
I’m attaching an example of how quickly you can write TLS 1.3 by a single person in Common Lisp. At the time I wrote this library most websites still struggled with TLS 1.3 support. The point is that most software creators don’t have the breath and depth of knowledge required to be proficient in LISP if they only posses siloed domain knowledge.
Links:
https://github.com/mateuszb/tls1.3
If you want an example of expressivity you can take a look at EC crypto with a full NIST test vector set in about 300 lines of code
https://github.com/mateuszb/tls1.3/blob/master/elliptic-curv...
Do you happen to know of a video or article with visual examples that demonstrates this? I always hear about these things with Common Lisp, but I have personally never seen it. I’d love to understand how this works or enables people.
http://danmidwood.com/content/2014/11/21/animated-paredit.ht...
It really feels more like you're working on a tree of code than just editing text.
Although, I think there’s still something missing between those demonstrations and the quoted claims.
I have never used Paredit when doing Scheme or Racket, so maybe I should just hunker down with it at some point (I have looked into it before). I don’t use Emacs (tried it a few times), but there are several Paredit-based VSCode extensions.
A common approach is to start with an initial core. The idea is that all programs start the same, with an initial core, that contains runtime and standard library. You can test things out in a REPL, that you run side by side with a text editor. As you work things out in REPL, you can start moving them to the text file and the program starts taking form and shape. It's a much more exploration driven design and with fast structure editing of paredit you can really transplant or reorganize pieces of code really quickly. The idea is to grow the program from the initial core into something else inside-out style. It's akin to being dropped into a running program, and modifying it as it is running but that is not possible with most languages easily and as interactively as in Common Lisp for example.
With SLIME the REPL and text editor integration, you can for example write a buggy function, test it in REPL and find out about the error. Modify it in the text editor, press Ctrl-c twice, and the function is hot reloaded on the fly. No reloads, no re-imports, no-caching, no program restarts. Next time the function is called, it is replaced with a new stuff you just wrote. It really is pretty magical. It just works, and works for classes, objects, meta classes, functions, variables, etc.
The language is unlike other languages, because it offers object introspection (often mistakenly referred to as reflection) and object intercession; introspection and intercession combined give you a true reflection. It is the latter technique that supports all of the above, for an imho superior development experience. It really just works, and it works marvelously well. No yak shaving for hours on end trying to get some dependency chain to work, compiler errors, package version mismatches, etc. No need for build team, no need for testing/qa team, no need for most of your usual dev overhead we are so used to these days :)
People seem to recommend portacle https://portacle.github.io/ for a first dev environment but I usually set everything up myself. Portacle seems to have all the essentials. You can follow SLIME installation instructions from the quicklisp page of portacle doesn’t come with it.
Then I recommend slime manual at https://slime.common-lisp.dev/doc/slime.pdf if you’re going to be using Emacs.
I’m not familiar with other tools.
By day 7 you should have a feel for what people mean when they say you are directly editing the AST.
Basically: there is the same amount of friction to moving constructs around at a high level as there is at a low level, and this is due to Lisp’s uniform syntax, and paredit leverages this a lot.
Contrast this to a language like Java, where the friction between modifying a single expression and modifying class-level interactions are totally different, syntactically. Now you find yourself leaning on highly specific IDE-level support or Lombok for things like setting up Builders for every POJO, to get around these constraints.
From a Lisper’s perspective, these syntactic constraints are unnecessary, and should at most require a library that you can trivially port to any editor. Which is exactly what paredit achieves.
What I don’t understand is what I stated and what I quoted.
Without seeing it demonstrated, I just don’t understand the quoted claims. When I’ve done Scheme and Racket, I never really needed to be copying and moving ASTs (i.e., parenthetic groups) all over the place. Maybe that’s just me and my style, for better or worse.
In a somewhat vague sense, the "units of editing" are not characters.
There are some videos on YouTube demonstrating ParEdit and SLIME.
I like Lisps/Schemes as much as the next person, but other languages don’t have parentheses to deal with, so by that same token, you’re getting that for free, such as an ML like F#.
Python: [x,y,z] 1,2
Lisp: '(x y z) 1 2
Python: [x,y,z,1,2]
Lisp: '(x y z 1 2)
But I do at least see there's a nicer process if you have something like paredit: you just move you cursor to the "whatever" (even if it's instead "whatever(a,b,c)") and a command will move it to the left/right/etc. and fix up anything that needs fixing up. In Lisp though the base syntax is so simple and uniform that there's not usually much needing "fixing up" -- there's no pesky commas to deal with for instance, and having the opening paren come in front of the function name instead of after simplifies a lot of things. The worst is adding/removing/moving a form that's at the end of a let binding, or sometimes introducing a let binding, or sometimes adding something to the end of a function that previously ended with ))).
I like to use vim (which does have paredit though I have it disabled) and just having the ability to jump between open/close parens by pressing "%" and to cut jumps as a whole, or the insides, without having to move my cursor character by character, is good enough for me. I still use some paredit-like commands in some instances like moving forms around or in those "worst case issues" I mentioned but I use them with these mappings: https://github.com/tpope/vim-sexp-mappings-for-regular-peopl...
There are more advanced things but how much I care about them varies; I don't tend to need them for Lisp, though every so often I'll miss something from Eclipse that I suspect not even emacs does (or does well). e.g. I know emacs can do a "templateized" completion just like a Java IDE where you type a function name and it completes it and inserts its required arguments as placeholder variables to later define/type over, I don't know though whether emacs can then let you place the cursor over each one in turn and with something as easy as 'ctrl+1' hoist that var to an assignment form just above (I did this all the time in Eclipse to avoid having to choose a name, type it, and type its correct type). (In Lisp it's complicated by needing to introduce a let binding if it doesn't exist or append to one if it does. It wouldn't surprise me if paredit can do this, it's just that I'm aware of some refactoring tools in Slime but they don't tend to approach what Eclipse or IntelliJ users expect even if in theory they could.)
1) Lisp acquired a reputation of being associated with AI, back when that was a bad thing, lots of government-funded "expert system" boondoggles and the like. It is not considered a general purpose language, despite being one.
2) Lisp is thought to have poor library support. Unlike Java or JavaScript, which have standard libraries for every task, Lisp is thought to lack this. Quicklisp helps, but it's not as comprehensive as Maven or npm.
3) Lisp is associated with a certain programmer type: highly intelligent, persnickety, only interested in solving interesting problems. Companies prefer programmers who have only a moderate amount of technical skill but who are diligent, who will take orders and get the job done without complaint.
4) Not many programmers actually know Lisp, because not many bother to learn it. If CS students are exposed to Lisp in college, they will complain about it and won't touch it again after passing required classes that use it. This means Lisp has a much smaller qualified candidate pool and it's much harder to hire Lisp programmers.
5) Software projects, especially enterprise software projects, are optimized to require lots of programmers each of whom develop one component without stepping on each other's toes. In the enterprise, not breaking existing things is more important than building new things rapidly. This is why OO languages like Java are so well loved... they help encapsulate programmers from each other as well as encapsulating units of functionality. Each programmer poses more risk in a small team where every programmer's responsibility is larger (and every programmer could theoretically rebuild the whole system themself given enough time) than in a large team where none of the programmers have a complete understanding of the system and everybody stays in their lane.
7) Larger programming teams are just more impressive. As a manager if you have 100 direct reports instead of 10, you are a bigger deal. You can command a bigger budget, which also makes you seem more important.
So the odds are pretty stacked against Lisp. It doesn't scale well to large teams, but it lets a small team get more done. The incentives in most software development reward large teams of programmers just barely smart enough to get the job done, not small surgical teams of wizardly Lisp hackers. So Lisp remains niche, perhaps rightly so.
I won't distinguish between projects that I wrote 100 percent myself, and ones where I was a contributor on a team. I include examples of both. Similarly, I won't distinguish between pure Lisp projects and ones where Lisp was one important tool among several, or between projects that I conceived from the start and ones where I joined someone else's.
- a knowledge-based application compatibility testing system used in production by Apple to identify and report thousands of compatibility bugs in its system software
- an experimental operating system for an Apple mobile device
- a network security appliance acquired and shipped by Cisco
- a direct-manipulation-based rapid application-development system
- an embedded system controlling a forensic fingerprint-scanning device
- a consumer-focused simple list-management database
- a commercial graph database
- a framework and compiler for building mobile simulation games
- a knowledge-based control system for laser-sintering manufacturing machines
- a strategy and control system for networks of real-time sensing devices
I've omitted hobby projects and things I built only for my own use. All those above were for employers, clients, or direct sales to customers.
Although, like you, I don't believe that Lisp magically transforms anyone into a 10x programmer, I have actually been a 10x programmer (according to DVCS statistics or stakeholder reports) on some projects, but I've only managed to do that using Lisp.
To be fair, I have also been a 0.1x programmer, but never for very long. It's hard to be happy or to prosper in such a situation.
Again, I don't disagree with your main point. Lisp won't magically make someone amazing. I do think, though, that it's a powerful force multiplier for programmers who get along with it well, and that for certain people (myself among them) it's the best kind of tool.
I also don't think much of anything is proven by failing to find anyone who's done substantial work in Lisp, or who did things in Lisp that they couldn't do as easily without it, but if that did prove anything, then I guess I would be the disproof.
> I won't distinguish between projects that I wrote 100 percent myself, and ones where I was a contributor on a team. I include examples of both. Similarly, I won't distinguish between pure Lisp projects and ones where Lisp was one important tool among several, or between projects that I conceived from the start and ones where I joined someone else's.
on the other hand the person you are responding to claimed to have "implemented" a console game of great commercial success. this is somewhat disingenuous because at least according to the credits of said game, he was NOT the chief person behind its development. moreover, the one game i managed to find in which he was chief turned out to be a flop. in any event, this is still far more programming experience than i have right now, but since he likes games maybe he would appreciate knowing about the use of lisp in that domain and look up 'Game Oriented Assembly Lisp' [1]
being responsible for this 10x mess (in this thread) i want to once again explain that it was just a playful joke. the parent claimed that lisp is not useful for large teams and in response i wrote that if he employs lisp he will have no use of large teams. i think this was a funny throwback to the 10x myth. but then again tastes in humor vary more widely than preferences for computer languages!
anyway, i definitely do not think that people must use lisp, or that if they do they will magically be a super-dooper-programmer, or opposite: if they don't use lisp that they are somehow less worthy as programmers than lispers. i am definitely not a 10x programmer (nor do i try to be), but i really enjoy programming in lisp and i feel like i am more productive in it. however, what i think is important in my experience is that for quite a while i wanted to learn lisp but certain mistaken preconceived notions about it always diminished my motivation. so whenever i advocate use of lisp it is aimed at people who are of similar state of mind as i was then and i try to address those issues that held me back
[0] https://en.wikipedia.org/wiki/Argument_from_authority
[1] https://en.wikipedia.org/wiki/Game_Oriented_Assembly_Lisp
Doesn't saying stuff like this just contribute to your reasons 1-3 that boil down to perception issues, and not anything wrong fundamentally with the language itself or even its modern day practitioners? I don't disagree that Lisp doesn't scale in the sense that there's not enough Lisp programmers available to actually scale to such grand sizes of even 1000 developers right away, but as I'm sure you already know I want to say for the benefit of passersby that Common Lisp itself is object oriented, has namespaces and can support a modular or component style of development, its global memory model isn't too unlike that of the JVM prior to Java 9 (i.e. everything dynamically loaded and being introspectable and to some degree changeable (more effort in JVM)), it's been used for multi-million line projects (not sure how large the teams for those are/were though), and nothing of the recent microservices trend to scale in a different way is forbidden to Lisp...
Besides, it's not like enterprise companies that have 1000+ devs actually have a 1000+ size team with a single standup. No, things are divided into smaller teams of around 2-10 with maybe 6 being average.
I disagree somewhat with your reasons 3 and 5 and 7, having worked at an enterprise company with thousands of developers and been involved in hiring processes. Maybe I was just lucky and other enterprises are different? In my case, the implicit incentives might sometimes align in the direction you indicate, but explicitly, no, they don't. Hiring selects for stronger technical skills over weaker technical skills, interviewers prefer candidates who have a story of leadership/taking action/dealing with people socially (i.e. how do you handle such and such kind of a disagreement) over heads down and take orders and pretends there are no disagreements, no participation. Each engineer is pressured into career growth and expected to reach a certain role level after N years. Getting promoted to such a level typically involves taking on some sort of leadership role that impacts more than just your own team, and often more than one such role or a secondary role in another's project because things fall through. Despite encapsulation, things very often broke in the monolith (note only in development, production breakages were rare), and many new hires had to be trained to not sync every day if they wanted a more peaceful life. The teams that end up with a domineering management who allow no input from the development side typically don't last long because the developers switch teams (easy to do in an enterprise org where there are tons of teams with headcount every quarter) and with no team left area ownership will be dispersed. No manager has 100 direct reports. They might have 100+ indirect reports, but after a certain direct size, they have to become a higher level manager. They may still retain a couple ICs as direct reports but most of their directs change into a small number of other managers or directors, in classic org chart hierarchy.
The excellent lisp programmers I've known have been polyglots -- folks who can dig deep, know how memory works, know how cpus work, know how their lisp works under the hood. The mediocre ones just bash stuff together, oblivious of the details, and the result is no better than javascript.
The driver is actually that the company needs people with a specific skillset, and their tooling is in lisp. As I saw elsewhere on HN today: "you can teach a biologist to program, but you can't teach a programmer biology". I don't think anybody is "desperate for lispers" unless their senior guru is recently departed.
I have heard, although it is always just a quick statement with no explanation that it is the macros. And then one of siblings children comments seems to support the idea that Lisp programmers as a whole may be more experienced/better/more well rounded developers in general, so maybe that’s it. I am genuinely interested in peoples thoughts on this matter and am not being flippant or sarcastic.
One day I was thinking it would be awesome if someone created a really powerful language with great facilities for abstraction: first class functions, objects, meta objects, etc. It should use dynamic typing during development with an excellent REPL and a built in debugger available everywhere, but it should have the ability to add type declarations when the code got more settled.
Even better, this language should include ways of hinting to the compiler how to make a chunk of code faster, and if that wasn't enough, you should be able to replace chunks of code seamlessly with highly optimized C code.
I thought about how to implement a language like this for a while, but when I looked into it I discovered Common Lisp does most of this already. It's static typing isn't perfect, though now there is https://coalton-lang.github.io/ if that's your thing.
So that did it for me, but really this is only part of what you can do with Common Lisp. I think it still has a niche in places where the "right thing" isn't know yet.
For example, I read a Reddit post saying that Common Lisp would be a poor choice now for a web startup, and that Paul Graham overstates Common Lisp's case as the reason for Viaweb's success. This is probably true in 2022, but I think it's instructive to look at why Paul claimed Lisp was so helpful back in 1996. There was no Rails or Django, no React or single page web apps. Viaweb built Scheme style continuations onto Common Lisp to simulate an SPA before AJAX was a thing, and on top of that created a slick HTML templating language based off lisp macros.
To do that in Perl, or C, would have required something like writing a scheme interpreter, and would have taken much longer for a 2 person team. Is that a 100x advantage? Hard to say.
Ultimately, I do think it still has advantages, you just have to find the areas where nothing else will really do the job.
In the best case scenario, once the foundation is laid and the project has momentum, you end up with what feels like a language finely tuned to express solutions to problems in your domain.
In other languages, you may reuse useful routines but it's not as powerful. Additionally, as the dependencies are added the interactions between them can become a real headache.
What's interesting to me is that for some problems Lisp feels like a cheat code and it goes exactly the way I just described. For other problems it feels like a weird old language with a lot of parens. Perhaps I'm just not good enough at Lisp though.
When you compare modern languages, including Lisps, to something like C99, I think it's easier to see how you can be much more efficient/productive.
If you take away the similarities of Lisps and modern languages, you're mostly left with the syntax and how the syntax enables macros that set Lisps apart from other languages. But, most of your Lisp code looks very much like your other code where you're writing functions, defining structures, calling object methods, etc. so it seems a stretch that the syntax and macros alone make you much more efficient/productive.
pick 2!
But seriously, what modern languages have all these features?
- Prototype code in Jupyter. Write first as a few lines of code per cell. Then merge into functions. Focus is on 5-10 cells becoming a single function.
- When working, copy the functions back into the main Python file.
- Periodically reimport the main Python file back into Jupyter and recalc the cells currently in scope.
- Repeat...
- The main Python file ends up mostly like an API. And the Jupyter notebook as docs on how to call that API.
while i continue to use python the same is no longer true for jupyter. i think using org-babel in emacs is a much more superior experience, with the added benefit of having the whole development environment available to you
I generally work in Emacs. I'll have one or more source files open, plus at least one REPL buffer. The REPL is my communication channel with the running Lisp.
The object of the game is to teach the Lisp interactively how to be the program I want. It's already a working program; it just isn't the one I want to build. I'll use Lisp expressions to teach it, one expression at a time, how to be what I want.
The contents of the source files depend on the stage I'm at with a program. If I'm just starting then it's probably one file with a few snippets that I C-x C-e to send to the REPL to modify the dynamic environment of the running Lisp.
If it's a more mature project, then I have a bunch of source files that fit into an ASDF system (which defines sources to build and dependencies among them and any libraries they depend on). The system as a whole will be loaded into the memory of the running Lisp, converting it into a work-in-progress version of my app.
I'll have some subset of the project files open in buffers so that I can add or edit definitions and C-x C-e them into the running Lisp and see in the REPL whether my changes have the effect that I intend.
I continue this way indefinitely, teaching the running Lisp expression-by-expression the features that I intend for the finished product to have. When the difference between my idea and what the Lisp does shrinks to epsilon, the program is implemented. I run a build function and I have a delivered application.
When something goes wrong during my work, I land in a breakloop, which is something like a backtrace, except live. Rather than a printout of an unwound stack, it's a live REPL on the dynamic environment of the stack at the moment the error occurred. I can use the breakloop to wander up and down the stack, inspect and edit variable, type, and function definitions, and resume execution at the moment of the error when I'm ready to see the effect of my changes. The Lisp then runs the same function from where it broke, except that the variables, types, and functions I modified now have the definitions I just gave them, rather than the ones they originally had.
If I change the definition of some class that already has live instances, then the Lisp reinitializes those instances with the new class definition and continues. If I neglected to show it how to reinitialize those classes, it drops me into another breakloop and offers me the opportunity to tell it how to do so, after which it will continue, as before.
If I stumble across something curious and I want to show it to a collaborator in context, I can save a heap image of the running Lisp and give it to my colleague. That colleague can start the image on their machine and see the same dynamic environment that I saw. In some older Lisps, that dynamic state could include all of the windows on my screen, and their contents, and which one was the active window, and where the text-insertion point was.
I can deploy my work-in-progress apps to a staging machine with a repl server built in. I can let it run in a testing environment indefinitely, and use Emacs to connect to a live REPL that I can use to rummage around in the running app, inspect and edit variables, types, and functions, and generally do anything I would do if it were running in my normal dev environment. If I see something odd, I can again dump a heap image, copy it back to my development machine, and crawl around in its running state while I let the deployed test version go back to doing what it was doing.
There's quite a bit more to it, but with luck, that gives a general idea of what the workflow is like. Common Lisp environments are generally like this; some have richer sets of affordances than others. Smalltalks are like that, too, and so is Factor.
Most other Languages and repls are not so much like that. Clojurescript with figwheel gives you part of it, and the part it implements is pretty good.
Not the GP. Thanks for the comment, I always find your comments about your experiences with Lisp insightful... But I'm curious, does/can this lead to situations where colleagues are stepping on each others toes in the live environments?
If you're going to test that way, you want to have your own staging environment, distinct from other developers', just like you don't want two different people's unit tests writing to the same test data as they run their tests (unless that kind of race condition is exactly what you're testing, of course).
What do you prefer about using C-x C-e on individual sexps rather than reloading everything and keeping the entire program synced between source files and the running environment?
Moreover, if I recompile everything, then I'm rebuilding my in-memory state from scratch. Mostly, that's not what I want. Mostly I work like a sculptor: I make a rough approximation and then refine it stepwise. I don't want to rebuild from scratch every time I discover something new. I want to keep most of what I've built and change just what needs to be changed.
That said, sometimes a discovery does call for a larger-scale reconstruction. That's when I reload a whole file or, in more extreme cases, reload an entire ASDF system of files. Once in a while, I change enough (or make enough mistakes) that it makes sense to blow the whole thing away--kill the Lisp, restart it, and load the system from scratch. But that's fairly rare.
Mostly I want to plop down my clay and tweak it a little at a time toward my goal, discovering and refining it a little at a time.
- launch the program you want to run and have it running in the background
- text editor open
- bits of code in editor loaded in the running program through text selection and sending them to the be loaded (the most common scenario is doing this in Emacs)
However my concern with this is... if you don't run the entire "code frozen" program at once, how can you guarantee the internal consistency of the program if over time you kind of haphazardly add bits of code to it? Maybe you loaded that function, maybe you didn't, did you add that dependency function in the correct order and at the correct "version" (where version is used loosely, every time you type in something, it's a new "version").
I'd really love for someone experienced with Lisp to describe their workflow and also maybe clarify how they handle this (in my opinion, huge) problem of possible program inconsistent states.
Between such reloads you might end up in inconsistent states, but you benefit from fast iteration:
My workflow is to update functions in the source code, send to to the repl, quickly test them in the repl, and once done, I save, commit and push the code. CI then runs the test suite from the source files.
I'm a fan of JRebel for improving efficiencies in Java development, in one of their marketing materials (https://www.jrebel.com/sites/rebel/files/pdfs/rw-mythbusters...) they claim that waiting on redeploys takes up about 20% of a Java developer's salaried time. So with naive assumptions if you had a team of 100 developers, you could fire 20 of them and maintain the same pace just by forcing the remaining 80 to use JRebel. Or just keep all 100 but now everyone's got a 20% efficiency boost and can do 20% more than before! (Yadda yadda real world caveats.)
For Common Lisp, this "no waiting for redeploys" is part of the language, you've been able to enjoy such a feature for decades without having to wait for JRebel to come into existence/figure out how to add it to a language yourself (some game studios accomplish approximations with their custom C++ engines), so it's a possible efficiency gain over languages that don't have a way to balance it -- like with JRebel which comes close, or to a lesser degree by more disciplined unit testing that lets you confidently and accurately make changes without needing to redeploy the whole thing for verification, or writing in a more purely functional style where you don't additionally suffer the burden of having to rebuild up tons of state after a redeploy, or to another degree by solving everything with an advanced type system, or just by having simple problems where you need integer-factor efficiency differences for them to matter for the duration you'll be dealing with the problem...
CL has a bunch of other stuff built-in that can lead to efficiency gains too; macros of course are a big part of that but not the only other part. I'll list a few examples of macros being useful. There's a macro that's been around in the ecosystem since the 90s that lets you write things in infix notation, so you can write your #I((-b + sqrt(b^^2 - 4*a*c))/(2*a)) and not have to live with (/ (+ (- b) (sqrt (- (expt b 2) (* 4 a c)))) (* 2 a)). (In any case negative sqrt will give you a complex number automatically, which is nice.) The built in "loop" macro for looping has some very advanced features on top of the more common for-i-.. style or for-each.. style of other langs, e.g. from Practical Common Lisp:
(defparameter *random* (loop repeat 100 collect (random 10000)))
(loop for i in *random*
counting (evenp i) into evens
counting (oddp i) into odds
summing i into total
maximizing i into max
minimizing i into min
finally (return (list min max total evens odds)))
Hopefully at least I've given you an idea of how it's at least possible for there to be any efficiency gains using Lisp, either in the past before other languages and their ecosystems caught up, or even now for certain classes of problems (e.g. if you believe 20% efficiency gain is possible by eliminating waiting on redeploys while developing). Though I think marginal efficiencies that sum up to 10x are probably pretty rare nowadays. Such a general claim was more tenable in the past when even garbage collection wasn't really mainstream...
LISP in all caps refers to classic Lisp, and was once widely used for the family: all kinds of dialects and implementations used LISP in their names, even Common Lisp ones. "Common LISP" was a thing; e.g. in the title of the book Common LISPcraft by Robert Wilensky.
This situation was similar to how other languages were capitalized: FORTRAN, COBOL, BASIC. I suspect part of the reason for all the shouting was the limited support for lower case characters in some printing/terminal equipment. Many of the names involved were acronyms, or near acronyms.
Today, and for probably at least three decades, those languages have been called Fortran, Cobol, Basic (e.g. Microsoft Visual Basic, not Microsoft Visual BASIC). It's the same situation with Lisp.
But that's just like most debates over words and changing common usage meanings. It doesn't even matter if the new common usage is or becomes the most popular usage or not, at some point it's common enough that a new entry in any dictionary is warranted. (The term "Lisp" obviously predates Common Lisp, but part of the success of Common Lisp led to "Lisp" gaining common usage to refer just to CL.) It's unfortunate that the alternate usage isn't something crisp, like how "literally" can be taken to mean something not literally true and is just an exaggeration or emphasis, and instead a common usage of "Lisp/lisp/a lisp/Lisp family/Lisp dialect" is as a poorly defined fuzzy feeling of the spirit or flavor of "Lisp-like" languages. It can be a mess trying to pin down whatever commonalities are meant by that feeling. My favorite attempt is something like a language written in its own literal data structures. But depending on the speaker the feeling might encompass things like WebAssembly (s-exps sorta!) or Dylan or even Julia, and in the past has once encompassed things like Ruby or Python (in the sense of "acceptable lisp").
For me and many others (like the title of this thread) unqualified Lisp still typically means Common Lisp, it's still a common enough usage, and I'll at least keep using it that way almost all the time for the foreseeable future just as others avoid using "literally" as "figuratively" or "virtually". But I don't think it's worthwhile to do so from a "fight the good fight" standpoint or to be consumed by a melancholic desire for a rectification of names. On the brighter side, at least for Lisp if there's communication confusion it's probably only a good thing in producing cognitive dissonance and driving some curiosity towards CL. (For example if someone talks about unqualified Lisp in conjunction with the form of GOTO that CL has which has been there since LISP 1.5, or in conjunction with OOP, or the function compile-file, someone reading with a particular fuzzy usage of Lisp in mind may think "I thought Lisp was a functional/minimal/interpreted language?" and go on to learn something. Similar positive learning benefits can come from using the fuzzy meaning of Lisp in e.g. setting up events or community spaces, where Schemers or Clojurists or whoever are welcome, since that gives more people in those other languages as well as the fuzzy-Lisp-curious a chance to learn about CL, and CL people a chance to learn about what's going on outside the bubble.)
Since when? Lisp is, and has almost always been, a family of languages.
Generally I've seen capitalized LISP used only to refer to the original language, from McCarthy's lab (e.g., versions 1 and 1.5). By the time Interlisp landed in the late 60's, the all-caps convention has already begun to slip.
That's not to disagree with your own experiences: I just suspect they are "regional", so to speak, and not reflective of the larger Lisp community.
yeah i think there is definitely a component of that which led to my claim. i think it is better that 'lisp' should refer to a whole family of languages when context is not obvious. even when context is clear it is helpful to distinguish what lisp one is talking about in order to avoid ambiguity
You write DSL, you write book!
But I guess if they're still updating this post (it mentions November 2020) then they must still be using Common Lisp.
This is why the majority don't see benefit in using it, 95% of devs are not solving difficult problems.
This problem is unlikely to be solved in any other language.
This whole HN post is about an NLP system that seems hitherto unrivaled.
So what I’d like to know, is would a modern “AI” approach (ML/DL) based on literature samples as input, as opposed to classical AI sitting atop an expert system/linguistic knowledge, fair any better if you were starting, say, Grammarly today?
That said, I enjoyed the article and it’s nice to have an example of Lisp being used in prod. My only experience of it is from using elisp.
What are the differences between elisp and lisp?
common lisp is a general purpose programming language with a well specified language and standard library. Elisp is intricately tied to emacs.
Notable differences are , till recently(?), elisp has only dynamic scoping for variables. Common lisp has, by default, lexical scoping for variables (while also allowing dynamic scoping). Common lisp has much better support for closures than elisp. Common lisp has very mature commercial and open source compilers that produce very efficient code.
By the way, Elixir does have macros.
I agree it's nice when something as common today as JSON is built in to a language. But for something like Common Lisp, whose roots are now nearly 40 years old, the fact it can easily support JSON literals is nothing short of amazing.
Wouldn't using the REPL during dev remedy the slow compilation issue?
These days I use LispWorks, although I tend to separate GUI code from back-end code and make sure the latter is portable across implementations. Running tests on SBCL also catches some bugs at compile time which is nice.
I think its usage has declined a bit since development of CCL has slowed down.
What’s been your experience with the debugger and integrated repl?
> "SQL, Lisp, and Haskell are the only programming languages that I've seen where one spends more time thinking than typing." - Philip Greenspun
> "I have heard more than one LISP advocate state such subjective comments as, "LISP is the most powerful and elegant programming language in the world" and expect such comments to be taken as objective truth. I have never heard a Java, C++, C, Perl, or Python advocate make the same claim about their own language of choice." - A guy on Slashdot.
"One can even conjecture that Lisp owes its survival specifically to the fact that its programs are lists, which everyone, including me, has regarded as a disadvantage." - John McCarthy, "Early History of Lisp"
"Some may say Ruby is a bad rip-off of Lisp or Smalltalk, and I admit that. But it is nicer to ordinary people." - Matz, LL2
I could keep going but most of the quotes on that page are gold.
To answer your question, though, you may need to write some programs in Lisp yourself. I recommend Racket (less decisions and setup than CL-just use DrRacket.)
- simple syntax
- extensible with macros that mesh well with the language
- tend to be expression-oriented, which some people favor
- usually have pleasant functional programming features
- usually have well-thought-out semantics
Of course, I personally don't think programming in Lisp has much to do with wielding lambda calculus in any superlatively direct sense.
We also use Python, MatLab, shell and bat scripts, C#, and C++. I work in all of them, but I use Common Lisp for whatever I can.
Edit: response to child: The most obvious thing is a lack of TCO. But there are plenty of other things like the limitations of a JVM based object system compared to the MOP. And then there’s the Lisp Machine heritage that shows through in forms like EVAl-WHEN. There’s the condition/restart system too. That’s just a few things off the top of my head. But in no way am I denigrating Clojure. It’s a good Lisp that meets important needs and I’d happily use it anytime it’s right tool.
TCO is a bit more general. Any tail call then is just a kind of jump.
To be fair, that kind of support is hard to come by outside Common Lisp and Smalltalk.
I miss Common Lisp when working in Clojure, but not vice versa. Clojurescript with figwheel is a pretty nice, pretty interactive way to develop web apps, though.
Edit: it literally asks for key logging permission on Mac OS.
https://www.reddit.com/r/discordapp/comments/hdahy1/any_good...
Is it true discord keylogs outside of its app?
Imagine a blog post titled "Using C in production"; it sounds absurd.
Even top universities didn't require it. Georgia Tech, one of the top technical universities in the US, did something like a 2-year stint with a mandatory SICP course before dropping it in favor of Python (IIRC), and had no mandatory Lisp course besides that (though they did have a mandatory Smalltalk course at the time).