Common Lisp Resources | Hacker News Reader

Common Lisp Resources

lisp-journey.gitlab.io

200 pointsoumua_don17113 comments

Fun fact I learnt the other day browsing some historic posts:

https://en.m.wikipedia.org/wiki/Dynamic_Analysis_and_Replann...

Written in Lisp at the time and widely successful to the point it paid back many times over the investment DARPA put into the language

Now the question for anyone who may know, are descendant systems still being written in Lisp?

I don't know about that one, but it wouldn't surprise me either way (still in Lisp or reimplemented/replaced by something else) since such things seem to be hit-and-miss and unless you do some digging it's probably unlikely many programmers of such systems check HN... But for a comparatively old government project in Common Lisp, there's at least one, SPIKE (https://www.stsci.edu/scientific-community/software/spike), that was developed for planning and scheduling the Hubble telescope (and other projects) and is still in use for the JWST (https://www.stsci.edu/contents/newsletters/2018-volume-35-is...).

Awesome, thanks for the links!

Automated planning is computationally intensive, so you'd want to reimplement any such system using a low-level language, with easier support for parallel and/or distributed compute.

They are also cognitively intensive, so it may be worth it to code them in slower but more expressive language.

What does "low level" even mean here? Lisp is second only to Fortran in age, so it certainly runs bare metal if that's what you want.

Like StarLisp.

Or...Common Lisp, most of the implementations of which can take full advantage of modern SMP cores.

minus the garbage collector, which for most CL implementations does not or only in limited ways benefit from multiple cores

How about reimplementing it in assembler? There's an automated tool that does that, it's called a compiler, and it comes in the box in most implementations of Common Lisp.

Lisp is quite good for that, really. Numerical computation, parallelism, etc. With a good compiler, using unboxed values and primitive arithmetic and Boolean operations on ints, most code will compile to not much slower than the C version.

For example, I just saw an old paper [1] where the author implements dynamic compilation from Lisp, to Lisp, in Lisp, for parsers. One describes the language to be parsed, in a high and abstract level in a DSL in Lisp. The Lisp program then compiles that definition into to a low-level sequence of unboxed boolean and arithmetic ops and conditionals in Lisp which, being primitives with direct machine code equivalents, the Lisp compiler can translate into reasonable machine code. The end result was probably only a few times slower than doing the recognizers directly in C.

Not so different from the usual parser generator technique. But the nice part of doing it that way is that you retain the full Lisp system; your generated blobs of "machine-level Lisp" are still just another Lisp object, so you can take your generated functions and single-step it in interpretation, or transform it further. I suspect that's a big part of why Lisp was so popular for so long. Other languages provide similar capabilities now, though sometimes not so general in ability. But for a long time, this sort of meta-programming where you write the program at the high level to generate the code at the low level for you, was basically a proprietary superpower for Lisp.

[1] https://www.plover.com/~mjd/misc/hbaker-archive/Prag-Parse.h...

I've heard CL is famous for having advanced libraries and language features decades before other languages get them.

What are some interesting libraries or language features developed since 2005 that haven't yet made their way to other languages? Not a simple "Do X" library that a C programmer could easily copy, but something more like CLOS, contracts, or pattern matching that are cross-cutting.

Coalton comes to mind: https://github.com/coalton-lang/coalton

It adds static typing to Common Lisp, while (I believe) still allowing one to escape to the dynamic world when needed. Sure, there are statically-typed languages, but I don't know of another dynamic language in which something like Coalton has been done.

Its not very smart to add static typing to Lisp, goes against the strengths of the language. At that point just program in another language, there’s a lot of benefit of static typing that IDEs can take advantage of but for anyone half decent at Lisp, they wouldn’t feel any need for static typing (outside of compiler optimisations, which is a different thing).

Now in terms of interesting things, check out European Lisp Symposium https://european-lisp-symposium.org/

The talks from this year were highly entertaining

The other really cool thing in CL IMO is Allegro Cache - in memory datastore that is a joy to use

> Its not very smart to add static typing to Lisp [...] (outside of compiler optimisations, which is a different thing)

i think this is precisely the point of coalton. cl allows quite handy compiler optimizations and having the option to drop into a native statically typed environment just gives more bang to your lisp ;)

coalton is not lispy, sure, but neither is the loop macro, yet it is probably the most powerful looping mechanism in any language :)

Those already exist and have been used by many.

Coalton is a set of macros to make it easier for to write with the safety rails of static typing.

Now if I write in Swift, for example, static typing is great, because the IDE can give a lot of hints during development. Non lisp languages have quite idiosyncratic syntaxes for various constructs, so static typing really does help a lot there. In large code bases in those languages it also helps a lot because they do not have the power of Lisp macros to reduce syntactical complexity via a Domain Specific Language.

Now with Common Lisp, with a powerful generic functions and object system, with runtime dispatch and introspection that far exceeds other languages, plus with the abstraction power of macros, trying to add static typing guard rails simply doesn’t make any sense. I rather program in TypeScript if I am so far removed from the core essence of CL development.

Each language has its strengths and the dynamic nature of CL is one of its biggest strengths (with the lack of a GUI and interop with other tools being its largest deficiencies today). Static typing in Lisp is like trying to fit a square plug in a round hole. If somebody wants so much guard rails, other languages are much more suitable because they have been designed (quite successfully if I may add) in that way.

TLDR; As per On Lisp by PG, lisp code should be built in layers over each other and hence static typing is unnecessary. Furthermore generic dispatch provides similar features to some extent (although i recommend combining with closer mop and catching edge cases etc)

> Static typing in Lisp is like trying to fit a square plug in a round hole. If somebody wants so much guard rails, other languages are much more suitable because they have been designed (quite successfully if I may add) in that way.

i belive the author of the coalton package is not very concerned with IDEs or putting up safety nets for developers. i think he uses coalton for making optimizing compilers. think of it as cl+ml not as cl+ts

EDIT: typos

> lisp code should be built in layers over each other and hence static typing is unnecessary

It could help for refactoring though.

You bring a good point about the loop macro (which I do like too). Maybe you are right. My personal feeling is trying to do static typing will lead to less elegant code (I say less elegant, because any code that works and is fast enough is good enough - style doesn’t matter that much, esp if readability is somewhat maintained) and has more far reaching implications on a code base than the loop macro which can be isolated to a specific section.

But I get your point too, for those who crave static typing, then at least there’s something for them to try.

Java's type system is just powerful enough that sometimes you can approach a problem in a way that resembles a different paradigm than normal, i.e. you solve it with the type system itself much more so than what's expressed by particular procedural or object-oriented or declarative code along the way. Supposedly Haskell encourages this approach as its primary one (and the approach sometimes gets called the "functional programming paradigm" but that I think is a misnomer as FP is primarily about immutability and higher order functions, not static types), to the same extent Java otherwise encourages a Java-style-OOP approach to most things, but I'm too unfamiliar with Haskell to make this claim with high confidence.

Still, different paradigms are useful, and one of the best things about Lisp is you aren't forced into one, and there's no threat of one being taken away from you. Coalton to me seems nice for people who like what it provides, it's nice that it exists, even though I don't see myself using it anytime soon since I don't personally enjoy working in that paradigm. There are two other non-type-proofs-paradigm benefits of static typing, but I don't really miss them in CL. The first is related to automatic refactoring tools, but Lisp has features to ease refactoring to compensate, and text-search methods for renaming aren't that bad (and even in Java necessary if you want to make sure you haven't missed reflection calls, though even then you can miss some) and anyway one can read the second edition of the Refactoring book that uses JavaScript if one needs convincing that refactoring can be done just fine even in a lower quality dynamic lang. The second is related to trivial conveniences like compile-time typo protection on symbol names, interface mismatches (like swapped function call argument order -- but failing to that indicates I have an unintuitive interface, and CL has many tools to help me make it better the simplest being keyword arguments), and faster assembly code. But SBCL's handling of the standard CL type system is good enough to get all of those things to a pleasing degree.

Great comment, agree a lot with this

I'd add one other feature of Haskell- and ML-style type systems that I like: they provide clear, succinct, convenient ways to precisely describe data structures.

Common Lisp is far and away my favorite surviving programming language, but I like Haskell and ML fairly well. I mean, when I've been paid to work in them, I definitely missed working in Lisp, but still, working in Haskell and ML is not torture. I liked it quite a bit better than working in some other languages.

One of the things I liked was the ability to succinctly express in some detail exactly what I intend some data structure to be.

I've even used Haskell as a data-description language to work out some data structure that I then implemented in Lisp.

I keep intending to do something with Coalton for this reason alone.

> coalton is not lispy, sure, but neither is the loop macro, yet it is probably the most powerful looping mechanism in any language :)

I would argue that iterate is more powerful as well as being more lispy in the sense that it's extensible and macro-programmable.

https://iterate.common-lisp.dev/

Iterate also breaks the count function. Or did the last time I tried to use it.

What do you mean?

Just checked and this is still a problem. Try this code with Iterate:

  (let ((list '(1 2 3 4 1 2 3 4))) ;; or whatever other sequence
    (iter (for i from 1 to 10)
          (sum (count i list))))

It will error out on you. The same works fine with loop:

  (let ((list '(1 2 3 4 1 2 3 4)))
    (loop for i from 1 to 10
          sum (count i list))) ;; => 8

Now, that's a bit contrived as an example, but an iteration library should not break a standard library function call.

https://gitlab.common-lisp.net/iterate/iterate/-/issues/12 - Apparently it's known and they intend to remove it in 2.0, which isn't out yet.

good to know, thanks

I like the lispiness and features of ITERATE but it breaks source location for debugging errors in SBCL with SLIME. Instead the ITER form is indicated, which dampens my enthusiasm for it somewhat.

  (defpackage :test
    (:use :cl :iterate))

  (in-package :test)

  (defun test-iter ()
    (declare (optimize (debug 3) (safety 3) (speed 0)))
    (iter:iter   ;;; <-- highlights whole ITER form
      (sleep 2)
      (error "test")))

  (defun test-loop ()
    (declare (optimize (debug 3) (safety 3) (speed 0)))
    (loop
      :do (sleep 2)
      :do (error "test")))   ;;; <-- highlights ERROR form

Did they record the talks this year?

I think so, try asking on reddit or IRC if you still can’t find them via some google fu

Thanks for sharing, nice to see SisCog still at it.

Racket. With typed racket.

In VSCode when my cursor is inside an s-exp, i can press Alt+Shift+E and it instantaneously shows the result of that call inline in the editor.

If i change the code at all, i just press the same key combo and see the new result. It remembers all the state of what i've done up to that point too so i don't need to mess around setting up the call with data to feed in for example.

The closest i can find to replicating this is the various hot-reload implementations in other languages but they all incur some kind of delay (naturally, because they're all doing a lot more work behind the scenes) which ruins the use of this feature, it stops you using it to evaluate your thought process in realtime. They also don't render the results inline - mostly because the syntax of JS/TS and other languages doesn't really lend itself to this kind of usage.

Jupyter notebooks manage to replicate a somewhat similar experience.

Hmmm, somewhat but it’s missing the ergonomics to the point that it can’t be used the same way.

The best analogue i can come up with is a sql session connected to a live database.

It is a nice approximation, but fails short of the whole experience.

Imagine that your shell was like those notebooks, and not only you can interact with the whole OS, you can even change existing notebooks dynamically by reloading code that they depend on, e.g. a displayed image gets a new effect added to it.

What extensions are you using for this?

https://github.com/nobody-famous/alive

In Emacs you can execute sexps anywhere with C-x C-e (eval-last-sexp) it's common in all classic environment treat code as a live thing, text as a live thing. A feature lost in most modern software.

Might be not exactly what you ask about, but, REPL-driven development. (Though, to be more accurate, it is more of image based development, rather than REPL, which is quite big difference) Function does not work? Change it , re-evaluate, and call the function again. No need for constant restarting, re-compiling or so. All on-the-fly, Once you get used to it, it's really something else. Its simply unique, also without sacrificing performance (Because it is quite performant)

Doesn't everyone in data science do this too with Python/R/Julia? I don't think that it's a unique Common Lisp feature.

Not quite. For example, languages like Python typically have generative class definitions -- meaning that a class definition always creates a new type, with no existing object instances being objects of that class. In comparison, Common Lisp has provisions for adapting old instances to match a new definition of an already existing class, including lazy updates. Taken to extreme, this allows for such things as large on-disk object databases (since presumably you don't want to convert a billion objects on the disk the moment you add a field to a class).

The erlang language provides a similar facility for replacing code and updating object state without ever bringing the program to a halt.

The main difference is that Erlang is very strict about the kind of programs it will run (purely functional green-thread-esque processes and OO systems that utilize message passing and code that is modified must be swapped in/out at strictly defined points in program lifecycle), whereas CL is a fully mutable environment where modification can happen at any time during program lifecycle. Much more powerful, which comes with benefits as much as it comes with the amount of guns to blow your feet off with.

In Python and Julia in most of the cases you have to restart ; don't know about R. In Common Lisp you can change a function that is running a loop while the loop is executing and you will see the result immediately in the next iteration after the change. As mentioned the closest thing with modern languages is JS in Chrome.

When is it useful to have a loop that executes partially with some code and the rest of the way with different code?

First example that comes to my mind: you have a game loop running and you would like to change the way the scene is rendered without restarting the game loop altogether and/or losing the current state of the game.

Yep, this is definitely useful. I've done this kind of work. It's much faster to build an interactive world if you can tweak everything about it while it runs and see the results instantly.

The same goes for some simulations I've worked on (and am working on). It's much faster and easier to get good results if you can see the effects of your changes in real time, as the simulation runs.

It's useful if you have some loop running and you want to stop it for some reason in an orderly fashion. Say you were processing several years of data but now want to stop when the first year is completed for some reason. Maybe it's taking too long and you'll rather keep working with partial results or whatever. You can easily recompile the processing function to raise an error when it reaches the stopping point.

Or maybe you noticed that the loop will eventually hit an edge case which is not properly handled currently in the code. You can fix it while the loop gets there.

And of course for GUI programming. You can redefine interface behaviours on the fly.

I try to explain diffirences with Python here: https://lisp-journey.gitlab.io/pythonvslisp/ + there is more in this thread.

Variations (to various degrees) are available in plenty of environments.

Most powerful one today by combination is actually JS inside Chrome, you naturally have the REPL that can do things.

But REPL's are good for the first prototyping whilst later on you might have modules (CL might make things inspectable?) that try to hide things (think JS/TS modules), Chrome however is pretty damn good at runtime detecting more or less incremental updates so you can just rewrite code and re-save the source files and Chrome will replace the function bodies in-situ without any state rebuilding or reloading being necessary.

Of more conventional languages, MSVC has had C/C++ Edit-Continue (as long as you didn't change type/function signatures) since ca 1998, it's a descendant of this that MS was hyping last year with Hot-Reloading for C# with body replacement (on top of the interactive shell("REPL") that has been in Visual Studio for years).

Especially the conventional ones might not be as flexible as the Lisp ones(but handle many practical cases fairly well) but all dynamic "scripting" languages like JS/Python/Lua,etc has been able to do most repl-y things fairly well from day 1.

The Chrome console is only useful if your code doesn't get bundled, which is rare

Those REPLs aren't even remotely comparable. You can TEST with them, but you never DEVELOP with them.

If they were the same, you'd load up a repl of your entire codebase. Now you'd dynamically update the part you're working on, test it live against your current environment, and finally save it to move to another thing. When errors happen, you just drop into the debugger to find the issue. If you missed a call site or two, you'll probably catch it here. Because it's instantaneous, it's not painful. I'll have a REPL open pretty much all the time when messing around with CL.

In JS (my day job), this wouldn't work at present if for no other reason than directly replacing a function is impossible. You'd create a second instance of a function, but all the old references from other places in the code would remain stuck with the old version.

JS falls back to "hot reloading" which is literally "restart the whole JIT" which means that you have to warm the JIT up again and (more importantly) that all your state disappears.

You make a change then wait a while for everything to recompile and run. Then you have to write/execute a bunch of code to get your state back to where it was. When the error is thrown, you now have to look through the things for that call site you missed, save, and reload everything all over again. Rinse and repeat. This heavily disincentivizes interactive and incremental development in favor of massive changes then trying everything out at once.

I wonder how Clojure(script) deals with this. Do you know if Clojurescript restarts the entire JIT as well? Is it the same in Clojure with Java?

Clojure and ClojureScript don't have the comprehensive support for livecoding that Common Lisp and Smalltalk do.

As an example, consider what happens if Common Lisp's UPDATE-INSTANCE-FOR-REDEFINED-CLASS, mentioned in my comment above, is called without a method definition for the redefined class in question. Lisp drops you into a breakloop by calling BREAK.

BREAK is another feature of Common Lisp defined by the language standard (http://clhs.lisp.se/Body/f_break.htm#break). It puts you in a repl that has the full language and runtime available, but also exists inside the dynamic environment of the control path where BREAK was called.

In this example, that means that we can see the lexical environment of the call to UPDATE-INSTANCE-FOR-REDEFINED-CLASS, along with everything that was on the stack when it was called. We can see the instance that needs to be redefined and all the definitions that are currently in effect. Using the features provided by the breakloop, we can interactively define the missing method on UPDATE-INSTANCE-FOR-REDEFINED-CLASS to properly reinitialize the instance in question, then activate a RESTART (another standard feature of the language) that resumes execution of the function, but now in a dynamic environment that includes the new method and any other changes we made in the breakloop.

The original control path that landed us in the breakloop can now complete successfully, referencing a properly reinitialized instance. The method on UPDATE-INSTANCE-FOR-REDEFINED-CLASS is defined now, so future references to previously-defined instances of the redefined class will be transparently reinitialized and behave as if they always had the new definition.

Clojure and ClojureScript don't have the runtime support to do any of this. If you redefine a class that has existing instances, and if those instances aren't dropped on the floor when your new definition is hot reloaded, then the old instances don't reflect the new class definition, and so code that relies on it is wrong. When a wrong access occurs, you are dropped--not into an interactive breakloop--but into a noninteractive printed stack trace.

The debugging features of modern browsers give you some parts of what a Lisp (or Smalltalk) system does--for example, some tools can watch the dynamic state of variables and enable you to edit them. There are tools available for ClojureScript that give you access to some of these features.

Even in the Clojurescript ecosystem, however, the full suite of livecoding tools offered by Common Lisp and Smalltalk (and mandated by the Common Lisp standard) is not available.

Lisp is so cool, I wish every language could do these kinds of gymnastics. Thank you for sharing.

Doesn't the ability to "hot reload" in either CL or Node largely depend on how the application is structured? For example, if you were developing a web server that sat in an infinite loop waiting for requests to come in, how would you go about redefining a function referenced inside that loop? Wouldn't there need to be two threads of execution? One that was the web-server and a second that acted as the REPL? To redefine something in the web-server, the two threads would have to synchronize in order for the new definition to be passed off. Couldn't this same setup be implemented in any language that supports threads and some type of `eval` function?

Edit: The two threads of execution I mentioned above are likely what the combination of SLIME + Swank are doing.

I have done this exact thing in CL before. What you do is have your event loop function just be a loop calling another function, where all the event handling actually happens. When the event handler function is recompiled, the new version will run /the next time the function is called./

If the recompiled function is currently running that stack frame and anything deeper won't be affected for that call.

The only thing you can't do is redefine the loop function itself since it never stops running. That's why you factor it so there's no reason to change that function, just the functions it repeatedly calls.

Hope that explanation made sense.

Edit: And yes, the way this is implemented under the hood is that the repl has its own thread. You could do this even in C. In fact people have done this for the Linux kernel for example(at least for binary patches). The great thing in CL is the language is designed with the functionality in mind, and the dev tools just support it, so you get it for free.

Incidentally, this is nearly identical on top to the pattern Erlang uses for the same reason, where most processes couple a long-lived “behavior” with a callback module with a particular structure and a pure-data state value threaded through everything. When the module is replaced, the new functions are picked up automatically, and there's also a callback for adjusting the state information on code changes if need be.

The underlying “system message requesting your process to ensure it picks up the code reload” part is quite different, though!

Others have answered the question but the reason this works in CL is rather simple: Function calls (by default) have a single level of indirection. When CL calls a function it looks up the address of the function in a table (the symbol table), then jumps to the given address. If you redefine the function associated with the symbol, the next call gets the new definition. Yes, you need a separate thread (native or green) for the REPL but it has less to do with the REPL being a separate thread and more to do with the symbol table being a global data structure.

This is why there is no "linking" step in Common Lisp compilation: CL keeps the symbol table around at runtime; C just throws it away.

Note, however, that a CL compiler is allowed to assume that calls to functions defined in the same file are to that definition (so it can short circuit the function lookup.) This is from the CLHS section 3.2.2.3. http://clhs.lisp.se/Body/03_bbc.htm

"All conforming programs must obey the following constraints, which are designed to minimize the observable differences between compiled and interpreted programs:"

[...]

"A call within a file to a named function that is defined in the same file refers to that function, unless that function has been declared notinline. The consequences are unspecified if functions are redefined individually at run time or multiply defined in the same file."

I suspect this has to do with deferring warnings from forward definitions, but I might be wrong. It's still common behavior for REPL redefinition to have an immediate effect on the next function call unless that function was declared inline. (Regardless of what file the function was defined in.)

A few years back I did a presentation on this very topic. Some people have said they appreciated it so I'm sharing a link here in the hope someone find it useful: https://www.youtube.com/watch?v=bl8jQ2wRh6k

What Lisp and Smalltalk have that still sets them apart is that the language runtime is designed to provide comprehensive support for livecoding.

Take, for example, Common Lisp's UPDATE-INSTANCE-FOR-REDEFINED-CLASS (http://clhs.lisp.se/Body/f_upda_1.htm). If you redefine a class in a running program, Lisp notices and the next time control touches an existing instance of the class, UPDATE-INSTANCE-FOR-REDEFINED-CLASS is automatically called to reinitialize the instance to reflect the new definition.

That's not all there is to it, of course. For example, how does Lisp know how to reinitialize the instances for the new definition? The Common Lisp standard also defines a complete suite of features to answer that question.

Smalltalk provides equivalent features.

Lots of languages have repls. Some of them support hot reloading. These are not the same kind of thing as automatically noticing that a definition has changed, and that implies that any time you touch a previously-created instance of that definition the runtime needs to automatically catch it and provide facilities (interactive when needed) to dynamically fix up the data to conform to the new definition.

Why would any language define such features? To support a specific style of programming, one in which the program is already running, it just doesn't have all the features you want it to have--so you tell it how to add them, while it runs, and test each new feature immediately, as soon as you've told the program how to do it.

Very few languages support the comprehensive whole-system approach to livecoding that Common Lisp and Smalltalk do. These are not new features, though; they've been around for decades.

I've tried some of those, and nothing comes to close to Clojure / LISP repl.

Just hitting a shortcut and getting the code executed using everything loaded into the REPL and getting inline results is great.

It helps that Clojure only has expressions and no statements as well, so every form can be run and it will return something.

Here's what I mean, this is me doing 2021-Day2 AoC puzzle part 1. WHen this pops up

    => foo

Next to a line, it's because I've hit CTRL+Enter on that line to evaluate the form.

https://www.youtube.com/watch?v=5HHLT2_a1tI

(You'll need to bump to 1080 resolution otherwise its probably unreadable)

My day job is C# and JS, I can't get anywhere close to this interactivity with my code. As the complexity of the code grows this becomes even more powerful.

Yes, this exactly. If you are doing k8s-based development, where your Lisp code is participating in k8s-hosted service communication, nothing beats the CL development experience, where you just connect Emacs to your image via SLY/SLIME and make live changes in the kerbernetes cluster, but your source files are local.

People are talking about REPL development, and they are right, but one of the best things that you can do with CL that you can't do with other REPLs, such as Clojure, is... at least using Slime in Emacs, probably other editors too... when there's an error/exception or whatever, you drop right into the stack at the error location. You can see all the values of the variables. You can evaluate expressions. Travel up the stack and do the same things. That is SO powerful, and can save so much time debugging simple errors.

You can also edit the values and change other definitions while you're roaming the stack.

Smalltalk can do it, too.

Whilst it is good the browser side windows provide also quite a lot of information about the JavaScript info. Perhaps as js came from scheme …

Lisp syntax and simplicity is my biggest selling point for Lisp vs other languages (except for Scheme, which is even more elegant).

Some people hate Lisp's heavy reliance on parenthesis and there have been so many languages that try to make a "Lisp without parenthesis", but to me they're missing half of what makes Lisp great.

I find programming in and reasoning about Lisp way, way easier than other languages. There's very little syntax to stand in the way or remember and the verbose way Lisp is written helps make Lisp programs self-documenting and way easier to understand than other languages.

This is super useful when doing a lot of language hopping or when coming back to the language/program after being away from it for a long time. Because of this I can quickly come back up to speed with the program and it also makes more larger and more complex programs more comprehensible -- that is, of course, if you don't use macros to turn Lisp in to some harder to understand language.

Oh yea, definitely, With rainbow parentheses, and paredit, the experience is top tier

Rainbow parentheses are such a great invention I can't understand why it isn't standard nearly everywhere.

I've found rainbow block-level highlighting to be even better: https://github.com/alphapapa/prism.el

I kind of wish editors focused on having different highlighting modes the way some strategy games have different map modes. I sometimes find myself changing to random language modes in emacs just to see different highlighting schemes that focus my attention on the code in different ways. I also do like rainbow block highlighting but it's not something I want all the time.

I never really bought the syntax simplicity argument, but what I eventually realized is it allows Lispy languages to implement any other paradigm or feature and it just naturally fits within the language.

Well, there are lots of "do x" libs that I don't think a C programmer could easily copy, but hey, we're all Turing complete here... As for language features, historically Lisp was not always first for some particular feature, but because of earlier design decisions that still haven't been made by newer languages, it can usually implement innovative features from other languages or PL research papers as libraries without having to update the language spec or an implementation itself. So we have immutable collections, laziness, takes on pattern matching, various concurrency models, more sophisticated type systems...

One very recent library that might be of interest is "just" a new regex library (https://applied-langua.ge/posts/omrn-compiler.html) but it leverages one unique design choice Lisp made in that you have access to the language's compiler at runtime, it's not a fenced-off separate program only used to create a runtime executable. The compiler can (and in SBCL's case does) optimize and produce assembly code, SIMD instructions can be used too, so this regex lib is very fast while not being that much code.

> but it leverages one unique design choice Lisp made in that you have access to the language's compiler at runtime, it's not a fenced-off separate program only used to create a runtime executable.

This is best understood as having no phase separation between compile time and run time. Effectively, languages that are so designed do everything at compile time, up to and including running the actual program.

Though sometimes they can practically recover the phase separation by "extracting" a program that can be compiled in a single step and ran as a conventional binary. For example, a LISP program for doing math may support "extracting" a custom function evaluator or numerical-symbolic solver written in FORTRAN.

I don't want to start a language war, given that I like both languages anyway, you should have a look at https://github.com/hanickadot/compile-time-regular-expressio...

However even counting with upcoming C++23 features, Lisp is much more ergonomic for this kind of stuff.

Oh I'd love to share it, I just can't. Well I made one that did make its way to Fortran by being independently developed, the Lisp version was in 2011 and the Fortran I think...2017. The brute forcer in AI-Feynman.

https://github.com/SJ001/AI-Feynman/blob/master/aifeynman/sy...

I talk about it in my portfolio, it's Guesser 1.0 and Guesser 2.0.

https://github.com/daniel-cussen/portfolio/blob/master/portf...

First-class functions, lambdas, and closures are mostly unheard of in the top 20 programming languages. Aside from JS, you're pretty much out of luck.

Proper tail calls. Technically not part of the CL spec, but implemented by the best CL variants (and by scheme), they allow a lot of expressive power. You won't find this ANYWHERE on the top 20 except for JS where it is technically part of the spec, but Google and Mozilla refused to implement leaving just Safari/JSC to have 100% ES6 spec compliance 7 years after the spec was finished.

Macros get overstated perhaps, but there are almost zero non-lisp languages that implement them and none of them implement macros ergonomically. In a language like JS, you'll often see a kind of DSL where they attempt to use closures and super-dynamic objects to create templates.

I see this a lot in React where you'll have huge config objects passed in and then very extensive branchy and loopy code to turn all the bits on an off. This is a poor man's macro. In a lisp, you would do all this stuff at compile time with all that inefficient branchy stuff executed once at compile time rather than millions of times on each of the thousands/millions of machines your code is running on.

The REPL has been mentioned several times, but it really is that good and nothing from other languages compares. People stress about "writing unit tests" because technically correct types doesn't mean your program actually works. In my time writing JS vs TS, I've seldom found that type errors were the problem aside from making the type checker happy. Most errors are in what the code actually does or does not do. The instant feedback from the entire system is far superior to unit tests because it can catch things that your unit tests might not be testing for.

Dynamic, but definable types is another advantage. You start out without a bunch of specific types which is better for getting things done, but can then add in type hints either as documentation or as a way for the compiler to spit out fast code. People may be down on dynamic code, but the truth is that the world is dynamic and runtime type checks are far safer than static types where they are then thrown out and you hope for the best. Because of the REPL-driven development, type errors are almost non-existent in practice.

>First-class functions, lambdas, and closures are mostly unheard of in the top 20 programming languages. Aside from JS, you're pretty much out of luck.

PHP, Perl and Swift all have first-class functions, lambdas and closures although I'd agree that that's not "most." This is probably getting too much into semantics, but aren't functions in JS just objects themselves? ;)

>People stress about "writing unit tests" because technically correct types doesn't mean your program actually works. In my time writing JS vs TS, I've seldom found that type errors were the problem aside from making the type checker happy. Most errors are in what the code actually does or does not do. The instant feedback from the entire system is far superior to unit tests because it can catch things that your unit tests might not be testing for.

I don't think having a "live" system like CL precludes the need for unit tests. Sure, it's beneficial to be able to play around with code right away, but unless you're planning on manually running through the same tests again and again, how do you know you've not introduced a regression at some point?

> PHP, Perl and Swift all have first-class functions, lambdas and closures although I'd agree that that's not "most." This is probably getting too much into semantics, but aren't functions in JS just objects themselves? ;)

Perl does have closures, but does not have lambdas as I recall (only subroutines) plus, but I didn't think it was particularly popular today. Dart and Swift do have all three of these things, but they also aren't particularly popular outside their niche mobile domain (or outside OSX in the case of Swift). Python has first-class functions and closures, but the lambdas are nerfed into near uselessness.

PHP has a Closure class, but you must manually populate this class with whatever things you want to save. This is hardly what would commonly be accepted as a closure.

> I don't think having a "live" system like CL precludes the need for unit tests. Sure, it's beneficial to be able to play around with code right away, but unless you're planning on manually running through the same tests again and again, how do you know you've not introduced a regression at some point?

You are quite correct. Unit tests are still required, but the coverage you get when running stuff manually is still usually better for the same reason that a person clicking buttons on your UI will find tons of issues your unit tests overlooked. By running your new code inside of the whole system, you are also integration testing on the fly. I'd also add that it is possible to use some of your setup state for unit tests later.

Perl does have lambdas...

  my $double = sub {$_[0] * 2};
  $double->(5);

No, you're right; just because you can test something at once doesn't mean you shouldn't automate repeated tests.

The thing is, though, not all tests are worth automating.

I generally start work by sketching out a strawman representation of what I think I'm doing, then interrogating it interactively to find out how reality differs from my naive understanding. As my understanding improves, I refine the representation incrementally and interactively, and refine the interrogations right along with it.

You can think of those interrogations as embryonic APIs and interactive unit tests, but most are not worth formalizing. They're ephemeral. The details they're interrogating are likely to change beyond recognition in a short time until discovery reveals the real shape of things.

Once the real shape starts to emerge from the lump of clay, the representation turns into actual data structures and the interrogations turn into actual APIs and formal tests.

I probably spend more time testing than in any other single activity while working in Lisp. I test pretty much every single expression interactively as soon as it's written, because it's so quick and effortless to do so.

But most of those tests are not worth formalizing. Many of them will be executed one time and then never again. Others will be useful long enough to earn a temporary home in a source file for a while.

The formalized unit and integration tests are the ones that have survived iteration, that are connected to structures and APIs that have survived and developed into something real.

And that's the process, pretty much: rough sketch; interactive experimentation to discover what it really ought to be; formalization of discovered features (including tests that are worth automating).

> First-class functions, lambdas, and closures are mostly unheard of in the top 20 programming languages. Aside from JS, you're pretty much out of luck.

uh, what? plenty of popular languages have all of those things.

Very few popular languages have all three and most that have them have them as bolted-on, unergonomic hacks that are generally more trouble to use than they are worth.

> First-class functions, lambdas, and closures are mostly unheard of in the top 20 programming languages. Aside from JS, you're pretty much out of luck.

This was true at one point, but most popular languages have them now, don't they? Even C++ and Java have them now (and have for a while).

Java has a terrible hack where it is technically creating a singleton class and attaching a function to that class that is then applied behind the scenes. The Functional interface is a horrible, unergonomic hack.

Likewise, C++ basically has function pointers (though I'd hardly put C/C++ in the same category). True closures aren't really possible without garbage collection or manually specifying the closure, but at that point, the closure is just a glorified struct and loses all its ergonomic advantages.

> Java has a terrible hack where it is technically creating a singleton class and attaching a function to that class that is then applied behind the scenes.

I'm not sure why that's a terrible hack. Scheme is just creating a struct behind the scenes anyway.

I actually find Java's approach to be quite ergonomic, because you can use lambdas anywhere you need a type that has exactly one virtual method. Of course, checked exceptions destroy the ergonomics anyway.

> True closures aren't really possible without garbage collection or manually specifying the closure

Rust's closures are not as ergonomic as e.g. Scheme's, but they're not bad.

Lambdas require a SAM functional interface to be defined for the thing they are going to be used for which is hardly easy to use or ergonomic compared to something like StandardML, Lisp, or even Javascript.

Also on this topic, but all things "closed" over must be `final`. Any attempt to change the variables closed over by a class will immediately fail.

The fact that Scheme is doing all these things behind the scenes is the point. The programmer isn't responsible for creating new structs/classes, adding all the things, and sending them around to the correct functions. In languages that require all this extra work, closures are seldom used because the extra code and cognitive overhead is almost always bigger than the advantages gained.

I’m not following. Are you somehow equating needing interfaces with having to manually construct classes?

In any case, I don’t see how needing closed over values to be effectively final is a problem. It’s no different than StandardML or OCaml.

I’ve seen lambdas used in Java all over the place, so they clearly have some value.

> First-class functions, lambdas, and closures are mostly unheard of in the top 20 programming languages

A quick Google search tells me the "top 20" includes Python, JavaScript, Java, C#, C, Go, Swift, PHP possibly Ruby/Perl/Kotlin/etc. I can't think of a single one that doesn't have either first-class functions or some form of closures.

> Proper tail calls.

This is an optimization issue for most languages, and they have made the trade-off of keeping a stack trace vs. optimizing it away. Having it in the Scheme spec was incredibly contentious and caused many heated debates.

> none of them implement macros ergonomically.

I think this really depends on what you mean "ergonomically." That's a weasel word in this context. Because there are two camps. There is the CL Lisp-2 that feels they should be allowed to use LIST as a variable in a macro so they went and created a whole namespace just for functions but are perfectly fine with the ugly GENSYM hack. Then there is the Scheme camp that failed to come up with a suitable hygienic replacement for DEFMACRO for like 20 years. Macros have been broken in LISP world longer than most of us have been alive.

> React where you'll have huge config objects passed

Not sure what you're referring to here. A/B testing is a thing. Can't do that with macros. Hydration is also a thing, because of SSR. Maybe you're seeing that?

> REPL-driven development, type errors are almost non-existent in practice

ehhhh. No. And I say this as someone that generally prefers dynamic over static typing. You can easily have objects in your REPL that were the result of previous bugs and get your app in an inconsistent state. You either reload, or spend your entire life cleaning up memory and hoping you get it right. There is no silver bullet here, despite what others have commented on. Because you're talking about schema migrations here. Bad data structures have to be transformed to good data structures. Or you'll see bugs in the REPL that aren't actually bugs in the source. Keeping the REPL and source in sync is also tedious, to put it mildly.

Look into Erlang does and what is required to get live code swapping correct. It's a headache that is only worth it in very niche apps (like massive telecom systems). It doesn't make sense doing this during development. Much saner to just reload and start from a fresh, consistent point. Otherwise you will spend time hunting down bugs that do not exist outside your REPL.

> A quick Google search tells me the "top 20" includes Python, JavaScript, Java, C#, C, Go, Swift, PHP possibly Ruby/Perl/Kotlin/etc. I can't think of a single one that doesn't have either first-class functions or some form of closures.

Most of them don't have all of these features together and they often have incredibly bad versions of those features that are basically unusable.

> This is an optimization issue for most languages, and they have made the trade-off of keeping a stack trace vs. optimizing it away. Having it in the Scheme spec was incredibly contentious and caused many heated debates.

The JS stack already goes away the second you do anything asynchronous. As this is something you do frequently, stack traces are already incredibly short.

Meanwhile, the alternative for the use cases of PTC is loops and I don't hear complaints that loops don't save your state.

It's in the spec, it should be implemented.

As to Scheme, you won't find anyone today arguing that PTC is bad. The bigger argument would be about how full continuations are a performance disaster.

> I think this really depends on what you mean "ergonomically."

Infix macro systems are terrible to use compared to S-expr macros and I don't think this is anywhere close to controversial. Can you point to a counter-example?

The best infix example I know of is Dylan, but not only is that language barely infix (its S-expr foundations are very apparent) and they still just aren't as easy to use.

> ehhhh. No. And I say this as someone that generally prefers dynamic over static typing. You can easily have objects in your REPL that were the result of previous bugs and get your app in an inconsistent state. You either reload, or spend your entire life cleaning up memory and hoping you get it right. There is no silver bullet here, despite what others have commented on. Because you're talking about schema migrations here. Bad data structures have to be transformed to good data structures. Or you'll see bugs in the REPL that aren't actually bugs in the source. Keeping the REPL and source in sync is also tedious, to put it mildly.

How does this entire bit have anything to do with dynamic vs static typing? You could have a REPL with static typing if you truly wanted.

In any case, reloading data from the code isn't that difficult and most data will be held in closures which means you can decide exactly how far up you want to replace local data (and if you're going about it smartly, you have created a few "mocks" to work with that can be used to refresh data).

The absolute worst case (restarting the REPL) is what other languages would force you to do anyway which still means you have a lot to gain for most development and bugfixing and absolutely nothing to lose.

As to data is held in a database, then you have all the same problems in other languages except that fixing these issues from the REPL is generally easier because you are already connected and ready to start fixing things.

> Look into Erlang does and what is required to get live code swapping correct.

BEAM is an amazing platform, but a REPL for development and live code swapping are two different discussions. Doing REPL work on production systems has a very big chance of catastrophic damage and I wouldn't recommend it.

> [Proper tail calls are] an optimization issue for most languages

It's an optimization issue for no languages; it's an optimization issue for some language implementations.

At the language level, it's a resource management (garbage retention) issue that leads to an expressiveness problem. See Clinger's papers on safety-for-space (e.g. [1]) and Felleisen's paper on expressiveness [2]. Support for proper tail calls in a language implementation also sometimes makes program execution faster, but that's not the main motivation.

[1] https://www.cs.tufts.edu/~nr/cs257/archive/will-clinger/prop... [2] https://www.sciencedirect.com/science/article/pii/0167642391...

The one that sticks out for me is condition handling, particularly restarts. I'm someone who reads books about programming more than I actually program, so I don't know how actually practical Lisp's ability to do extremely sophisticated responses to error conditions and the like (including choosing where to throw to programmatically at the error site) is. It strikes me that it might be one of those things -- like hot-reloading -- where there are so many factors that it's mostly impossible for humans to get right. But!! I've never seen it in any other language, and I think about it a lot.

@phoe-krk's book on it, https://news.ycombinator.com/item?id=24867548 is very good.

This isn't an exact answer to your question, but I have to say macros.

Going from CL (and scheme probably too) to macros in any other language is like going from a lambo to a tricycle. It's one of the things I miss most about CL.

I also love the live-editing aspect of CL. When I was doing game programming with it, being able to replace functions while running was incredibly useful. You don't have to re-create all your state, you can jsut live-edit a function and boom fix a bug or add a new feature right there. Very good for exploratory programming with lots of otherwise-hard-to-replicate state.

Well put. I agree on both points.

CL has a very advanced TDD. You actually develop by executing snippets in REPL and pull them all together in a function (already tested!).

I don't know about "since 2005" but there are two features in the original CL spec which I find indispensable but which still have not made it into any other languages (AFAIK):

1. Generic functions

2. Being able to redefine a class and have existing instances of that class be automatically updated to conform to the new definition

When would you ever legitimately use #2?

If e.g. an API wants to support monkey patching that's a decision that can be made. But thinking that any e.g. plugin or addin could could redefine extant class instances sounds like a nightmare, and a security issue just waiting to happen.

In longrunning/high-availability applications, this sort of thing is necessary in order to avoid restarting and losing uptime or state. E.G. per nasa:

> The Remote Agent software, running on a custom port of Harlequin Common Lisp, flew aboard Deep Space 1 (DS1), the first mission of NASA's New Millennium program. Remote Agent controlled DS1 for two days in May of 1999. During that time we were able to debug and fix a race condition that had not shown up during ground testing. (Debugging a program running on a $100M piece of hardware that is 100 million miles away is an interesting experience. Having a read-eval-print loop running on the spacecraft proved invaluable in finding and fixing the problem

(from https://flownet.com/gat/jpl-lisp.html)

Interesting, thank you!

Any code modification is a potential security issue. There is nothing special about dynamic class redefinition in this regard.

I use it for development and deployment. I can deploy new code without having to take my application down. In fact, not only do all my existing instances get updated, but I also use an ORM [1] that automatically updates my database tables too.

[1] https://github.com/rongarret/ergolib/blob/master/layer1/sql....

I use Common Lisp in my day job, and I generally build software by livecoding, so I use this and other interactive programming support on a daily basis. Seems "legitimate" to me. I don’t want to have to stop my work-in-progress program running and lose all of the incremental state I’ve built just because I redefined a class or something.

So it's a feature that you would use during development, but not in the final (ha, ha, that never happens) state of the application?

Shinmera's Trial game engine, used in Kandria (which you can find on Steam), uses bits of the same machinery to implement the resourc system inside the game.

IIRC it has separate classes for "unloaded entity" and classes for various types of resources, and the protocol (think interface) for them includes the generic function that loads them into memory. The implementation of said function for unloaded objects uses CHANGE-CLASS method redefine the instance in place - meaning all direct memory pointers to the class are still valid and you don't need to use extra layer of indirection.

I can agree that the most common uses are for development in progress, but I wouldn’t go so far as to say I wouldn’t use those facilities in a fully worked-out program. A sibling comment offers one reasonable example. I’ve done similar things before and probably will do them again.

What is a generic function?

It's a function that has one or more required arguments, and a set of methods. Each method has a class (or an EQL specifier) associated with each argument. When the generic function is called, the best method for which each actual parameter is in the corresponding argument's class is invoked. Think of it like a method in ordinary OO, but it can dispatch on multiple arguments.

There's also some interesting stuff on top of this (method combinations) that allow multiple methods to be invoked and their results combined.

https://en.wikipedia.org/wiki/Generic_function#In_Common_Lis...

http://clhs.lisp.se/Body/07_f.htm

For an interesting application, see how https://github.com/joaotavora/snooze#rationale maps generic functions to HTTP/REST.

Hello, that's my website o/ I'm a normal programmer enjoying CL, writing documentation and shipping simple but useful apps in production©. Not solo since a friend contributed to a web app once (without ever touching any Lisp before). Feel free to ask any questions :)

What made you go CL instead of something like, say, racket? Is it being a lisp-2 as much of a pain as it looks from the outside?

Any tips for groking the package/namespace system? I've taken a couple runs at CL and that's where my brain always breaks down.

A package is a container for symbols (symbols are a basic type that references variables and functions (or both)). Where you are "inside" a package (in-package), you can access all symbols directly. Otherwise, you need to name them with the package prefix. This is how it serves as namespace. This quick demo might help: https://youtu.be/XFc513MJjos?t=249

I very seldom tried Racket. CL picked my interest, and no language feature in a Scheme was attractive enough to me. CL seemed the most capable of all Lisp dialects, it seemed the most robust (a long history of industrial success stories), when I searched for CL libraries I found a lot, less so with Racket, etc. And so, the more I committed to learning and building stuff in CL, the less likely I was to switch to another Lisp. (Clojure? It takes so much resources when starting up, especially compared to CL, and I can't even install a library in a running REPL. That's my excuse for my disgust and fear of Java). I compiled some things lispers say about Racket: https://gist.github.com/vindarel/c1ef5e043773921e3b11d8f4fe1... and it makes me want to stay in my comfy CL environment (built with sweat).

I myself was never annoyed with the lisp-1/2 thing… You like the fact that in lisp-1 we can call a function given as reference directly, without `funcall`? Well, we can do that in CL, it's a `setf` away. It turns out we don't and it's alright.

The resources are as expected very enacts oriented. I use slimv with vim. And in particular under iOS Textastic to a ssh tmux instances, it is much easier due to its modal approach. Anyway a good resource but need some adjustment to me.

Reply on news.ycombinator.com