Common Lisp Community Survey 2024 Results
blog.djhaskin.com
blog.djhaskin.com
I would like the entire Python scientific ecosystem be ported to Common Lisp instead of Python. Lisp is vastly superior for numerical work.
What would be required for this to happen?
I think empirically we can say that people who wrap Intel MKL, BLAS, etc. prefer a stable C-API with stable C functions that access the internals and do not want an FFI.
The native FFI from SBCL is difficult to handle and performance is somewhat unpredictable due to garbage collection.
I know this is a hard problem that alternative Python implementations like PyPy also fight with, but if SBCL somehow managed to publish a C-API that is as fast as CPython's (single threaded is good enough for the start) and retain the excellent speed for pure Lisp, I think it might be a great success.
Even tricks like automatically activating some form of a GIL when loading a C module would be completely acceptable. In general, roughly copy the Python C-API, it is safe to say by now that people want this.
it is not feasable to do numerical work in python itself, in reality python acts as a construction kit dsl for allocating and manipulating foreign (c and fortran) structures. this is usually sufficient as one can see by the massive success of libraries like numpy. when you are writing novel numeric code though, you have to figure out how to match it to the numpy's model. this is also not usually a problem, since matching it to numpy's model usually makes it architecturally performant. but one is still wearing a kind of straight jacket at the end of the day.
when i was doing computation chemistry, i wrote a lot of code from scratch in fortran and lisp, and it was entirely feasable to do mainloops purely in lisp, implementing algorithms close to their paper versions.
there are all kinds of aspects of lisp that make it pleasant to work with in computational science area, but this is already a tldr. i'll mention one, it's possible to rig your code in a way that a long running batch process will not lose its state without much code overhead. since lisp lets you recover from error without unwinding the stack, you can trust that after hours or days of computation an error is not going to cost you full progress loss. often times you can redefine the offending part of code, and continue processing.
From the point of view of code generation, its JIT takes advantage of being built on top of LLVM's optimisations.
* Julia has multiple dispatch like in CLOS, except generic functions can devirtualize their arguments, and can be inlined, making the composition of many small function calls significantly faster.
* All functions (except a dozen or so internal builtins) are generic functions which can have methods added to them, and all objects can be dispatched on.
* Objects can be isbits and allocated inline in an array or stack allocated without any pointer indirection.
* Julia's type system is parametric, and things like Array is parameterized on it's contents, meaning that you can dispatch on thing like Array{Int} as a distinct type from Array{Quaternion{Float64}}.
There's lots of things Common Lisp does really well, but I really do think in the niche of numerical computing, Julia just blows it out of the water for performance and also ecosystem size / vibrancy.
* Lisp compilers make stupid fast self contained executables but the language is still dynamically typed
* C FFI story is mature and works great but you actually have full garbage collection still, no GIL, no reference counting.
* Did I mention no GIL? Full OS threads available for multithreaded workloads.
I am, however, continually surprised that SLY doesn’t have more mindshare. Similarly jonathan. I guess cl-json and SLIME are _good enough_ for most people.
…or maybe I need to revisit them?
Thus I used Debian and not Ubuntu, Vim and not Neovim, Emacs and not Spacemacs (though that’s not exactly a derivative), etc.
I do use SBCL rather than CMUCL, as I do understand why SBCL is better for me: CMUCL won’t run on my platform.
I look at the Sly page and I have no idea why I would want any of the stuff it does, so I stick with Slime.
Plenty of folks might not try Lisp because they’re happy with the tools they have. And that’s ok.
You could even have posted to comp.lang.lisp.
In any event, I thought it would be worth documenting such for any survey.