Comment by rltuax | Hacker News Reader

Comment by rltuax | Hacker News Reader

> What libraries or tooling would you like to see written for Common Lisp next?

I would like the entire Python scientific ecosystem be ported to Common Lisp instead of Python. Lisp is vastly superior for numerical work.

What would be required for this to happen?

I think empirically we can say that people who wrap Intel MKL, BLAS, etc. prefer a stable C-API with stable C functions that access the internals and do not want an FFI.

The native FFI from SBCL is difficult to handle and performance is somewhat unpredictable due to garbage collection.

I know this is a hard problem that alternative Python implementations like PyPy also fight with, but if SBCL somehow managed to publish a C-API that is as fast as CPython's (single threaded is good enough for the start) and retain the excellent speed for pure Lisp, I think it might be a great success.

Even tricks like automatically activating some form of a GIL when loading a C module would be completely acceptable. In general, roughly copy the Python C-API, it is safe to say by now that people want this.

Can you explain in what way Lisp is better than Python for numerical work? For someone who has never used Lisp?

smart common lisp compilers are orders of magnitude faster than cpython without annotations and can be in the same ballpark as c or fortran with annotations, which means that you can do your numeric work in lisp itself, without calling out to any kind of foreign libraries.

it is not feasable to do numerical work in python itself, in reality python acts as a construction kit dsl for allocating and manipulating foreign (c and fortran) structures. this is usually sufficient as one can see by the massive success of libraries like numpy. when you are writing novel numeric code though, you have to figure out how to match it to the numpy's model. this is also not usually a problem, since matching it to numpy's model usually makes it architecturally performant. but one is still wearing a kind of straight jacket at the end of the day.

when i was doing computation chemistry, i wrote a lot of code from scratch in fortran and lisp, and it was entirely feasable to do mainloops purely in lisp, implementing algorithms close to their paper versions.

there are all kinds of aspects of lisp that make it pleasant to work with in computational science area, but this is already a tldr. i'll mention one, it's possible to rig your code in a way that a long running batch process will not lose its state without much code overhead. since lisp lets you recover from error without unwinding the stack, you can trust that after hours or days of computation an error is not going to cost you full progress loss. often times you can redefine the offending part of code, and continue processing.

Any experience with Julia and how it stacks up to CL for numerical computing?

I would say that Julia's great advantage is following Dylan's footsteps, being a Lisp without parenthesis, for those that can't get their head around them.

From the point of view of code generation, its JIT takes advantage of being built on top of LLVM's optimisations.

It's a lot better for numerical work.

* Julia has multiple dispatch like in CLOS, except generic functions can devirtualize their arguments, and can be inlined, making the composition of many small function calls significantly faster.

* All functions (except a dozen or so internal builtins) are generic functions which can have methods added to them, and all objects can be dispatched on.

* Objects can be isbits and allocated inline in an array or stack allocated without any pointer indirection.

* Julia's type system is parametric, and things like Array is parameterized on it's contents, meaning that you can dispatch on thing like Array{Int} as a distinct type from Array{Quaternion{Float64}}.

There's lots of things Common Lisp does really well, but I really do think in the niche of numerical computing, Julia just blows it out of the water for performance and also ecosystem size / vibrancy.

* Lisp has a numerical tower, meaning you almost never have to worry about rounding errors or overflow errors. It's numeric computing primitives are extremely well designed.

* Lisp compilers make stupid fast self contained executables but the language is still dynamically typed

* C FFI story is mature and works great but you actually have full garbage collection still, no GIL, no reference counting.

* Did I mention no GIL? Full OS threads available for multithreaded workloads.

I don’t know if it is still actively maintained, but their copy used to make it sound like it was able to compete with Julia.

https://lisp-stat.dev/

Reply on news.ycombinator.com