Lessons Learned Implementing Common Lisp with LLVM [video] | Hacker News Reader

Lessons Learned Implementing Common Lisp with LLVM [video]

youtube.com

152 pointsStreamBright30 comments

Wow!!! This is a wonderful talk.

The title is so old school though that need changing.

- How I use lisp to fight human aging or - why I choose lisp over python to combat protein generation - my protein generation using llvm was done all by lisp for. Reason

The talk is amazing. This one chemistry prof. who seems to be an IT expert but really just start lisp programming by building a lisp interpreter.

Wow!!!

https://github.com/clasp-developers/clasp And The cando one.

Amazing talk for sure, I think I saw a similar presentation a couple of years ago. His older video was called "Molecular Metaprogramming", one of the coolest titles ever.

oh cool, I had no idea it was a new talk on Clasp.

Thank you. It was a talk for the llvm developers meeting and I'd been turned down for talks a couple of times a couple of years before that. I REALLY needed to get to this meeting to talk with llvm developers - so I pitched the title that I thought people there wanted to hear...

This talk references the Cleavir compiler, which is part of Robert Strandh's (of McCLIM and Climacs fame, at least) modular CL implementation called SICL: https://github.com/robert-strandh/SICL. Looks like there's at least one discussion on HN: https://news.ycombinator.com/item?id=8284835.

This is fascinating to me because coming at it as a Lisp programmer, my impression of LLVM is that it's primarily a library for doing Lispy things in C. It kind of makes sense, then, that you should be able to use the codegen parts of LLVM for implementing Lisp itself, and I love that someone has actually done that. And that he bootstrapped it using an existing self-hosted compiler.

The obvious question I have is why not just use something like SBCL as the starting point. His own benchmarks show it's ~4x faster (so far). Is it not easier to add GPU support to SBCL, than to write a new Lisp compiler from scratch? Of course, this way is more fun.

"easier to add GPU support to SBCL" excuse me?! Where'd you even begin?

I'd begin by adding a new architecture, and merging an existing library which can convert expressions to GPU kernels -- same as he's doing now with his LLVM-based one. Why would it be any different? Is LLVM so GPU-friendly that it's worth it to write 250K lines of code just to be able to call it a bit more easily?

Yes it is, LLVM toolchain is directly supported by Khronos for GPGPU standards (LLVM <-> SPIR-V integration), NVidia also makes use of it, and it has a backend for shaders emulation on CPUs.

I don’t think anyone anywhere has achieved general GPU offload for a general purpose language yet, so your idea that it would be easier than writing a lisp compiler, which we’ve done millions of times, is probably not realistic.

Millions of times? Toy lisps, maybe; full Common Lisps, no.

Good to see ansi-tests mentioned. :)

Common Lisp is not just "a lisp". I can probably count on the fingers of one hand the number of Common Lisp compilers I've seen in my life. I can write "a lisp" in a couple dozen lines of code, but my copy of CLTL2 on the shelf is about 2 inches thick.

I'm not saying GPU offload is easy. As you point out, no general-purpose language today does that (and LLVM doesn't seem to offer it as a feature), so you'd have to implement that either way. The choice is "new compiler + GPU target" versus "mature portable compiler + GPU target". Why is the latter "not realistic"?

I find it hard to believe that the architecture of every (free) compiler is so horribly suited to this that we need to start from scratch. We've forked CL compilers many times, for new architectures, but I've never seen any architecture that required starting over.

The main reason to write a new Lisp compiler based on LLVM was the full integration with the hundred of thousands c++ lines of code he previously wrote.

What does "full integration" mean, and why couldn't that be achieved with some macros? He already showed how he used macros to achieve nice integration with LLVM (written in C++) for his interpreter.

Worst case, you'd have to write a name mangler, to teach your FFI about C++ symbols, but I imagine that would be at least an order of magnitude easier than an entire compiler (150K lines C++, 85K lines CL).

FFI is very slow. He probably needs C++ calls in hot loops.

There's a lot more to it. You need to translate native objects to and from C++ objects efficiently. You need to work with C++ exception handling to unwind the stack without clobbering RAII.

I have no idea why he did it instead of forking SBCL, but I thank him for it dearly. Lisp on LLVM is one of the things that will save it from dying. LLVM comes with extreme portability (future proofing to future platforms) and lots of industrial strength tooling. SBCL is very good, but it’s hard to say how long it will keep being developed.

> I have no idea why he did it instead of forking SBCL

It's basically explained in the video. He forked ECL, because he wanted to have a CL implementation based on LLVM and very deep C++ integration. ECL is already a C-based Common Lisp.

> SBCL is very good, but it’s hard to say how long it will keep being developed.

As long as people work on it.

It's easy to say something like this in retrospect But even after working on this for a couple of years I have no idea how I would get started with SBCL. There was a team at Google that tried to add a llvm backend to SBCL and they stopped working on it after a year. I don't want to knock SBCL - it's an amazing compiler! But I think there are way too many problems that I would have had to solve, all at once, to get C++ and SBCL to interoperate.

Having a hard time finding words to express how interesting of a topic Mr. Schafmeister is working on!

I’m playing around with compilation, self-studying during weekends. I want to learn how to compile Closures from my small Lambda Calculus language into something like C++. “Baby’s first FP codegen”, or the like.

I have struggled to find reading material for the intermediate beginner on this subject. So, if anyone on HN has a favourite link on the web about compiling functional languages, please feel free to share them.

Here's a book dedicated to compiling closures to C:

http://t3x.org/clc/index.html

It starts with lambda calculus and ends with compiling a tiny Scheme subset to portable C code.

You might want to look at Ferret (https://github.com/nakkaya/ferret), which compiles a subset of Clojure to C++, it could be useful to help you understand. It's also written in a literate style, so it's also well-explained.

Still 4x slower than sbcl. And lot of threading problems, thread-unsafe hashes and symbols all over.

A proper C++ ffi for CL with a good clbind and GPU binding would probably help him better than this monster.

You should pitch the idea to him, I think he'd probably respond favorably considering how important the biochemistry topic is.

"A proper C++ ffi for CL with a good clbind" is easy to say but much harder to implement - no one but I have achieved it and I had to implement a Common Lisp to achieve it. It's not just calling functions and translating arguments and return values that is difficult. The much harder problem is unwinding the stack over C++ frames that use RAII.

I have heard him talk before about Clasp, really entertaining. Coincidentally, I just built Clasp from source last weekend. I don’t when I will have time to do this, but I would like to choose a deep learning library with a good C++ interface (perhaps mxnet) and try to get a good Clasp wrapper written.

If you do this, using PyTorch’s new C++ frontend and API might be better; I think PyTorch is generally more popular, at least for use by individuals (maybe MXNet is better in production or something).

https://pytorch.org/cppdocs/

(i was recently working with PyTorch and wishing I could use Common Lisp instead of Python, for various reasons.)

Thanks, everyone! I missed this thread when it first came out.

The video is preternaturally clear. Not sure if it's because I'm on a new monitor or what.

Reply on news.ycombinator.com