drmeister

I did - and we compiled our first batch of molecules using an application implemented in Cando (Clasp + computational chemistry code) yesterday. I'm absolutely serious about this.

Lisp (Common Lisp) beats all the other dynamic languages by a considerable margin. This is why I am developing Clasp - a Common Lisp implementation based on LLVM that interoperates with C++/C (https://github.com/clasp-developers/clasp.git) for scientific programming.

With Clasp, we get the best of multiple worlds. We get a dynamic language (Common Lisp) with automatic memory management and enormous expressive power that can directly use powerful C and C++ libraries. All three of these languages are "long-lived" languages in that code that was written 10 and 20 years ago still works.

Performance is really important to me and I have written a lot of code over the past four decades. I won't develop meaningful code in any language that falls below Racket in table 4 because these language implementations are too inefficient. I furthermore want to keep using my code over the years and decades and so I won't develop meaningful code in any language where someone else can break my code by changing the standard. My program "leap" was written 27 years ago in C and it is still being used daily by thousands of computational chemists. But it's really hard to improve leap because the code is brittle largely because of malloc/free-style memory management (brrr). For a compiled, high-performance, standard language with proven lasting power and performance - Common Lisp is the best choice.

Huh - I somehow submitted the same message twice. Hacker News doesn't let me delete it. So I'll edit it down - see the version above about Common Lisp and our implementation of it called Clasp.

Thank you. So debs for debian? I have great collaborators working with me now that could do this. We have one for Arch I believe. I just haven't prioritized this yet.

I can testify to a lot of the difficulties Robert Strandh talks about here. We implemented a Common Lisp in C++ that interoperates with C++ (https://github.com/clasp-developers/clasp.git). I ended up implementing a lot more C++ code than I wanted to and then replacing much of it with Common Lisp versions (during startup) because it's easier to maintain and we can use type inference to improve the performance of Common Lisp code while C++ code is 'stuck' with the types you compile it with. Bootstrapping is hard.

Dang - that's helpful. Earlier today I posted here (https://news.ycombinator.com/item?id=22456642) about how throwing exceptions in C++ threads was very expensive because there is appears to be a mutex in the exception handling code. Now I can put a name to it "false sharing" - thanks!

In libgcc it's in Unwind_Find_FDE - we think it's a lock around walking the loaded dynamic libraries. I haven't personally dug much deeper into it but my folks here and the llvm engineers seem to be pretty certain that's the problem (this: https://github.com/gcc-mirror/gcc/blob/master/libgcc/unwind-...). Right now we are rearranging our compiler so we throw fewer exceptions because you don't have to optimize things that you don't do :-).

That's the point of everything. We have spun up a company and big things are in the works: http://www.thirdlaw.tech/

It's going well. I'll post something soon. We've just been working on it quietly. We have multithreading, unicode, cffi etc, good debugging support, cross-language profiling and more.

This sounds interesting - thank you! We need to interoperate with C++ - so we couldn't use this, could we? We could add this pointer to our own frames but C++ frames won't have this info - so I'm not sure how they would interoperate. We need to be able to throw an exception and invoke both Clasp frame cleanups and C++ frame cleanups up the stack. We have a crazy mix of C++ and CL frames on the stack at any time.

Yes, gcc/libstdc++ have the same problem. That's where I saw it first and then I tried llvm/libunwind and saw the same thing.

Thank you. We have developed a Common Lisp implementation that uses LLVM and interoperates with C++ and uses C++ exception handling to unwind the stack. Common Lisp code relies on stack unwinding a fair bit. Imagine my surprise when my fancy multi-threaded compiler can't get out of first gear (tops out at ~150% cpu) on Linux. Sheesh.

Threads are very expensive if you start throwing C++ exceptions within them in parallel. You see the overall time to join the threads increases with each thread you add. There is a mutex in the unwinding code and as the threads grab the mutex they invalidate each other's cache line. I wrote a demo to illustrate the problem https://github.com/clasp-developers/ctak

MacOS doesn't have this problem but Linux and FreeBSD do.

Oh yes, please someone answer this. I have used adobe illustrator for 15 years - then affinity designer for a few more years. I just want vector graphics with arrows without enormous subscription fees - damnit!

IMHO - you should learn Common Lisp - it solved the problems described in the talk (Nim can't redefine types on the fly) decades ago. In Common Lisp you can redefine _everything_ on the fly. You can change types (classes) in a running system and objects in memory that have the layout of the old class get upgraded automatically when they are used. Programming with Common Lisp in Slime/emacs is an amazing experience that will spoil you for other languages.

We are traveling down this road. We've implemented Clasp (https://github.com/clasp-developers/clasp.git) - an implementation of Common Lisp that interoperates with C++ and uses llvm as the backend. On top of this, we built Cando (https://github.com/drmeister/cando.git) - a computational chemistry programming environment for designing large molecules. The end of our story hasn't been written yet.

Clasp could be used for a lot of other things.

Lisp without s-expressions is like bathwater without a baby.

S-expressions enable homo-iconic macros and make compile-time computing super easy! I spent 35 years programming in C and C-like languages and then developed my own Common Lisp implementation (http://github.com/clasp-developers/clasp.git) - the only Common Lisp that interoperates with C++ and uses llvm as the backend. When I started with lisp I wrestled with the syntax and sweet-expressions and just decided to let it go. Now I've found language love all over again!

In a world that runs on Javascript, nobody should be able to complain about a few measly parentheses.

Obligatory relevant XKCD comic: https://xkcd.com/297/

It's things like the visitor pattern that led me to implement Clasp - a Common Lisp that interoperates with C++ and uses llvm as the backend (github.com/clasp-developers/clasp.git). Common Lisp has generic functions and multiple dispatch - so it doesn't need the wretched visitor pattern. Try writing a compiler using the visitor pattern - I dare you. :-)

A growing group of us are implementing Clasp (https://github.com/clasp-developers/clasp) - a new Common Lisp that uses Robert Strandh's Cleavir compiler, interoperates with C++ and uses LLVM as the backend. Clasp is the foundation of Cando - a programming environment for the development of molecular devices (https://www.youtube.com/watch?v=mbdXeRBbgDM&feature=youtu.be). Robert Strandh is an exceptionally clear thinker and a fantastic programmer. I can't count the times that Robert has implemented changes to the Cleavir compiler source code and without him being able to test it (long story) passed it to me and it ran flawlessly in Clasp. I implemented his fast generic function solution within Clasp based on another of his papers (http://metamodular.com/generic-dispatch.pdf). Edit: Basically - if Robert writes it up in a paper - I want to have the implementation.

Once we develop molecular robots - they are going to implement LispOS. :-)

Nice - I think we will make the switch to this. We wrote a Common Lisp kernel for Clasp (Common Lisp system that interoperates with C++ and uses LLVM as the backend -https://github.com/clasp-developers/clasp) We implemented ipykernel and ipywidgets in Common Lisp by translating the Python code. I think we could switch to Xeus and lower the maintenance burden and contribute to Xeus.

Thank you. It was a talk for the llvm developers meeting and I'd been turned down for talks a couple of times a couple of years before that. I REALLY needed to get to this meeting to talk with llvm developers - so I pitched the title that I thought people there wanted to hear...

There's a lot more to it. You need to translate native objects to and from C++ objects efficiently. You need to work with C++ exception handling to unwind the stack without clobbering RAII.

It's easy to say something like this in retrospect But even after working on this for a couple of years I have no idea how I would get started with SBCL. There was a team at Google that tried to add a llvm backend to SBCL and they stopped working on it after a year. I don't want to knock SBCL - it's an amazing compiler! But I think there are way too many problems that I would have had to solve, all at once, to get C++ and SBCL to interoperate.

Yes, there is never enough time to cover all that I want. There will be more talks in the future and more as Clasp gets more exposure.

Thanks, everyone! I missed this thread when it first came out.

"A proper C++ ffi for CL with a good clbind" is easy to say but much harder to implement - no one but I have achieved it and I had to implement a Common Lisp to achieve it. It's not just calling functions and translating arguments and return values that is difficult. The much harder problem is unwinding the stack over C++ frames that use RAII.

My group is developing Clasp (github.com/clasp-developers/clasp.git). It's a Common Lisp implementation that interoperates with C++ and uses llvm as the backend. It's something you might want to look at because it could change your options in the following ways:

(1) Clasp interoperates with C++ - C++ libraries like graphics and physics engines can be easily exposed within Clasp Common Lisp. (See https://github.com/clasp-developers/demo-expose-bullet-physi...)

(2) Clasp uses llvm as the backend - it generates llvm-ir from Common Lisp code and uses link time optimization to link C++ and Common Lisp generated llvm-ir.

(3) Clasp uses the Memory Pool System garbage collector, which can be configured to minimize GC pause times to a programmer-defined level.

(4) Clasp generates debug info and can be profiled with standard tools like dtrace and C++ functions, Common Lisp functions and GC functions are all profiled against each other.

I'm a professor of Chemistry at Temple University - I'm developing Clasp to design molecules and materials - but it could be applied to a lot of other problems.

Caveat: Clasp is not yet as performant as Steel Bank Common Lisp, but it uses the open source and highly configurable Cleavir compiler developed by Robert Strandh and we have incorporated inlining and are currently incorporating type inference.

If you want to see a little bit more - I have a 25 min talk up on youtube that I presented a couple of weeks ago at the 2018 llvm developers meeting (https://www.youtube.com/watch?v=mbdXeRBbgDM&feature=youtu.be)

Author here - thanks folks! I'm glad you liked it.

Counterpoint - I implemented a Common Lisp compiler that uses llvm as the back end (https://github.com/clasp-developers/clasp). I've kept up with the versions from 3.6 to 6.0 and I'm about to upgrade to 7.0.

I'm not sure why other projects find this difficult but I have noticed that if I get too far behind it does get a bit more difficult.

I exposed the C++ API to Common Lisp and write all of the compiler in Common Lisp. The Clang C++ compiler tells me what is broken when I upgrade to a new version of llvm and then I fix it. I talk about it here https://www.youtube.com/watch?reload=9&v=mbdXeRBbgDM&feature... at about 13:30

I hired/partnered with one of their former engineers. QPX is most certainly still written in Common Lisp.