An Exploration of SBCL Internals (2020) | Hacker News Reader

An Exploration of SBCL Internals (2020)

simonsafar.com

132 pointsv9v106 comments

I wish other Common Lisp implementations, particularly Clozure CL, documented their internals as well as SBCL does. Apple Silicon support has been a long-standing feature request, but nobody can really work on it besides rme himeself due to the sheer amount of "institutional knowledge" (and time!) it takes to understand CCL's compiler backend then add support for a brand new architecture. SBCL, on the other hand, was able to provide support fairly quick, possibly due to more maintainers, but I also think a big role that played in it was the fact lots of corners of the implementation are documented and it's fairly easily to get a hold of people actively working on the compiler (doug, stassats, et al)

Fresh news from 2 weeks ago for CCL:

> I recently decided to leave my current full-time job, and I’ll be working on CCL again. […] I’ll be able to work about half-time on an ARM64 port. Please write me privately if you want to talk about supporting that ARM64 work.

> I've worked on CCL for quite a few years, and I did the 32-bit x86 port, so I have experience in this area. I’m not as good a hacker as Gary Byers, but then again, few people are.

rme

https://lists.clozure.com/pipermail/openmcl-devel/2024-April...

SBCL 25th anniversary workshop is in Vienna thu-fri next week. See you there!

My issue prevent me from investing in Lisp is that Erlang gives me a similar REPL which I believe, though I am not a Lisp developer, is a huge value proposition of Lisp. I utilize REPL-based development using Erlang.

The macro narrative is worse in Erlang; but Lisp isn't perfect either in other ways.

The point is, I net out staying on the BEAM. What I like about Lisp is that I think it's a tool that will serve you for the rest of your career, even if you don't use it for production. That's a pretty cool proposition and maybe one day I'll get there. Emacs is a similar tool.

Last time I played with Elixir (I'll assume it's similar enough to Erlang), I found the REPL / interactive experience not as rich as CL's (which I must say, reassured my beliefs), even though the command-line tools are great:

- can't install libraries from the REPL while working on your program

- can't "compile this function" with a keystroke (I did find "send region/line/buffer to REPL" which is different, or "reload module")…

- … and get type errors and warnings

- can't get an interactive debugger on an error, go to the bogus line (while keeping the debugger), fix the function, compile it, switch back to the debugger, resume execution from any point in the stack. I only found out IEx.pry which is useful and all, but looks like Python's ipdb break point.

- lack of little useful editor commands, at least in the Emacs modes I tried like "write this function in the REPL", "eval and print the last expression". I struggled to have the "go to definition" feature but that depends on the IDE I suppose. The Emacs modes are not in good shape :S

- no cross-references? doubting. And maybe other small things only about the REPL, didn't find my notes.

differences of runtimes are also at play here, but I still find it interesting to compare the REPL experience, I'll welcome more comparisons.

BEAM/OTP is rather different from CL:s image-based REPL experience.

You can get relatively close by putting your application into processes and keep them in a collection or registry, then when you've found an issue and changed the code in your editor you can run recompile and relaunch affected processes. You'd lose a subtree in the application, and hence in-memory storage tightly coupled to those processes, but I find it rather easy to do it this way. Often you'll put in-memory data in separate caching processes and that reduces friction in restarting processes with application logic even more.

The 'let it crash'-foundation of the BEAM system isn't exactly tailored towards trapping and recovering from errors but if you really want to you could probably implement something like that with try/rescue but I suspect it would turn out quite obtuse compared to the CL mechanics.

> This describes fairly well what we've seen so far. The "lowtag" is those 4 bytes at the end of Lisp objects; also, we're clearly on a 64-bit architecture here, with N_WORD_BITS being 64, 8 of them reserved for the lowtag. It also details various values for the lowtag, the important one being...

Shouldn't that say bits? As 8 bytes would be the whole thing ayways.

A lot of people write Lisp interpreters, post it to HN and call it a day, but I think compilation is the far more interesting task for a Lisp implementation.

SBCL is great to poke around in and the decompiler’s great. Also check out ABCL, ECL for what compilation to JVM bytecode and C looks like.

The scheme compiler in Paradigms of Artificial Intelligence Programming by Peter Norvig is a great introduction to compiling Lisp. I was particularly impressed by it only taking a change in one line to go from regular recursion to handling tail call optimization.

This is a bit Haskell oriented, but the "Implementation of Functional Programming Languages" by SPJ is one of my all time favorite books.

https://simon.peytonjones.org/slpj-book-1987/

These compilers were pulling heroic data structure optimization stunts in 1985, that no modern production compiler for major compiled languages can replicate in 2024.

Why? Is it mostly because of Haskell's purity (side-effect-freeness), so optimizations can change expressions more freely and safely?

That, and the uniform s-expr/gadt representation of data structures instead of going wild with clever pointers.

I am not sure I understand what you mean in the context of Haskell.

https://wiki.haskell.org/GADTs_for_dummies

Huh – I didn't know he'd left MSR for Epic Games.

He did. He’s working on a new language called Verse now:

https://dev.epicgames.com/documentation/en-us/uefn/verse-lan...

Huh interesting.

The language seems sort of complicated for the domain (gameplay scripting).

Any explanation for the name?

Steel Bank Common Lisp (SBCL) is a fork of Carnegie Mellon University Common Lisp (CMUCL).

University founders Andrew Carnegie, Andrew Mellon, and Richard Mellon are best known for their work in the steel industry and banking.

  remember that we're on x86 here, which is a big-endian architecture

While I wish this were true, x86 is little-endian.

The author correctly states that x86 has "lower bits at the beginning, bigger bits on the end", but, as you said, that's the definition of little-endian.

Apparently, I even made up an explanation for why "big-endian == start with the least significant bit" totally makes sense... couldn't convince the Universe this way either though.

Thanks for catching it, fixed!

It's a bit sad that this ecosystem has fallen out of favor for mainstream software development.

One could have designed OOP like frontends to this fairly easily, and I'm sure there are existing ones. How did Java and C++ take over?

I’ve tried multiple times to get into SBCL, but every time I run into the problem that I have to learn Emacs first.

I know there are lots of other ways to work with SBCL, but they are either second-class citizens or paid.

Vim plus sbcl has been ok. Follow this I am happy since

https://susam.net/lisp-in-vim.html

Not 2nd class and I can use it mostly on mac; linux and windows (linux side)are ok. All good.

Why would you have to learn emacs? You can largely batch compile and run lisp code same as any other language, if that is what you want to do. In many ways, the workflow can be very similar to python, if you don't want to get too advanced in the repl. And, to stress, you don't have to get too advanced to get going.

Simply using rlwrap with sbcl is workable.

rlwrap is a great tool to have in your toolbelt anyways

You can use emacs just for the REPL/debugger, and another editor for editing files. I did this for a couple of years, after previously using CLISP for its REPL. You can use the menus for things; the only keyboard shortcut you MUST know is CTRL-G (which if you hit enough times will cancel whatever operation you accidentally started).

Alternatively try geany-lisp[1], it was created specifically for those with emacs phobia. There are probably some sharp corners using it, so feel free to file issues if you run into any. It should work with Geany 1.26-1.38; when posting this, I just noticed that there is now a Geany 2.0; I'm assuming the major version bump broke plugins so it may not work anymore.

1: https://github.com/jasom/geany-lisp

You don't strictly need to learn Emacs, but you should probably just join the cult. I bit the bullet and did it about 20 years ago now, and it's been a good thing. Heck, I've got a non-programming dayjob now, but I've still got a keyboard that puts the control key under my thumb: https://x-bows.com.

I'm in the same boat. I just don't care about EMACS, don't like it, and there's little incentive when I'm just playing around. It's a shame because Lisp is really fun and interesting. I just don't want to spend so much time dicking around with an editor just so I can write some code.

Try neovim + vlime!

There is Lem too.

https://lem-project.github.io/

I really want to like lem, so I followed all the steps at both https://lem-project.github.io/installation/sdl2/macos/ and https://lem-project.github.io/installation/ncurses/macos/.

Neither ends in a working development environment or even text editor.

Compare this to learning Python or Javascript or the C family. VSCode or even just your standard OS text editor is all you need. You can decide to start learning, find a tutorial on Youtube, and execute your hello world in 60-120 seconds.

Paul Graham used VI and clisp when he built viaweb.

It's because of his continual praise of lisp that I wanted to check it out.

What I'm complaining about is that it's much harder to "check it out" than it perhaps could be.

At one point many years ago, I wanted to check out this new Python thing. It was so easy, I've been hooked ever since.

> It's because of his continual praise of lisp that I wanted to check it out.

I can't argue with that. It did take me a long time to get comfortable with Lisp. But I felt it was worth the efforts.

> At one point many years ago, I wanted to check out this new Python thing. It was so easy, I've been hooked ever since.

IDLE is a underrated feature of Python. The default install includes a barebones IDE together with the REPL, making setup for beginners nearly zero effort. On the other hand, in lisp land, step 0 to learn most lisps is to learn emacs beforehand.

I use slimv with vim, but basic Emacs is not difficult: Ctrl-x b to swich betwen buffers, Ctrl-x s to save, Ctrl-x c to exit, Ctrl-c Ctrl-c to evaluate the CL buffer under Slime, Ctrl-x e to eval most Elisp and CL code functions.

Also, knowning CL makes learning Elisp a breeze, so you can customize your editor like nothing else.

Learning Emacs is not a bad thing you know..

It has a lot of other goodies like : Org, Magit etc which makes it worth while.

I agree that it's a hurdle.

Portacle, https://portacle.github.io/ , is a way around config and whatnot, lowering the threshold a little.

I normally dabble with CLISP (notably because of its built in readline), and if I could SOMEHOW get it to build with SSL, I'd play with it more.

This is what I stick at the top of my Lisp files when dabbling.

  (defun l ()
    (load "file.lisp"))

  (defun e ()
    (ext:shell "vi file.lisp"))

And then I just muddle my way through with a (e) and (l) cycle. Since I tend to not work on 10,000 line files -- this works fine (my files are < 5K lines). More than fast enough, and I get to retain any work variables within the image (plus my command history). It's quite effective really. (print...) debugging works because you can run your code at whatever granularity you like (and you can always refactor it to a finer grain if you want). Don't doubt it until you try it. Turn around is very fast.

That said, I discovered that for some reason, firing off something as simple as "vi file.lisp" from SBCL is stupifyingly complicated. I made several 10s google searches and 5s GPT attempts to make that work, but gave up.

So, can't say I can recommend that style of development in SBCL unless someone is willing to chime in with the, apparently, paragraph of code necessary to launch vi on a file.

It's not that bad:

  (require 'uiop)
  (defun c ()
    (uiop:run-program '("emacs" "-nw" "file.lisp")
                      :output :interactive
                      :input :interactive))

EDIT: note that SBCL comes with UIOP included by default, but if you want to use SBCL's implementation-specific facility, it's only slightly different:

  (defun c ()
    (sb-ext:run-program "/usr/bin/emacs"
                        '("-nw" "file.lisp")
                        :input t
                        :output t))

If you don't want a full emacs, jmacs from 'joe' an do the trick.

The plugin in Pulsar (Atom) is pretty good: https://lispcookbook.github.io/cl-cookbook/editor-support.ht... (Sublime, Vim, Jupyter notebooks, VSCode (OKish), Eclipse (simple)…)

> I’ve tried multiple times to get into SBCL, but every time I run into the problem that I have to learn Emacs first.

I have the same problem with Hylang.

Java took over by targeting large teams of inexperienced developers.

C++ took over by adding OO to C with zero cost abstractions, and then evolving into modern C++ (which discourages OO and encourages composition via template instantiation) without leaving programs behind.

Lisp has supported composition forever (it is one of the big advantages of functional programming), and the macro language is similar to templates. I think the big problem is that it is difficult to scale lisp projects with inexperienced developers (loses to java) and is not zero cost (has a GC, is often interpreted; loses to c, c++ and now rust).

> is often interpreted

Every competitive Common Lisp implementation has compilation. SBCL defaults to not even using an interpreter for eval.

SBCL eval interprets until you hit a lambda form, at which point it will compile (last I checked). But you are much more correct than who you replied to :)

Unless I'm misremembering, this is only true in REPL, which should be better read as "SBCL is actually sometimes *AoT-compiling in interactive REPL sessions!!!"; #'LOAD-ing files would AoT-compile them by default.

This is reminding me there is the sb-fasteval feature, which I have not been testing.

It didn't use to - for various reasons, evaluator inherited from CMU CL had bitrotted at one point, and for many years every form more complex than essentially single function call resulted in compilation.

Few years ago there was work to fix the evaluator and now EVAL has limited evaluation back.

And CCL doesn't even have an interpreter. Every form you type into the REPL is compiled. If you want an interpreter in CCL you have to write it yourself.

worth mentioning is that CCL is finally back into active development !

Who is doing the development currently?

Matthew Emerson

https://lists.clozure.com/pipermail/openmcl-devel/2024-April...

I thought SBCL was a compiler only implementation? CMUCL did have an interpreter though.

It was until someone resurrected the evaluator it inherited from CMUCL

CMUCL's interpreter evaluated IR1 (the first intermediate representation of its compiler) IIRC, so it wasn't possible to have a truly compilerless CMUCL _and_ a functional EVAL. I believe this IR1 interpreter was dropped from SBCL very early on. When SBCL gained an interpreter again it was a simple metacircular evaluator a la SICP that was unrelated to anything inherited from CMUCL. (This is all as of 15 or so years ago, I'm sure things have evolved since then!)

SBCL has two evaluators now, sb-eval and sb-fasteval. I don't know how much structure they share. By default it builds with sb-eval, but this can be changed with options to make.sh. sb-ext:*evaluator-mode* is still :compile by default.

> I think the big problem is that it is difficult to scale lisp projects with inexperienced developers (loses to java) and is not zero cost (has a GC, is often interpreted; loses to c, c++ and now rust).

i think scalability problem is a bit overstated. the biggest issue with lisp becoming popular is that it doesn't have a large user base and is not a new language like rust. however despite being old, common lisp /still/ has so many 'novel' things in it that in metrics of programmer ergonomics and programming user interface make common lisp a big winner for me.

as regards zero cost, lisp programs can be made arbitrarily close to being zero cost, but here be deamons.

The big team thing matters for two reasons:

First, it implies lots of people are paid to learn it. Second, it implies companies will have to invest in adding/fixing support libraries.

Well take over is relative. I built my startup using SBCL and it is very profitable. Couldn't have done it without SBCL.

>Couldn't have done it without SBCL

Could you elaborate this part?

I still haven't found an environment which lets you build the system as seamlessly as SBCL. The REPL allows you to build the code at runtime and update the system in place. Since I was a solodev, bug fixing and building the thing without going on the wrong path was worth I think months and years of my time.

edit: Once you get the idea of a running Lisp image and you are updating the image it's a game changer. You are not writing a program, then compiling it and then running it anymore (even though those steps happen seamlessly).

> image and you are updating the image it's a game changer.

Also Smalltalk.

https://lists.cuis.st/mailman/archives/cuis-dev/2023-August/...

Also Factor.

https://factorcode.org/

Not obvious to me that it provides the same experience: "The image is a map of the memory after the code was loaded. Unlike in Smalltalk, Factor code is always distributed in files rather than in the image."

https://concatenative.org/wiki/view/Factor/FAQ/Install

Factor provides support for both file and image based workflows. Factor is image-based and uses files when sharing, loading or refreshing vocabularies. however, just like lisp and Smalltalk, you can also save, share and restore system state via an image. For example you can save the whole state of Factor with the word

save-image

and you can restore that image using ./factor -i=path-to-image

This image workflow and interaction via the inspector provides a very similar workflow experience to both Smalltalk and Common Lisp.

It's likely to be more obvious if you tried it out.

So a jupyter notebook is just a bad lisp repl

In my opinion it’s indeed like Greenspun’s 10th rule; it’s a vague imitation. If people weren’t so squeamish about parans the world would be a nicer place. The tech is there for a long time already.

I love lisp. I love the stability, the macros, the crash handlers, the repls, the elitism, slime, emacs, the works. But IMO sexpr is straight out terrible and hacking around that is a chore.

Yeah, it's interesting. I find them elegant and want all my languages to have that syntax (no deviation either like [] for lists etc). Guess we cannot argue about taste :)

I suspect most of the power that you already _love_ flows from the sexp based homoiconic syntax :-)

I tried to convince myself of that, but putting a function/operator in the same parens as the arguments and then making a special case for the first element in the parens will never make sense to me, since you can get the "expressive power" in any number of ways. Ruby makes more sense.

Not sure if I'm up to date, but jupyter (as good as it is) suffered from strange rules of evaluations for cells which makes it difficult to do anything large since side effects will impede thinking quick.

This is what the lisp repl is like. There's not much else like it: https://www.youtube.com/watch?v=_B_4vhsmRRI

PS: You can also do copy and paste :)

Same here. It is just really annoying to me (ymmv etc etc) to use another env: it just is so much harder to inspect everything, debug easy and still end up with a really fast system because of sbcl.

What about state belonging to the older code, updating http handlers, user sessions, etc, on production? While living-in-the-image seems admirable, I always wondered how it blends with real production and real ops.

The only thing I heard about this is Paul Graham's story on fixing particular bug in viaweb while talking to the user, but that's just one story from 90s. Would be great to hear more with regard to more modern setup.

Could you share the name (and what it does)f your startup?

Let’s make a trip back to the early and mid 1990s when C++ was a young language and Java was under development (it was released in 1995). At the time the most popular object-oriented language was Smalltalk, Common Lisp was at its peak of commercial popularity (largely in symbolic AI niches), and there were other niche OO languages that were heavily influenced by Smalltalk and Common Lisp (Objective-C in the NeXT world, Apple’s Dylan under development).

However, if I know my history correctly, Smalltalk and Common Lisp implementations were generally not cheap. Objective-C was largely tied to NeXT, which required buying into a niche ecosystem. Dylan was a casualty of Apple’s mid-1990s business struggles.

C++ benefitted from inexpensive implementations from vendors such as Borland and Microsoft, and it also benefitted from its close relationship with C, which already gained a foothold in the 1980s.

Java benefitted not only from Sun’s marketing machine, but also from its free compiler and runtime from Sun. Java overtook Smalltalk, despite the fact that Java lacks Smalltalk’s dynamics.

I think had there been solid cheap or free Smalltalk and Common Lisp implementations around 1992, Java wouldn’t have gained a foothold, though C++ would have still been very appealing to C developers, though perhaps if Objective-C were more broadly available to non-NeXT developers, it would’ve been a formidable competitor to C++ in terms of providing a “C with objects” environment.

>I think had there been solid cheap or free Smalltalk and Common Lisp implementations around 1992, Java wouldn’t have gained a foothold, though C++ would have still been very appealing to C developers, though perhaps if Objective-C were more broadly available to non-NeXT developers, it would’ve been a formidable competitor to C++ in terms of providing a “C with objects” environment.

100% agree. i think it is also interesting that in comparison to before lisp community of today is so supportive of free software, standing more on the left wing of the open source community.

Interestingly enough, the free software movement came about due to Richard Stallman’s frustrations with Symbolics, one of the Lisp machine companies that came from the same MIT AI lab he worked in. It’s just that RMS chose to reimplement Unix (GNU) instead of creating a free Lisp operating system, though one could argue half-jokingly and half-seriously that GNU Emacs is that free Lisp OS.

I understood it came about due to RMS wanting the source for the printer driver for the Xerox 9700 - nothing to do with Symbolics.

The printer driver situation was RMS’s last straw, but he was already saddened by the demise of the sharing culture of the MIT AI lab once Symbolics and Lisp Machines, Inc. were founded, and he spent a considerable amount of effort reimplementing features from Symbolics:

https://www.gnu.org/gnu/rms-lisp.en.html

There's a lot more to the history.

My favourite quote from some other participants was "he didn't see us at lunch not because AI Lab culture died, but because we avoided him"

Source?

I was about to say that, point to point. But, think about that. Unix was the cool thing from the 80's, so even if ITS was a hackers' dream, mixing Unix' utilitarianism with Emacs' Elisp hackability on top of that wouldn't be so bad under a proper GNU kernel + Emacs userland system.

There's basically one commercial Common Lisp left (Allegro from Franz), and they're focusing more on things built on top of the Lisp (like AllegroGraph), not the Lisp itself. So the free CLs have sort of won by default, and perhaps also by sucking the oxygen out of the room for expensive proprietary CLs.

Do you not count LispWorks in this? I haven't had a chance to play with the commercial Lisps myself, but anecdotally it seems to still have some currency.

I didn't, but perhaps I should have. There's a question of whether it's live enough to count, or if it's just in maintenance mode. The last release was in 2021, but there have been gaps that long before that.

Common Lisp hasn’t changed a whole lot since 2021. And it supports Apple Silicon, and it even has, apparently, a Mobile runtime.

I imagine that’s ticking the boxes for most of their users, and they’re not in the endless “security update” treadmill.

compare the last release dates

    2012 Allegro CL  9.0
    2015 Allegro CL 10.0
    2017 Allegro CL 10.1
    2024 Allegro CL 11.0

    2012  LispWorks 6.1
     2012  LispWorks 6.1.1
    2015  LispWorks 7.0
    2017  LispWorks 7.1
     2018  LispWorks 7.1.1
     2019  LispWorks 7.1.2
    2021  Lispworks 8.0
     2022  LispWorks 8.0.1

Looks similar to me. LispWorks also had patch releases in between. Both are on the market for more than 35 years. Both are mostly written with CLOS and thus are especially to update with patches, additionally to the usual ways to update code (-> late binding). One just loads patches (which are mostly compiled Lisp code) into a running Lisp and that's it. Alternatively one can save a new image with patches loaded.

When one needs a patch or a feature, one would typically contact them directly. Both provide patches to the users.

Franz has made that simple for the user, they have a relatively continuously stream of patches, one can call an update function and it gets the necessary patches and installs them.

SBCL has monthly (!) releases, where the user (that's what I do) would typically compile it from scratch using the supplied sources. Updating is quick, around a minute for a recompile.

LispWorks still exists and works great.

The economics of this is interesting to hear.

GNU Common Lisp (https://www.gnu.org/software/gcl/) seems to have existed in various forms from the mid 1980s, but it was probably outclassed by the contemporary big tech companies.

Gnu CL also had problems as a Common Lisp implementation. It works (like ECL, which is related) by compiling to C, which is then compiled with a C compiler and the object code loaded into the image. This approach generates inferior code to that which an implementation like SBCL can achieve.

All the Lisp implementations needed more memory than C programs would, which limited their impact before 32 bit machines became standard. And by that time, C was strongly in place.

There's arguably also the part where it was, AFAIK, mainly driven by needs of Maxima and it's declaration (or so I recall) of focusing on CLtL first didn't help things either.

In comparison, ECL which forked from the same family seems to work out much better over time.

Another big user was ACL2, but that's been ported to SBCL (meaning, standardized enough to run on SBCL).

GNU clisp it's still alive, but I had troubles getting compiled under 32 bit with threads support needed for Bordeuax-Threads from quicklisp.

Well, there is this: https://www.norvig.com/java-lisp.html

Java claim to fame was to be close enough to C++ but portable in binary form. This made it possible for millions of people to learn it quickly. Lisp is too distant from C++ to have had any change.

Which is funny because it was marketed as close, while in reality being quite different from C++: all methods virtual, interfaces, type erasure, no method overloading, no operator overloading, no templates, to name just first things coming to mind.

We've been fooled in 90s so easily!

For entry-level programmers making the transition to Java, those deep semantics like "all methods virtual" didn't matter, nor did the lack of features like operator and method overloading or templates. They didn't really need those to build stuff.

The killer "like C++" feature that made adoption easy was the "C/C++ like syntax, with braces and everything".

> for mainstream software development.

LISP feels like both an "assembly language" and a "higher level language" at the same time. It's excellent for solving unusual problems, but it carries a lot of baggage when aiming for "usual" problems.

The big problem for Lisp is that it was too slow during the 80s and 90s. At that time, C and C++ took the market share and maintained it since then. New languages had to conform to C, since Lisp has its own independent way of doing things.

Reply on news.ycombinator.com