Common Lisp's block / return-from and unwind-protect
axisofeval.blogspot.com
axisofeval.blogspot.com
You can return from a function from within a lambda (and I believe this works on any number of levels), for example:
fun go(list: List<String>): Boolean {
list.forEach { if (it.isEmpty()) return@go true }
return false
}Smalltalk, and I believe ruby, allow non local return from blocks.
unwind-protect is a more general form of the c++ raii (in fact c++ got raii from Common Lisp).
I think that MSVC++ longjmp actually does proper unwinding, calling destructors in C++ functions on the stack, but don't quote me on that. I think it is also dependent on the compiler flags.
I think that RAII is different from unwind-protect and other scope based cleanup (finally, defer, with) as it is tied to object lifetimes. The fact that automatic objects lifetimes are tied to scope is a nice feature, but RAII goes beyond that.
I wouldn’t say RAII is “tied to object lifetime” except in its name; at least I think of it as every {} pair defining an unwind-protect with object creation/destruction being how it is effected by the programmer. Perhaps that is the same thing as you said, simply viewed from opposite sides.
I do like the automatic nature of RAII, though implementing a whole class for it always feels clumsy to me even after doing it for decades.
This is a low-level memory management strategy being conflated with object-orientation, that is alien to higher level languages.
Yet I think an high level language could have the same value/reference distinction. Or you could do it with linear types.
For example WITH-OPEN-FILE closes the file automatically when leaving its scope.
(with-open-file (stream "hello.world")
(read stream))
Thus anything where a destructor would automatically clean things up is done in Lisp behind the scenes automatically using an UNWIND-PROTECT form.
But the essence of RAII is more than that.
In c++ if I add an opened file object to some collection and later remove it or destroy the collection, it is implicitly cleaned up.
For example let's say you are implementing an n-way out of core merge: in c++ you would create a priority queue of files objects ordered by the front current from item. You pop the front file object read the item and, if the file is not empty, push it back into the queue. Cleanup is implicit by removing the file from the queue and not explicitly adding it back. On early exit (because of an error or exception) the queue is automatically destroyed and recursively all the file objects.
There might be a way to implement this recursively with unwind-protect, but I think it is less natural.
For example python has ExitStack, which in practice is an ad-hoc container that supports recursive cleanup of contained objects, but is still not as convenient as having all containers do proper cleanup.
I know and understand this is true.
Can you point to any sources (projects, papers) that further substantiate this?
The concepts of constructors and destructors have been introduced by C.A.R. Hoare in November 1965, in "Record Handling", a proposal for the extension of Algol.
At that time, Hoare was using the Cobol terms, i.e. "record" for what later was named "object" and "record class" for what later, in Simula 67, was abbreviated to "class". All the "records" discussed by Hoare were allocated dynamically, in the heap.
Constructor (Hoare, 1965-11): "In order to bring records into existence in the first place, the record class identifier should be used as if it were a function designator"
Destructor (Hoare, 1965-11): "a standard procedure "destroy" is proposed, which takes as parameter a reference to a record, and which reverses the effect of having created that record"
The next step towards RAII has been done by Bjarne Stroustrup in "C with Classes" in April 1980, when he made the invocation of the destructors implicit at the exit from a block, by introducing the special member function "delete" for this purpose. Despite the name, the 1980 "delete" member functions corresponded to what later, in C++, were renamed as destructors.
So in 1980, RAII was complete, but it was not yet promoted as a universal strategy for managing resources.
In 1980, Common Lisp did not exist.
Most older Lisps did not have any concept similar with the Algol block, so it would have been impossible for them to invoke implicitly some cleanup functions at block exits. They relied only on the garbage collector, where there is no RAII in the Stroustrup sense, even if GC and RAII are alternative methods for avoiding the explicit invocations of "free", "close" and the like.
In the Lisp Machine OS there is a concept called RESOURCE for manual memory management. For example the CHAOS network stack has to deal with network packets. There is a macro USING-RESOURCE, which allocates/gets an object and on exit frees it. The macro expands into a form using also UNWIND-PROTECT to ensure freeing a resource on non-local exit. I would think that this is from around 1980, for the MIT CADR machine.
http://www.ifarchive.org/if-archive/infocom/info/MDL_Primer_...
Nevertheless, the MDL UNWIND and the later UNWIND-PROTECT are more limited in applications and they require much more work from the programmer than the mechanism introduced by Stroustrup in 1980.
With implicitly-invoked destructors, the destructor body is written once for each type of data, and normally there is no need to ever invoke it explicitly.
After writing correctly the constructors and destructors, the programmer's work becomes identical with using a garbage collector, because the objects are allocated explicitly, but they are never deallocated explicitly.
On the other hand, UNWIND is intended for handling exceptions. It can also be used as a normal cleanup strategy, but it still must be written every time for handling the exit from a block or from a hierarchy of nested blocks. In the latter variant, there is some economy in code writing, but the lazy deallocation is less efficient.
The UNWIND of MDL has little resemblance to RAII, but it resembles the UNWIND of Mesa (programming language used at Xerox, starting with 1976, which has introduced many innovations that have been included only much later in most programming languages).
It would be difficult to determine whether UNWIND has appeared first in MDL or in Mesa, or if both have taken it from another language, because experiments with exception handling were fashionable during those years and there were many places where various variants were tried.
Resource management in the MIT Lisp OS ca. 1980, approximate example.
One would define a resource of arrays, where arrays can be allocated and deallocated. They will be managed via a pool.
(defresource 2d-array (rows columns)
:constructor (make-array (list rows columns)))
(using-resource (my-array 2d-array 100 100) ; get me a 100x100 array from a pool
(setf (aref my-array 42 42) 'the-answer) ; setting the array
(print (aref my-array 42 42))) ; reading the array
...
(unwind-protect
(progn ; protected form
(setf my-array (allocate-resource '2d-array 100 100)) ; allocate the array
(setf (aref my-array 42 42) 'the-answer) ; setting the array
(print (aref my-array 42 42))) ; reading the array
(when my-array ; exit form
(deallocate-resource '2d-array my-array))) ; deallocate the array
...
Thus the user will not explicitly use UNWIND-PROTECT, but some macros which use it in their expansion...
I agree that this is pretty much equivalent to RAII.
Nevertheless, it is also obvious that this was not a source of inspiration for Bjarne Stroustrup.
He has started directly from the constructors and destructors of C.A.R. Hoare (1965-11) and Simula 67 (1968-05, Kristen Nygaard & Ole-Johan Dahl).
The only change is that in 1980 he has enhanced his compiler for "C with Classes" to generate automatically all the invocations to the appropriate destructors in all the block and function epilogues, relieving the programmer from this task.
In relation to all this, in POSIX, there are often good reasons to use sigsetjmp and siglongjmp rather than setjmp and longjmp, because these also save and restore the signal mask. If you jump out of a context that locally disabled certain signals, you likely want them restored, like you would if that code returned normally. (It doesn't necessarily have to be a jump out of a signal handler!)
Anyone here have any recent practical experience in this direction who would confirm this in Lisp vs in other programming languages? Does effort in Lisp really compound/produce great code and great applications that much faster than other languages? (For web development primarily)
I feel that the lisp code was easier to write and reason about, the being able to hot reload code from the repl significantly decreased the time to completion. I also 'connect' to a networked repl when sentry reports an unhandled error to figure out what has gone wrong.
You might be tempted to believe that CL repl is "similar enough" to python's repl, however this is NOT the case. Being able to redefine functions, variables and macros while working on the code (without a restart) allows you to deal with errors.
The syntax is a 'no brainer', Extreme consistency in function calls means that you don't need to think about it. Other languages which SOMETIMES use infix, sometimes require brackets, that is just crazy.
Lisp libraries have less churn than 'modern languages', some libraries have not been touched for some time, unlike python/ruby/js. The code does not seem to rot and old lisp code runs on modern implementations.
I work in emacs, lem, I know people who use vscode and alive, and vim. There is no 'hard' requirement to use emacs, you will get by as long as you have 'emacs like' repl integration.
All in all, I do not regret working in lisp. It doesn't have the cult of the other languages and I'm fine with that.
The only other language like that I can think of is smalltalk.
I work in LEM for my personal projects, that dont have other LSP integration needs and hooks into my org-mode/org-babel workflow. These should not be show stoppers, but will eventually be a non issue.
There are however some features that I'm missing in LEM before I can fully switch over.
1. The documentation for Lem is lacking. 2. I have lazy hands and hit shift on some keys when I shouldn't (shift and backspace for example). I'd need some way to ignore that. 3. An built in undo, (c-x u equivalent). 4. A documented library interface (this is kinda #1 again)
I will eventually get around to making my own plugins to solve most of these. Its a matter of time vs tradeoff.
Download micros, put it in your local-projects
git clone https://github.com/lem-project/micros
Load it.and run.
(micros:create-server :port 50000 :dont-close t)
Then use m-x slime-connect in lem.
It is not slime, but it works.
We have a choice of libraries, but no big batteries-included web framework, so you must be ready to assemble pieces (or simply, to know web development). Ex: a login system? There are examples to look at, not a use-and-forget library.
A CL webserver coupled with a client library like HTMX is great. I don't recommend Reblocks (weblocks) or CLOG yet for serious stuff.
resources: the Cookbook, my lisp-journey #web content, my last youtube video (@vindarel channel).
SBCL is blazingly fast, CL abstractions are profound yet practical. The type system gives you power without pettiness. Language-oriented programming is a superpower in a world of cut and paste library-oriented programming. Image-based computing often allows one to skip the impedance mismatch with DBs. In many cases and you can just persist the data structures as they are or serialize them where needed. The tooling with Roswell, Quicklisp, Qlot, Doom Emacs, and Sly make prototyping and exploratory programming wiz by. CLOG, Reblocks and Hunchentoot provide powerful abstractions for web programming, but Phoenix with Liveview is probably better for most. The debugging experience is so insanely powerful it freaks me out and makes other systems look rudimentary. CLOS, the object system, is best of breed and makes most of the standard criticisms of OO programming moot. The homoiconic list-based syntax and prefix notation are beautiful and never get in your way. Parentheses are invisible. The condition system makes flow control and error handling in other systems seem rigid and convoluted. Aspect-oriented programming is just what we do, not a thing tacked on. Macros and the meta-object protocol allow you to build a language on top of Lisp that allows users to conceptualize problems using nouns and verbs they already know.
Drop into Coalton or Shen for strong, static typing where necessary, or April (hosted APL) for array wizardry that makes NumPy look like data-flow programming in Basic on a Timex Sinclair with a cassette drive. You can slice and dice whole datasets in a few characters of code. Bundle existing C libs with CFFI when you find a good one you need. Lisp-based systems are alive and grow and change as requirements do. It's harder to program yourself into a corner. Refactoring is continuous and mundane. In-source documentation is the right place for it, but Lispers tend to be shamefully lax in providing it. Shaaame! Newer Lispers tend to be better about docs--the culture is changing. Often you ask a question and get no response. Other times you get a detailed response from a legendary mind and just sit there in awe working your way through their thinking and feeling your inadequacy. The code is about as succinct as it can be to maintain readability. There are two camps on readability, some experienced Lispers find it harder, and most find it much easier to parse than ALGOL-derived syntax.
Once mastered, quick things are quick, hard things are straightforward, and impossible things are doable. This is a system for people who need, not want, to change the world. It gives single programmers and small teams the power to take on armies of bureaucratic programmers. But you have to be willing to break the mold, handle charges of elitism, not be required to justify your choice to managers, tolerate a small hiring pool, train newbies hard and long, not get freaked out using some libs that are stable and haven't been updated in years, deal with the pace of development of a small community, have the patience and joy in problem-solving to handle frequent bafflement, read a ton of source code rather than search the forums, and be willing to roll your own when needed. If you can handle all the idiosyncrasies and have a burning need for elegance and power, then CL will reward and surprise into until dementia takes you.
In short, all the good things you've heard are true, many of the standard critiques are uninformed, and problems people talk about that apply to most languages often don't apply.
Do you have a blog where you share your learning experiences?
I don't have a blog and haven't gotten far enough in my learning to make anything useful, but I do plan to produce a newbie's guide to help people get started quickly. My perspective as an older person for whom CL is my first language could be valuable since my sticking points are different and I don't yet have bad habits to break.
So far the language I've experienced the most power with is Python, and your description of Lisp's power and tooling are very intriguing. I will take note of your recommendations and play around with the tools you described. It is worth the investment if it means unlocking something even more powerful than Pythonic thinking and code.
Do you find that one needs to be well-versed in Emacs to fully appreciate the CL stack you described and the capabilities of the language itself? Would you learn Emacs separately/after spending time building things with CL, or is building things with CL in Emacs part of the entire experience?
Emacs is the weakest link in the Common Lisp experience but, yes, you have to defeat that boss to move up. I love/hate Emacs and mostly struggle to do anything. GNU Emacs is a ball of mud with decades of technical debt in dire need of a ground-up re-imagining. Emacs Lisp made a lot of inexplicable choices but it is a Lisp and Lisp is why Emacs survives. RMS' stewardship has been consistent but underwhelming. He's no Jose Valim. The editing model is inadequate, as shown by Kakoune and structural editors. And yet, Emacs is alive and malleable while so many other editors/IDEs are inorganic artifacts that start to rust as soon as they are released. We do have the nascent Alive for VSCode and VIM options, but they are not really improvements over Emacs, nor are they likely to be. VSCode is the latest CodeWarrior and will suffer the same fate. It's only propped up by daddy's money.
Doom Emacs tooling, package management, and sensible defaults take much of the pain out of Emacs. Sly, the LSP-like Emacs mode for CL, is too much fun and gives a hint of what Lisp Machine development must have been like. Remote editing and daemon mode are so practical. Once you've painstakingly hammered out a config and workflow, Emacs starts to hum along pleasantly. But getting there is relentlessly frustrating and trying something new sends you right back to the treadmill. Once I figure out why Lem SDL2 (an Emacs clone in Common Lisp) isn't building for me, I'll attempt a switch.
I have daily fantasies of jumping back to my high school graduation in 1986 with everything I know now. I'd put up sexy posters of Guy Steele and Alan Kay in my dorm room. The only real advantage of starting programming at my age is that I know what I want and why. I'm following the humanistic thread of computing from Paul Otlet -> Vannevar Bush -> Doug Englebart -> Ted Nelson -> Bret Victor. But I read the same chapters over and over and still haven't written any code beyond homework exercises. The task feels Sisyphean and my trauma-addled, post-alcoholic brain struggles to stay focused. My only consolation is that I can't think of anything better to do with my remaining time and the Common Lisp community is good company. I love being around eccentric people who are much smarter than me.
Python is an eminently practical, perfectly respectable choice. So naturally, my life goal is to kill it. In my head, Symbolics was open sourced after the AI winter and grew to become the obvious choice for most tasks. I wish that people wrote new languages as DSLs on top of Common Lisp rather than littering GitHub with vanity projects that die on the vine. Think of all that wasted effort--like turning up the thermostat with the front door open.
When people ask about the choice of Common Lisp or Scheme, I ask which Scheme? No two are alike and none of them has achieved enough momentum to be truly practical. You have to pick one and pray your requirements don't exceed it. In CL, we invest in implementations--ABCL, ECL, Clasp--each one practical, each one mostly runs your code unchanged. Our spec is ancient by today's standards so old code just works. We innovate in our package ecosystem which gives users of whichever implementation access to fresh work. Compare to the whitewater churn of the NodeJS ecosystem. I have a Ghost blog that's a few years old and I can't update it as things have changed so much there is no practical way.
I'd like to file off the most common pain points in CL, spackle over obvious gaps in functionality, improve documentation, and polish the developer experience until nothing stands between Lisp and world domination. It's the once and future language. I have no qualifications for the job whatsoever except resentment of the status quo, a burning need for elegance, and lack of employment. It's a short resume, but the nice thing about being self-appointed is that no one has to want to hire you.
When I found functional programming in Erlang I got SO excited. When I found Common Lisp, decades of pent-up frustration melted away and I knew I'd come home. I guess my Unix Hating led to some intuitive thinking over the years and I've been making up for lost time over the past couple of years and mapping what I knew I wanted to what has been there in CL all along.
In this vein, it’s great that decades-old Common Lisp books are still current. The paucity of Web content is not as bad as it seems: you can get a real book.
Your comment left me with a strange sentiment. I'm somewhat disheartening because it implies that learning Lisp might be redundant given that contemporary languages have already incorporated its best aspects.
Yes. See effect handlers for an example, which are making rounds around the programming world as of late. They are equivalent to the Lisp condition system, except formalized to work in strongly statically typed programming environments.
> I'm somewhat disheartening because it implies that learning Lisp might be redundant given that contemporary languages have already incorporated its best aspects.
Sort of, lots of things have thankfully trickled from Lisps to other languages (including whole languages like Julia). The pleasant feeling of conversing with the language and programming bit by bit in it is hard to replicate with things like LSPs, though, since the implementation is always running in the background and programming in it is based on mutating it until it contains the program you seek.
Not exactly correct due to the lack of higher order effects (https://news.ycombinator.com/item?id=20513370), but the condition system is "good enough" for a lot of use cases
I'm also the kind of weirdo who thinks that the old and new testaments are baseline requirements for a human to be "literate" (along with Ovid and Aristophanes).
Despite these works having been recycled a large number of times, the originals still bear close scrutiny.
You'll never know what modern $proglang authors screwed up until you go back and read the fundamentals. For a concrete example, consider the Python module system, and the pirouettes you have to do to use it programmatically. Until you look at (just for example) how this works in CL and RS5/6S, you'll not have the context to readily apprehend the hack job in question (although if you have a modicum of taste you may wrinkle your nose regardless).
It'll also give you a good feel for what the true inventions have been since the early nineties.
Similarly powerful to having the compiler available at runtime, another thing trickling very slowly into mainstream languages.
"But why would you want this?! That sounds dangerous!"
It's quite dangerous, but so is a 700cc dirt bike.
"Can you give me any practical reason you'd want this?"
Aside from the freedom to redefine other folks' code (see: the derogatory term "monkeypatching"), you can leverage runtime namespace access and manipulation to, when loading a "fasl" into a running Lisp image, identify all classes that you care about and then upgrade them to new definitions without restarting the Lisp process.
Most mainstream languages completely punt on "deployment" (PHP [accidentally] aside, where you can joyfully edit production all day, a paradigm too many folks sneer at in the name of cargo cult professionalism), forcing the system operator to restart the process with new binaries (or pycs, whatever, I'm sure you can generalize the point).
A CL system supports that model, of course, but one can also precompile binaries to be loaded into the running system, redefining system behavior without downtime. Binaries, we should note, that don't include the entire Lisp runtime.
It's not that this provides some sort of day-to-day omg ergonomic bonus over the popular Algol families, but that I feel respected as a software author in a Lisp context in a way that I really don't in many other contexts ("nobody needs generics", to pick on some old low-hanging fruit), and the language providing interfaces to itself in itself is a delightfully low-friction cognitive model.
To return to the dirt bike analogy, I could prattle all day about this, but unless you're the kind of person inclined to spend five hours a week learning to enjoy riding dirtbikes, I don't know that I can spill enough ink to convey the fun of managing a few hundred cc's around the track at speed. Countersteering, inertia management, traction, all of these things are going to go over your head if your idea of a fun time with motive power is a Rivian.
But back to your examples - I feel that a lot of ideas CL is built on just aged poorly, I guess? No way I want to see some magical binary where some guru adjusted a parameter in the image or the ugly head of monkeypatching or macros all over the place in production code. CL serves the myth of lonely hacker where the rest of the world has accepted the fact that writing software is mostly a social activity and that code is as much for other people as for the machine.
byte-code enhancement in Java?
You'll find dynamic features in a bunch of other languages, too. R, Ruby, Python, JavaScript, ...
Note that I'm not strongly advocating anyone ship CL in hot loops in prod, merely that there are wonders in there worth examining, as perhaps the surviving mainstream descendant implementations missed a nuance or three.
It's really not a great fit for middle-of-the-road software teams with quotidien concerns.
It does have plenty of great food for thought for the thinking programmer.
Indeed, Lisp is no longer as revelatory as it once was, since many innovations have been adopted, even surpassed in specific cases. Still, no other language gives you the total package. The environment is far more than the sum of its features list. CL just gets out of your way and allows you to think about problems not programming.
Ah, and most relevant, many folks still believe that writing OSes in what was once a macro Assember for PDP-11 is the only way.
It was as kitchen sink as having POSIX (basically a full blown UNIX specification), to be expected to fill in the stuff missing from ISO C standard library.