Honest question, how is this different from what you can do in e.g. python? The python interpreter supports reloading of modules and evaluation of expressions, is there functionality that CL has above and beyond this that makes it more powerful?
Those are some examples off the top of my head, but the general point is that unlike mainstream languages, Lisp languages evolved into environment for interactive construction and modification on a continuously running program, which is a bit different way of working than the regular write-compile-run cycle.
I think it's not so much whether the features exist in other languages since we all know they can be added but the whole being more than the sum of its parts.
To me the thing that CL has that has yet to be matched by any other languages, REPL-based or not, static or not, is the amazing condition system and dynamic restarts for handling problems. For building and debugging systems this is an utterly amazing thing. It also allows composing exceptional conditions in a reasonable way.
Every other language I use (e.g., Clojure these days a lot) I constantly miss this amazing stack-walking ability and conditional restarts.
Every time I boot my old Symbolics computers and something happens and I get a GUI restart, I'm just amazed at what we have lost through two decades of non-CL living.
Heck, it's easier to handle recovering from an error by just restarting a process and letting it follow the normal unit logic.
Do have any knowledge of how CL and Erlang/Elixir/OTP are similar or vary on error handling?
When an error occurs in a Common Lisp program the runtime signals a __condition__. A condition is an instance of a class that represents unexpected or otherwise exceptional situations that can arise during execution. When a condition is signaled, control searches for a registered handler that matches the type of the signaled condition. If one is found then control passes to the matching handler. If none is found then control passes to the default handler, which by default starts an interactive repl called a breakloop__.
The breakloop has access to the full dynamic state of the suspended computation, including all local and dynamic variables and all pending functions on the stack. From the breakloop a programmer can alter variable values and redefine functions and classes. For example, if you conclude that the error happened because of an incorrect function definition, you can redefine the function and tell the breakloop to resume execution as if the new definition had been called instead of the original one.
A __restart__ in this context is an option for continuing execution. Common Lisp's condition system offers the programmer the ability to choose from among available restarts in a breakloop, or to write a handler that will make the choice automatically when a condition is signaled, and it also offers the ability to define custom restarts.
(defun fn1 ()
(if (yes-or-no-p) (print :yes) (print :no)))
(defun fn2 ()
(fn1))
(defun fn1 ()
(restart-case (error "yes or no?")
(yes () (print :yes))
(no () (print :no))))
(defun fn2-interactive ()
(fn1))
(defun fn2-automated ()
(hander-bind ((error (lambda (c) (invoke-restart 'yes)))) ;; handler
(fn1)))
yes or no?
[Condition of type SIMPLE-ERROR]
Restarts:
0: [YES] YES
1: [NO] NO
2: [RETRY] Retry SLIME REPL evaluation request.
3: [*ABORT] Return to SLIME's top level.
4: [ABORT] abort thread (#<THREAD "repl-thread" RUNNING {1003167FA3}>)
Having said that, practically, you wouldn't release a tool which has users interact at the debugger, at least for any level of more serious commercial work no?
As said, you can have a custom "debugger" which does not look like a debugger, but a general purpose user interface in the text terminal. For example, being "custom", it can also hide the restarts which are only meaningful for true debugging (like restart 2,3,4). (see detail [1])
It is possible for a debugger to even invoke a GUI (imagine a debugger hook which pops up a window with an OK button) or a web UI (sends a Websocket message to the client, wait for the user to click something on the browser, receive a reply and invoke the restart). In fact, SLIME is written this way: it overloads the debugger hook, and it sends and receives messages to the Emacs TCP client (thus you can click on the menu on an Emacs buffer to restart -- this action is notified back to the underlying lisp process via TCP, which then calls invoke-restart).
[1] You can obtain the list of restarts with COMPUTE-RESTARTS and filter unassociated restarts (similar to a method dispatch but is temporary / has dynamic extent -- see http://clhs.lisp.se/Body/09_adbd.htm).
Or to put it another way, trace is an option that is passed to the Python interpreter. In Common Lisp (trace ...) is a function that can be called in the REPL.
Python really has a glorified edit-compile-run cycle. The interpreter loop is nice for quick tests or calculations, but isn't too helpful for the incremental development of your program.
This is the crux of the issue. Even in Python 3 the semantics for reloading modules is wrong:
Other references to the old objects (such as names external to the module) are not rebound to refer to the new objects and must be updated in each namespace where they occur if that is desired.
If a module imports objects from another module using from ... import ..., calling reload() for the other module does not redefine the objects imported from it — one way around this is to re-execute the from statement, another is to use import and qualified names (module.name) instead.
If a module instantiates instances of a class, reloading the module that defines the class does not affect the method definitions of the instances — they continue to use the old class definition. The same is true for derived classes.
https://docs.python.org/dev/library/importlib.html#importlib...
In principle Python has a runtime with type-tagged objects that you can inspect and code you can load, so it should be dynamic. Because of the way module and class loading behavior is defined you cannot actually take advantage of this dynamic runtime. Python does not really have a REPL, it has script loading akin to most Unix shells, and a second-class REPL that is useful for some limited debugging. This unnecessary dichotomy imposed on code loading makes it very different from Lisp and Smalltalk.
It also supports writing condition handlers that can do all of the above under program control, rather than under human control. In other languages you would say "exception handlers", but Common Lisp conditions include representations of situations and responses that exceptions typically don't.
These facilities, along with things like a debugger, a tracer, support for built-in documentation, a compiler, and a disassembler, are part of the language standard, not the development environment.
Common Lisp is a language designed from the ground up for writing programs by interacting with them as they run. There aren't very many other languages like that. Smalltalk is one of them.
* Python REPL does not check and warn the trivial type/arguments errors when a function is defined. It is only after the program reaches that erroneous line that an error is signaled. SBCL always warns about the function signature. Perhaps this may be customized by some global variable or a command line option, but not being warned by default is a huge waste of time, as it takes non-negligible amount of time to reach there (specifically because it is a ML task). (SBCL also warns me about type errors in most of the standard functions as well as ftype'd functions.)
* Python does not have restarts. I mean the condition system in Common Lisp. This again imposes a significant amount of slowdown on the debugging cycle as an error may be signaled only after a multi-hundreds-MB data file is loaded in the beginning of a function and then the erroneous line of code is run. In CL, when an error is signaled, the program enters the debugger and halts there, during which you can correct the program, redefine the function, then resume the code from one or two stacks above the current. In python an error always rewind the stack to the toplevel, so you have to rerun the whole function, loading that huge data file again in the middle of the stack. Don't say you are afraid of some side effects in the code, I follow "Mostly Functional Style" in CL.
* Python debugger shows the stack frame, but does not by default show the values of the arguments in the function call stack when an error is signaled. It forces you to use an ugly printf debugging style. I would like to know if it is possible to alter the way the stack is printed. In CL, a debugger is just a function stored in debugger-hook. I hope you can do something similar.
* Somehow Emacs C-M-f selects the whole line, not an expression.
When clojure came around, I tried lisping again, and for whatever reason i was ready for it, and clojure is my goto language.
That said, I'd never connect to a repl on a server and do any kind of bug fixing unless it was really dire. It's too easy to get the run state out of sync with the text source files, and I don't see why this would be different with common lisp.
I think the repl is great for development and figuring things out, but the whole idea of "program as state" seems antithetical to the ideas of functional programming. With the main issue for me being that I have to somehow keep track of this program state in my head once I start tinkering with it.
For example, say I find a function has a bug and I go into the repl on the server and fix it, in situ. Say I spot some other issues, or had them on my list of things to fix when I had time... I start modifying functions here or there. Now my "program state" no longer matches my source code, and I have to remember to fix it so that it matches. That's error prone. I could enforce my own methodology of fixing the source and copy-paste to the repl or send it in whatever way, still error prone.
I have to remember what changes I made, to what functions, and if I iterated, now I have to remember which iteration was final. Multiply that by any and all functions that are touched. It's a lot to keep track of.
So when you say "inspect the state," I have to think common lisp must have some feature for state inspection that is missing in clojure. I know there are a lot of features of common lisp, especially in stack-tracing, that clojure has no parity to. But I'm curious if common lisp has some way of completing a "round trip edit" such that an edit/correction made on a server process, in situ, are able to be easily propagated back to your master source?
If you read all that, thanks :)
In SLIME you just connect to the Lisp process, open up your versioned-controlled file, edit it, hit C-c C-c for slime-compile-defun or C-c C-k for slime-compile-and-load-file. Isn't that how CIDER works also?
I remember one time, it was actually with Clojure, I had to deal with Paypal. Since paypal only sent results back to the server and not to localhost I couldn't debug the problem locally. So I connected to the server, bunged all the debug information into an atom and made a few requests. Then I could inspect the atom, make some changes and find out what the problem was and fix it right away. It took just a few minutes.
At about the same time I was also working on a C# server. There was a bug there that could only reproduce itself on the server. To fix that bug I had to put a load of logging into the source. Check the file into source control. Wait for the CI system to build and deploy. Then kick off some requests. Inspect the log files. Make some changes. Check in.. wait.. etc.. It took days to fix that bug.
Admittedly the Clojure project was mine on my own server and the C# project was on some corporate enterprise behemoth so there were other factors that hindered that project, but it did highlight to me how much quicker you can get stuff done when you have that instant feedback.
When I say inspect the state, generally I just mean inspecting the values of global variables. In CL it is possible to put breakpoints in the source and inspect all the variables up the stack. Mind you, this can be a bit hairy on the server if there are multiple people connected as there is no guarantee as to whose Slime connection will pick up the breakpoint.
The round trip edit is pretty similar to Clojure. You would generally have your source file open in Emacs, change the source code and press Ctrl+CC with the cursor in your function definition to apply your edits to the server. Then you save the file and check it in to source control. You don't have to type directly into the repl.
You don't have to work like this if you don't want to. Without mentioning any names, it has been known for the server image to be completely out of sync with the state of the source files to the point where there weren't really any source files. The work around was to just treat the server image as the master source of truth. Luckily in CL you can just dump the image to create a new executable with its current state. So for several years this one server existed only as a constantly updated binary image. I wouldn't recommend working like that, but it has been done.
Depending on your implementation - I know Clozure CL at least allows this - you can extract the source code out of the running image. So it is certainly possible to propogate back to master source. It was eventually how this one server was restored to a more orthodox source code tree..
To be fair, though, I run an nREPL TCP listener on my production servers too, so I can do the same thing with Clojure. :)
I think it was in his Beating the Averages essay - the one in which he talks a lot about the advantages Lisp gave them at Viaweb, over the competition.
[1] http://paulgraham.com/avg.html
[2] More Technical Details: http://paulgraham.com/lwba.html (note: redirects twice)
Here's the relevant section:
[ Interactive Toplevel
Lisp's interactive toplevel is a great help in developing software rapidly. But the biggest advantage for us was probably in finding bugs. As I mentioned before, with Web-based applications you have the users' data on your servers and can usually reproduce bugs.
When one of the customer support people came to me with a report of a bug in the editor, I would load the code into the Lisp interpreter and log into the user's account. If I was able to reproduce the bug I'd get an actual break loop, telling me exactly what was going wrong. Often I could fix the code and release a fix right away. And when I say right away, I mean while the user was still on the phone.
Such fast turnaround on bug fixes put us into an impossibly tempting position. If we could catch and fix a bug while the user was still on the phone, it was very tempting for us to give the user the impression that they were imagining it. And so we sometimes (to their delight) had the customer support people tell the user to just try logging in again and see if they still had the problem. And of course when the user logged back in they'd get the newly released version of the software with the bug fixed, and everything would work fine. I realize this was a bit sneaky of us, but it was also a lot of fun. ]
[1] Discussed in this same thread, e.g. this comment by mikelevins:
https://news.ycombinator.com/item?id=14677016
Not sure if PHP has that.
Incidentally, I seem to remember the following: some years ago, after playing around some with Lisp (Allegro CL from Franz, and the Lisp IDE from Lispworks, both of which supported this feature or something like it), I was trying out a then-current version of Visual Studio, and saw that it had a new feature called Edit-and-Continue, which I thought was similar to the Lisp feature.