Comment by hajile | Hacker News Reader

Comment by hajile | Hacker News Reader

Parent Full thread

Those REPLs aren't even remotely comparable. You can TEST with them, but you never DEVELOP with them.

If they were the same, you'd load up a repl of your entire codebase. Now you'd dynamically update the part you're working on, test it live against your current environment, and finally save it to move to another thing. When errors happen, you just drop into the debugger to find the issue. If you missed a call site or two, you'll probably catch it here. Because it's instantaneous, it's not painful. I'll have a REPL open pretty much all the time when messing around with CL.

In JS (my day job), this wouldn't work at present if for no other reason than directly replacing a function is impossible. You'd create a second instance of a function, but all the old references from other places in the code would remain stuck with the old version.

JS falls back to "hot reloading" which is literally "restart the whole JIT" which means that you have to warm the JIT up again and (more importantly) that all your state disappears.

You make a change then wait a while for everything to recompile and run. Then you have to write/execute a bunch of code to get your state back to where it was. When the error is thrown, you now have to look through the things for that call site you missed, save, and reload everything all over again. Rinse and repeat. This heavily disincentivizes interactive and incremental development in favor of massive changes then trying everything out at once.

A few years back I did a presentation on this very topic. Some people have said they appreciated it so I'm sharing a link here in the hope someone find it useful: https://www.youtube.com/watch?v=bl8jQ2wRh6k

Doesn't the ability to "hot reload" in either CL or Node largely depend on how the application is structured? For example, if you were developing a web server that sat in an infinite loop waiting for requests to come in, how would you go about redefining a function referenced inside that loop? Wouldn't there need to be two threads of execution? One that was the web-server and a second that acted as the REPL? To redefine something in the web-server, the two threads would have to synchronize in order for the new definition to be passed off. Couldn't this same setup be implemented in any language that supports threads and some type of `eval` function?

Edit: The two threads of execution I mentioned above are likely what the combination of SLIME + Swank are doing.

I have done this exact thing in CL before. What you do is have your event loop function just be a loop calling another function, where all the event handling actually happens. When the event handler function is recompiled, the new version will run /the next time the function is called./

If the recompiled function is currently running that stack frame and anything deeper won't be affected for that call.

The only thing you can't do is redefine the loop function itself since it never stops running. That's why you factor it so there's no reason to change that function, just the functions it repeatedly calls.

Hope that explanation made sense.

Edit: And yes, the way this is implemented under the hood is that the repl has its own thread. You could do this even in C. In fact people have done this for the Linux kernel for example(at least for binary patches). The great thing in CL is the language is designed with the functionality in mind, and the dev tools just support it, so you get it for free.

Incidentally, this is nearly identical on top to the pattern Erlang uses for the same reason, where most processes couple a long-lived “behavior” with a callback module with a particular structure and a pure-data state value threaded through everything. When the module is replaced, the new functions are picked up automatically, and there's also a callback for adjusting the state information on code changes if need be.

The underlying “system message requesting your process to ensure it picks up the code reload” part is quite different, though!

Others have answered the question but the reason this works in CL is rather simple: Function calls (by default) have a single level of indirection. When CL calls a function it looks up the address of the function in a table (the symbol table), then jumps to the given address. If you redefine the function associated with the symbol, the next call gets the new definition. Yes, you need a separate thread (native or green) for the REPL but it has less to do with the REPL being a separate thread and more to do with the symbol table being a global data structure.

This is why there is no "linking" step in Common Lisp compilation: CL keeps the symbol table around at runtime; C just throws it away.

Note, however, that a CL compiler is allowed to assume that calls to functions defined in the same file are to that definition (so it can short circuit the function lookup.) This is from the CLHS section 3.2.2.3. http://clhs.lisp.se/Body/03_bbc.htm

"All conforming programs must obey the following constraints, which are designed to minimize the observable differences between compiled and interpreted programs:"

[...]

"A call within a file to a named function that is defined in the same file refers to that function, unless that function has been declared notinline. The consequences are unspecified if functions are redefined individually at run time or multiply defined in the same file."

I suspect this has to do with deferring warnings from forward definitions, but I might be wrong. It's still common behavior for REPL redefinition to have an immediate effect on the next function call unless that function was declared inline. (Regardless of what file the function was defined in.)

I wonder how Clojure(script) deals with this. Do you know if Clojurescript restarts the entire JIT as well? Is it the same in Clojure with Java?

Clojure and ClojureScript don't have the comprehensive support for livecoding that Common Lisp and Smalltalk do.

As an example, consider what happens if Common Lisp's UPDATE-INSTANCE-FOR-REDEFINED-CLASS, mentioned in my comment above, is called without a method definition for the redefined class in question. Lisp drops you into a breakloop by calling BREAK.

BREAK is another feature of Common Lisp defined by the language standard (http://clhs.lisp.se/Body/f_break.htm#break). It puts you in a repl that has the full language and runtime available, but also exists inside the dynamic environment of the control path where BREAK was called.

In this example, that means that we can see the lexical environment of the call to UPDATE-INSTANCE-FOR-REDEFINED-CLASS, along with everything that was on the stack when it was called. We can see the instance that needs to be redefined and all the definitions that are currently in effect. Using the features provided by the breakloop, we can interactively define the missing method on UPDATE-INSTANCE-FOR-REDEFINED-CLASS to properly reinitialize the instance in question, then activate a RESTART (another standard feature of the language) that resumes execution of the function, but now in a dynamic environment that includes the new method and any other changes we made in the breakloop.

The original control path that landed us in the breakloop can now complete successfully, referencing a properly reinitialized instance. The method on UPDATE-INSTANCE-FOR-REDEFINED-CLASS is defined now, so future references to previously-defined instances of the redefined class will be transparently reinitialized and behave as if they always had the new definition.

Clojure and ClojureScript don't have the runtime support to do any of this. If you redefine a class that has existing instances, and if those instances aren't dropped on the floor when your new definition is hot reloaded, then the old instances don't reflect the new class definition, and so code that relies on it is wrong. When a wrong access occurs, you are dropped--not into an interactive breakloop--but into a noninteractive printed stack trace.

The debugging features of modern browsers give you some parts of what a Lisp (or Smalltalk) system does--for example, some tools can watch the dynamic state of variables and enable you to edit them. There are tools available for ClojureScript that give you access to some of these features.

Even in the Clojurescript ecosystem, however, the full suite of livecoding tools offered by Common Lisp and Smalltalk (and mandated by the Common Lisp standard) is not available.

Lisp is so cool, I wish every language could do these kinds of gymnastics. Thank you for sharing.

Reply on news.ycombinator.com