Hacker News Reader

Comment by agentultra | Hacker News Reader

> Is it actually possible to do hot code reloading while preserving the state of the running system in Common Lisp?

Yes.

> I'm genuinely curious, I kind of had this impression that it was a feature fairly unique to Erlang.

It's not. iIRC there's even a version of the JVM with code reloading.

What's interesting about Common Lisp is that not only can code be dynamically compiled into the running image but class definitions can be changed and live instances will be updated without stopping the program or reloading anything. There are a tonne of features in Common Lisp like this that are geared towards robust, maintainable software.

View on HN

> It's not. iIRC there's even a version of the JVM with code reloading.

I've seen some info about the JVM implementation, but I could've sworn it wasn't able to preserve state while hot swapping. I haven't dug in to it though, so it's totally possible it handles that.

> There are a tonne of features in Common Lisp like this that are geared towards robust, maintainable software.

I'm a fan of Lisp in general, though my experience is limited to a tiny bit of Scheme and experimenting with Clojure. I'm glad to see that stuff like this is available in at least one of the Lisp variants! Question for you (if you program in Common Lisp): What are the main selling points? I spend a lot of time already with a couple of languages, Elixir/Erlang (for fun), C#/F# (for work) - what is it that would make me want to switch from one of those to Common Lisp?

Thanks!

As someone who's written a bit of Common Lisp, I can say that the macros are a pretty awesome feature to use. Common Lisp has the advantage over Clojure and Scheme that the SBCL compiler is incredibly fast; it can come close to C#/F# on Mono for some tasks, in spite of Lisp being a dynamic language. It also compiles code and expands macros much faster. It's got more power than the other lisps, but it's also less elegant, so make of that what you will.

> I can say that the macros are a pretty awesome feature to use

Erlang has them in the form of parse transforms and Elixir has them too and makes them easy to use. Not as easy as sexp based languages, but almost.

There is also LFE (Lisp Flavored Erlang) [0] which gives lisp style macros and some other goodies to Erlang

[0] http://lfe.github.io/

BTW, a very Erlangish actor system, complete with hot code swapping (while preserving state), is available for Clojure and is called Pulsar (all open-source): http://docs.paralleluniverse.co/pulsar/

In no specific order:

1. Configurable compiler. You can tell the compiler where it's safe to ignore types and where to be strict about types. For the battle-hardened critical sections of code turning off run-time type checking can be a pretty big win performance-wise.

2. Extensible compiler. Macros are like tiny compilers. It's possible to create DSLs for the programmer to work with a problem in terms of nouns and verbs that make sense for their task while still compiling to VHDL for your application. Common Lisp becomes the substrate on which you build the language you need to solve your problem.

3. The best error handling system I've seen in any language. Signals, handlers, and restarts allow higher-level code to choose an appropriate strategy for recovering from errors under various conditions without losing state and restarting the computation.

4. Top-class introspection tools. The JVM has been able to do some of this stuff for a while but with CLOS and the error handling facilities available Common Lisp makes this so much better. You can inspect any object, change values on slots, trace any function, step through the entire stack, and recompile code into the running image. There are nice tools written on top of these facilities which provide some of the most elegant debugging sessions I have ever encountered. Stop your program? Why? I can connect to a running image, inspect the running program interactively, and compile the fix into it without restarting. Even over SSH directly from my IDE.

And that's just the stuff in the specification. I'm a fan of the newer ASDF systems for defining systems and their dependencies (implicit file-system structure for defining module composition? No thank you). There are some really solid libraries available (iolib, bordeaux-threads) and a very nice FFI for interfacing with alien code. And various implementations provide interesting extensions depending on what you need. Clozure Common Lisp, for example, has really good native Cocoa bindings. SBCL has a very slick compiler for generating fast code. Most of them provide threading extensions and there are libraries which abstract away the differences if you're writing a library that depends on threads.

Thanks for breaking it down! I'll have to investigate further, but there is a lot here to like.

I use Common Lisp and Scheme at work, and have written and maintained several systems heavily relied on class redefinition and run-time code loading. They're great. OTOH, I feel Common Lisp's concurrent process handling is rather weak, and I envy Erlang for that (Not that CL can't do the trick, but usually what implementations offer is fairly low-level and I had to roll my own higher-level IPC.)

For example, how well CL's class redefintion and instance reinitialization work with preemptive threads? It isn't obvious how each implementation address this issue by skimming docs, so I'm curious if anybody with direct experience. When I implemented CLOS/MOP style OO in my Scheme, I ended up using a global lock during class redefinition, for it need to modify lots of places. It's MT-safe now, but I'm still wary to run class redefinition on a system written in my Scheme, which is busy handling requests.

> there's even a version of the JVM with code reloading.

Yes, because of Java's extensible classloading mechanism, all you have to do is drop an war/ear to the 'deploy' folder of your application server. New files are deployed, rewritten files are redeployed and deleted files are undeployed.

Is there a mechanism to migrate existing data to the new code?

If you have external state stored in say a database, I think you have to control your deployments to migrate the schema before the model objects can be deployed.

So no.

One of the beauties of Erlang's hot code loading is the ability to migrate internal application state. An example:

I'm storing a lookup table going from key to value as a tuple {key, value}. With a change going out, I'm going to start doing {key, {value, other-value}}. In Erlang there's an easy pattern in place to do that. Just provide a function that goes from {key, value} to {key, {value, sensible-default}} and as part of the code reloading process that function will be run before the new code starts executing.

You may have misunderstood. A file can have multiple, say 'deployment units'. You can have a 'migrate deployment unit' which depend on the 'model deployment unit' which forces the app server to deploy the migrate unit first. You can have code there that migrates the tables and fill with default values before your model entities become available for use. From your comment this seems to be the same as what Erlang does for your example, unless you are claiming that Erlang's implementation is superior which could be the true, since I have never written any non-trivial programs in it.

I think the difference here is that the migration of internal state happens the moment the module is swapped out in memory. Erlang doesn't have mutable state, so state is instead passed around within a module. When a fix needs to be deployed, the two versions (old and new) run simultaneously, any new processes immediately run the new code, old processes call a `code_change` function which takes the current state within that process, performs any transformations required for the new code, and then the old module code is swapped out for the new.

That's not at all a perfect explanation of the process, but it's more or less useful enough to describe the difference between the two platforms.

sure, any existing connections continue to be serviced by the old code, and new connections are picked up by the new code.

if you need more elaborate lifecycle management of components as they are upgraded, then maybe OSGI could do that.

erlang's approach really appeals to me though -- not only does the code get upgraded, but it's in cooperation with its message loop so you don't miss a step, and you also get a callback to migrate your actor state.

Reply on news.ycombinator.com