- Erlang, the language and system, was designed for highly concurrent and fault tolerant systems. In many ways these were more important than raw throughput in that if it could not handle the massive concurrency and build truly fault-tolerant systems then it wasn't interesting. At all.
- Same with pre-emptive scheduling, the system had to be non-blocking. Again if it wasn't then it wasn't interesting. Yes, you can provide primitives to allow the programmers to do it but this does make things difficult and not-reliable.
- The language design was focussed on these types of systems and and implementing the types of architectures which OTP supports.
- From Erlang's point of view using OS threads as a base for Erlang processes is not an option. They are way too heavy and you can't have enough of them to be interesting. 10k processes are child's play, 100k processes is starting to get interesting and 1M process production systems exist, for example WhatsApp.
- For Erlang processes are the basic building blocks in a similar way to objects in an OO language. What would a Java programmer say if they were told that they couldn't have more than 1000 objects?
- Having the erlang VM handle using all the cores, or as many as you want, by default is just the natural way to do things. If I had to do any form of restructuring of my system because I was running on 4, 8, 16 or 32 cores I would consider that to be intolerable and so primitive I would wonder what the implementors were thinking of.
- While the Erlang syntax is different (which functional language doesn't have a different syntax?) it is actually very concise and consistent, this by design. The elixir syntax is "Ruby influenced" and more complex and feature filled. Which you prefer is up to you.
- I am very fond of lisp so there is at least one native implementation of lisp on the Erlang VM, LFE (Lisp Flavoured Erlang), http://lfe.io/ and https://github.com/rvirding/lfe.
- There is nothing which you can do in one which you can't do in the others, after all they run on the same VM and you can easily combine them and use them together.
That's about all for the moment,
Robert
Can you find the magic sauce in another language?
I believe the honest answer is no. You can't do it in Java because the JVM doesn't support it. You can't really build this on top of something that was not designed with parallelism and concurrency in mind, and designed correctly. For that reason Go doesn't do it either. Scripting languages aren't even thinking about it and to do it in something like C or C++ you'd basically be recreating a poorly implemented half working version of erlang.
So more interesting is--- why do people have this aversion to Erlang and Elixir?
It doesn't take but a couple weeks to get used to erlang programming, and Elixir is even faster I think.
For the record I've been taking interns thru Programming Elixir in about 2 weeks and getting them to where they can program in the language (just at that point now with a new group so we'll see how much they struggle... but I've already seen code written by them.)
Erlang did it right and has put in several decades of effort getting it even more right. Other languages, like Go, aren't even trying (though of course advertising "concurrency" is very popular, none of them really do it.)
All these people choosing things like node.js or go for problems where erlang is the correct solution make me think that too many of the people in our industry aren't really engineers, but more scripters following the herd.
And engineer knows why erlang is the correct solution for distributed systems.
Because you may want good performance without dropping down to C. Many serious Erlang applications are mixed Erlang/C applications, where they delegate processing to C, or use Erlang for the control plain and C for the data plain.
Or because you want more sophisticated, fast transactional shared-state than ETS allows.
Or because you want to take advantage of a much wider selection of high-quality libraries.
Note that these aren't problems with Erlang or Elixir as languages -- they're both great -- but rather with the runtime they're running on, whose developers simply don't have the resources to make it all that it can be. Which brings me to:
> the JVM doesn't support it.
The JVM most certainly does support it. It was designed with parallelism and concurrency in mind, and includes some of the best high-quality, production-ready implementations of concurrent data structures and schedulers in existence. The effort that's been poured into the JVM -- and even JVM concurrency alone -- dwarfs the effort put into Erlang at least by an order of magnitude.
I think Erlang would benefit greatly if it were to target the JVM. There is a project, Erjang, that runs BEAM code on the JVM, but it is not very actively maintained and doesn't take advantage of some of the recent, relevant innovations in the Java world. Since that project was developed, there have been major improvements in the pertinent areas on the JVM: concurrent GCs, JITting of dynamically-typed languages, fibers and work-stealing scheduling.
OpenJDK is the second-largest open-source project in the world (after the Linux kernel), and I don't see why Erlang shouldn't take advantage of the vast resources it has at its disposal. OpenJDK and Erlang are a great fit!
Can you state which language constructs Erlang has that other languages don't, that are critical to concurrency? Cause it looks like someone built a great VM and cluster library, then tossed in a language for fun.
And please explain why Erlang's features are not possible in other languages. Not just unavailable at the moment. Or do you mean the guarantees are different? As in other languages will let you have corrupted shared state? (I view that as a positive; let me escape guarantees when I want too.)
Or perhaps point me to a decent intro that'd cover all this? I looked at it before and didn't get the real point why we need Erlang the language or why in theory, you can't have OTP with the CLR or JVM.
Erlang has built-in immutability done right (more so than in Scala, OCaml, Clojure(?)) - if it's not built in the language from the start, there's nothing you can do about it.
Erlang has built-in binary serialization/deserialization that makes messages across network easy and natural. Please show me a language that can serialize a closure and send it over network. A rich type system and general-purpose language features are obstacles here, types of values have to be dumb to serialize seamlessly.
Other than that, it has a lot of convenient and coherent features like good pattern-matching (useful for parsing and routing messages), proper tail-call optimization (just look at Scala, again), the built-in actor model, lightweight processes, process monitoring, the large library of battle-tested distributed practices, OTP, – these things can be recreated in another language/VM, but it will probably take huge resources and efforts and self-limitation of the platform, not features.
Scala/Java: Akka http://akka.io
Haskell: Cloud Haskell http://haskell-distributed.github.io/
and none of the above have solved preserving types across machine boundaries (or at least not in a stable/non-experimental let's-bet-the-bank-on-this release).
Quasar looks pretty insane at first blush; if Java 9 or 10 delivers on the missing piece [1] for this library, watch out.
[1] "Quasar fibers are implemented by creating and scheduling continuation tasks and since the JVM doesn’t (yet) support native continuations, Quasar implements them through selective bytecode instrumentation: methods that can block a fiber currently need to be explicitly marked through annotations so that Quasar can insert the continuation suspension and resumption hooks."
If Erlang is not your cup of tea, look at htons() and nstoh() functions to have another example.
My trade is in real time distributed systems applied to the automatic control of trains. You wouldn't believe the number of times I have had to fix this issue in multiple codes, both new or legacy.
And my example is limited to individual values, not data structures. Think about how ugly things can turn if you have things like lists or dictionaries.
When you consider that Erlang adds on top of this built-in marshalling and demarshalling of data structure. As of today, I haven't found a more efficent environment to do this kind of stuff safely.
Or is it just that Erlang has this all batteries included, the way Python's popular with scientific programming?
I'd actually love to see some performance numbers comparing performance on Erlang/OTP vs the alternatives, though. I can't find anything quite like that, maybe someone knows a link.
But I agree that Erlang's design is beautiful, which is why I've decided to port it to the JVM with Quasar. I wanted to bring that great design to a more powerful runtime and a much bigger ecosystem. Again, the problem isn't with the Erlang language but with the runtime that, while really nice, simply does not have nearly enough resources to compete with the bigger runtimes like the JVM and .NET (or even V8).
We will have some benchmarks eventually, I promise, but those take a long time and aren't too critical, because, like I said, the Erlang ecosystem is very small.
Would you say that it's missing something?
That's a far cry from Akka Typed [1], the recently released experimental branch, which allows you to match on the "real" type; as a result distributed code is compile time checked...a dream scenario ;-)
Having said that it doesn't change the fact that Akka preserves types in most arbitrary hierarchies today, across machine boundaries.
What about Static Pointers[0]? Do you feel the solve the problem or are likely to if you characterize them as non-stable and/or experimental?
0: https://ocharles.org.uk/blog/guest-posts/2014-12-23-static-p...
I work on a Storm library for Python that's called streamparse[2], and the goal of the project is to allow us to easily achieve Erlang OTP-style reliable processing atop open source infrastructure while still writing pure Python code.
I also gave a PyCon talk about streamparse which you can find on YouTube[3]. It describes the motivation for the project -- which is to solve a large-scale real-time analytics problem with Python and do so in a reliable way, while beating Python's multi-core limitations (the GIL, etc.)
[1]: https://storm.apache.org/
that reliability wasn't the case for at least one of the internet heavyweights, quite the opposite [1]
[1] http://blog.acolyer.org/2015/06/15/twitter-heron-stream-proc...
So far Storm is working as advertised. Frankly, what it does isn't that terribly difficult, but it's good to have a well-tested implementation of it so we can focus on our business logic.
Also, my team is working on an open source example project called birding that uses streamparse and pykafka together to build filtered firehose tweet streams from Twitter's public API. We think this will illustrate both Storm and Kafka really well, and be beyond the typical word count examples. See https://github.com/Parsely/birding to track that effort.
"OTP contains functions to safely spawn and initialize processes, send messages to them in a fault-tolerant manner and many other things."
Also, "supervisors are one of the most useful parts of OTP you'll get to use." And, "an OTP application specifically uses OTP behaviours for its processes, and then wraps them in a very specific structure that tells the VM how to set everything up and then tear it down."
Given that, I don't see how you could think Storm isn't at all related. (I never said it was equivalent, just very similar goals.)
Definitely worth taking a look at it.
It does cut down on a lot of Erlang boilerplate, so it’s still a better option… but I’m hoping someone will give it a third try.
The syntax itself is slightly disappointing though. Optional parentheses around argument calls doesn't really add much except confusion, and the extra cruft around blocks and functions make things like pattern matching quite a bit clunkier.
And two different lisps:
* Lisp Flavored Erlang (http://lfe.io/) * Joxa (http://joxa.org/)
There is nothing Erlang VMs provide that the JVM doesn't, except for some guarantees that are simply different on the JVM, and I think most applications would prefer the JVM's advantages anyway.
To get erlang/OTP you would have to replicate it from the beginning in C, and there's no point in that, when you can just use erlang.
That is a strange comment, ofc the Erlang VM is not the single best VM that will ever be created. Some day, someone will create a better one.
Fixed! Read your comment, understood it, but I think you probably meant the opposite
Tail call optimization is more painful. On the JVM, it is done at the language level[1] (Erjang does it), and will eventually be added to the JVM itself[2]. In any case, because Erlang the language needs it, it can have it on the JVM. But the "Erlang model" of processes etc. doesn't require it.
[1]: http://cesquivias.github.io/blog/2015/01/15/writing-a-langua...
[2]: It is unfortunate that the JVM doesn't yet have TCO, but the compiler generates such fast code that implementing it at the language level still yields better performance than Erlang VMs.
I like Quasar, but I suspect you removed reduction scheduling because function-call-site reductions, as erlang does natively, is not great when implemented in-language, as it blows up a number of optimizations and muddies the register picture.
That said, I think MCRed is doing a slight disservice even he's got good intentions; erlang is amazing for what it does, quasar is pretty nifty and doubtless has many superior use cases; even go has a lot of utility in the concurrency space. Let a billion processes bloom.
Sure, but again, it's certainly doable (and has been done).
> I suspect you removed reduction scheduling because function-call-site reductions, as erlang does natively, is not great when implemented in-language, as it blows up a number of optimizations and muddies the register picture.
We removed it because it's unhelpful, and let me explain why. Any preemption that's based on time-slices/reduction -- basically, preemption at any point other than the thread blocking on something -- means that the thread still wants more CPU, but you can't allow it to have it because there are others that need it. This is fine when you have roughly the same number of such CPU-hungry threads as CPU cores, or even a bit higher -- not when you have 100,000 of those or 1M. So the number of CPU hungry threads must be very low, or your system is grossly under-provisioned.
Now, Erlang has to do it because all Erlang processes are scheduled in user mode, but even in Erlang this doesn't help if you have more than a few such processes. OTOH, on the JVM, where you can pick if you want user-mode or kernel scheduling, you can simply choose the kernel to schedule those few CPU-hungry threads. The kernel does it much better than anything in userspace can.
If you don't have CPU-hungry threads, but well-behaving threads that occasionally become CPU-hungry, it still doesn't help because the work stealing scheduler can very easily deal with such occasional behavior.
In short, time-slice/reduction based preemption in the user-mode scheduler simply doesn't help you in any way. Erlang does it because it has no other scheduler and it must support (a few) processes that behave that way.
1. Preemptive multitasking is strictly better in almost all cases than cooperative multitasking, when you have more processes, not fewer; especially when these processes are heterogenous. Throughput decreases over the best-case cooperative scenario, but jitter significantly decreases, and most importantly, you -- the user, the operator, or the programmer -- have a strong guarantee that the schedulers will never lock up because someone wrote code wrong. That guarantee alone is why every operating system abandoned a cooperative model and pushed schedulers as close as possible to preemptive multitasking.
2. Kernel scheduling is radically worse than user-space scheduling in nearly every scenario. The kernel schedulers are necessarily very generic and workload-agnostic; kernel threads and the thread management infrastructure are incredibly heavyweight and slow relative to user space; the context switching penalty is gigantic; processes relying on kernel scheduling are forced to compete with other processes on the system; and the list goes on. This is why unikernels, exokernels, and user-space tcp implementations like onload are growing prevalent, especially in the low-latency space.
Sorry I didn't make myself clear. Quasar most certainly employs preemptive multitasking, just not time-slice- (or reduction-) based.
> Kernel scheduling is radically worse than user-space scheduling
Except when descheduling on time-slices. Remember two facts: 1/ your hardware can only support very few threads that require time-slice preemption so those task-switching costs are negligible, and 2/ no other thread is dependent on low-latency access to data produced a thread that is descheduled on a time-slice, so the latency doesn't matter much either.
None of that is my intuition. I've been profiling and measuring this stuff for a few years now, and I'm a huge proponent of userspace scheduling. Adding time-slice preemption didn't make things any worse -- it just didn't make them any better either, so we took it out, and we tell users to use kernel scheduling for those threads. If the user makes a mistake, we recognize it at runtime and tell her that kernel scheduling might be better for that thread.
"Runaway Fibers A fiber that is stuck in a loop without blocking, or is blocking the thread its running on (by directly or indirectly performing a thread-blocking operation) is called a runaway fiber. It is perfectly OK for fibers to do that sporadically (as the work stealing scheduler will deal with that), but doing so frequently may severely impact system performance (as most of the scheduler’s threads might be tied up by runaway fibers). Quasar detects runaway fibers, and notifies you about which fibers are problematic, whether they’re blocking the thread or hogging the CPU, and gives you their stack trace, by printing this information to the console as well as reporting it to the runtime fiber monitor (exposed through a JMX MBean; see the previous section)."
(http://docs.paralleluniverse.co/quasar/)
that describes a classic symptom of cooperative multitasking, and that kind of broken behavior doesn't happen in a preemptively multitasked system. Certainly it violates the programmer intuition invariant that she can just write programs, if she has to occasionally check on a JMX MBean to find out if the framework has pooped on the bed.
additionally, when you say you are doing preemption without timeslicing, that doesn't really make sense, as all preemption requires slicing time: that's literally what it means.
On the kernel scheduling piece, your argument doesn't make any sense to me. Hardware support is only tangentially to do with high thread context switching costs; and it is frequently the case that many actors can fit into cache, where many threads cannot.
Fundamentally I'm sure Quasar has some great use cases, and probably shows some interesting performance gains over erlang in some of them, but this discussion has dimmed my interest significantly. Quite unfortunate.
> Certainly it violates the programmer intuition invariant that she can just write programs, if she has to occasionally check on a JMX MBean to find out if the framework has pooped on the bed.
That is easily solvable by having Quasar automatically migrate a thread from being scheduled in user-space to being scheduled by the kernel. However, even that cool feature doesn't justify itself, because things work great as they are. The programmer doesn't need to check JMX occasionally. The behavior is immediately detected in testing and reported as a warning. Simple and effective. Note that all this works even if this behavior occurs in native code, which is more than Erlang does. Threads that are less suitable for user-mode scheduling must be few, and they are very easy to recognize.
Doing any of that automatically may be cool, but has no real value. We might do that at some point, but it's not a high priority.
> as all preemption requires slicing time: that's literally what it means.
No, it means that the scheduler has the ability to deschedule the thread when it wishes[1]. We have this ability, we just no longer use it on time-slices (we easily can, it's just not helpful in the least).
> On the kernel scheduling piece, your argument doesn't make any sense to me. Hardware support is only tangentially to do with high thread context switching costs; and it is frequently the case that many actors can fit into cache, where many threads cannot.
All you say is true, yet none of it matters in practice when it comes to time-slice preemption. Only a handful of threads can enjoy that form of scheduling anyway, whether they take 10MB or 10 bytes of RAM.
[1]: https://en.wikipedia.org/wiki/Preemption_(computing)#Preempt...
Those two sentences are directly contradictory. Either the behavior doesn't happen, or it needs to be checked (via a test suite, or programmatically, or manually). Test suites will go a certain way, but behaviors in production change with new or different data, and the last thing I want to deal with as a developer is trying to remember all the corner cases in my framework when the system is not responsive. Even erlang is difficult; I can't imagine trying to deal with an erlang-like framework with even fewer friendly invariants.
Anyway, it sounds like we inhabit completely different universes, so really there's no point in continuing. Good luck with Quasar. Again, I'm sure it has some great use cases; just not, apparently, common erlang-like ones.
Trust me, there are real issues with Quasar, but the one that you fear will bother you isn't one of them.
You could, I suppose, hack something together using multiple JVMs per machine (avoid global GC) and Akka and implement some custom live diagnostics, but if you're really doing a telecommunications project there's no real shortcut. Plain Akka will cut it in many cases but it's not a thoroughbred like Erlang. Give up, Erlang just can't be beaten at its home turf by a framework.
But recently I've been trying to understand OTP/Erl and I'v had no luck.
Now seeing these two as an alternative makes a lot of sense for diff langs.
With `jRuby` and more and more with `Rubinius`, the concept of actor-based concurrency itself is coming into range of being language-unspecific, with Erlang not being the default authority on that.
Akka gets a lot of respect, and of course Elixr, Go, Rust, etc get a lot of concurrency attention... But Celluloid with Ruby is far superior in my opinion, because it is not only performant and functional, it actually allows you to get into the internals and change how your actor system behaves. It's as simple or as complex as you want it to be. Oh and you don't feel like killing yourself while you're writing code.
One quick question:
Aside of the docs, any suggestions for learning on further Celluloid & Ruby /actor based patterns?
For now..
Google Groups: https://groups.google.com/forum/#!forum/celluloid-ruby
IRC: #celluloid on Freenode
Or drop me a message with your question. @digitalextremist on github
Orbit for the JVM / Scala by EA (BioWare actually)
https://github.com/electronicarts/orbit
Then there's Orleans by Microsoft for .NET:
https://github.com/dotnet/orleans
They feature a concept called Virtual Actors as opposed to other frameworks like Akka for the JVM.
* Project Iris http://iris.karalabe.com/ * Go Circuit https://github.com/gocircuit/circuit * NSQ http://nsq.io/ * Consul https://www.consul.io/ * SkyNet https://github.com/skynetservices/skynet * Grace https://github.com/facebookgo/grace
Each having their pros/cons with varying quality.
Replicating Erlang processes -> usually you use goroutines. When that level fails you fail the whole process and restart it using some supervisor. You may need computer level failover as well, depending on the requirements.
For the servers/protocols part of OTP, the stdlib is usually sufficient to get things running; although may need some additions for rarer encodings.
For machine level deployment there are cluster managers i.e. kubernetes, aws, google cloud etc.
Regarding debugging/monitoring I haven't seen anything that is close to Erlang.
Basically, I have no idea what you are doing and cannot recommend anything particular. For the full feature set there is no single replacement. If you need OTP then use OTP.
So, Erlang/OTP are correct. The rest, so far, are not correct.
Use the right tool for the job. Not the one you like best. Just because you can use the blunt end of a drill to hit a nail, doesn't mean you should give up hammers... You might even break your drill!
If your problem's solution requires extremely concurrent, high availability, low latency, throughput then you must use BeamVM. The JVM will be too slow and largely memory inefficient compared to Beam.
If concurrency isn't needed, or isnt applicable (parallel, or synchronous, problems), then it's not the answer nor is OTP. E.g. I wouldn't dream of writing and image processor in Elixir-- Beam isn't the right tool to solve that problem.
Also, checkout dialzyer if you think you need type safety. It'll catch plenty of stupid mistakes... I had that urge (for types) coming from the JVM, or a non-functional non-dynamic language, but it's actually a straw man. I'd highly encourage you to get past classes, and state, so you can really enjoy the platform for what it offers-- it's pretty eye opening.
If I take a high resolution image and spawn N*M BEAM processes which each deal with performing processing on some subcomponent of the image, and then use some scheduling algorithm to reassemble the image after the processing is done, wouldn't you have substantial speedups compared to using some other image processing library in a language like Python?
In the same vein, things like matrix multiplications or other numerical algorithms that are known to parallelize well seem to lend themselves really well to BEAM processes, although the libraries aren't written yet since you can trivially call Python from BEAM anyways.
Elixir is thus able to execute Erlang code directly including OTP. This means you can use OTP as you would in Erlang but without the somewhat arcane syntax. Thus it can be said, you can use the "real" thing.
However it is very useful to have access to actor style concurrency outside of the BEAM ecosystem.
One of the really nice things about Celluloid is you can start up the actor system inside an existing non-threaded and non-actor aware system and write concurrent code.
[0] https://launchpad.net/candygram - main page is on SourceForge which is down at the moment, and should be avoided even when it is up.
I'm not sure what makes real difference to Erlang's processes here, I hope someone can explain this with more details.
What I'm learning here is that people don't understand what erlang/OTP is.
Java concurrency primitives like ExecutorService and BlockingQueue with support from libraries like Google Guava (ListenableFuture & ListeningExecutorService and, more loosely related, EventBus), as well as libraries like Netty form an excellent basis for concurrent software development in Java. I've developed a number of event processing type applications in Java (including our email delivery system) using message passing styles and future/promise styles and feel like Java does a good job accommodating them, helping them perform well, and allowing them to be easy to operate and troubleshoot. You can make this work all the way down to asynchronous IO at the operating system level if you wish, although in practice we get acceptable performance allowing that lowest level to use blocking IO for simplicity.
It's really easy to set up a "pipeline" of message processors in a Java application that produce and consume from BlockingQueues; or alternatively, submit work to an ExecutorService or publish an event to an EventBus. Maybe not as easy as other languages or frameworks, since there is some scaffolding, but it feels pretty minimal to me. (Again I say this as someone who has not worked in Erlang or other concurrent languages / frameworks extensively.) Unlike some other concurrent code, pipeline-type code is pretty easy to write, understand, and debug. The hardest part is usually orchestrating safe controlled shutdown.
One area where I understand that Java probably does not compete with Erlang is in the reliability of individual processes and threads and whatnot within a machine. There is no simple way in Java, for example, to continue processing requests in the application while you shutdown and restart with a new version. However, we typically accommodate problems like this at the next level up by routing traffic away from a machine in preparation for it to receive a software updated.
Java's exception handling is also robust. There is not much need to worry about "termination" of individual processes. A top level try/catch block in your message processing system goes a really long way to making the app itself immune to any kind of lower level failure. It might be difficult to convey the supreme confidence of error handling in Java if you've only worked with other languages, but a subjective feeling is: "There is absolutely nothing that can go wrong in any Java code that I might execute that will not unwind in a controlled way through my try/catch blocks and give me a nice, clear stack trace and error message." Whatever thread was handling that work can move onto the next message nicely and cleanly. And beyond this, "There is absolutely no way that any Java code I execute can interact with any part of the application except the objects being passed to it."
People credit garbage collection in Java with making applications and libraries much easier to compose. I personally believe that Java's error handling and behavior containment is also a big part of it. A library that I call simply cannot crash me or interact with anything except what it's passed. (OK, there are some exceptions to these rules, such as code that calls into native code, or running out of heap space, or weird dynamic/reflective stuff, but (i) you can avoid most of them in practice (ii) they don't come up much anyway (iii) exceptions to the rules don't tend to be a problem.)
Application crashes like OOM are also problems that we solve in other layers: if they happen, it means the software is not correct or not tuned correctly, and the crash of an application at this level is handled by the routing layer on top. We don't consider application crashes as something that we need to worry about on an ongoing basis; it's more of a QA issue during development. We do not tend to have systems though where "this particular machine must be available at all times".
None of this specifically supports distributed application development along with concurrency, beyond making message passing easy within the app. That's where client libraries like Netty or frameworks like Akka go much further.
It's not trivial, but Java certainly supports hot code swapping. How hard it is to use it depends on how much you want it to do, like preserve state etc. For stateless services its not hard to load the new version using a new class loader to live side-by-side with the old one, and then kill the old one (automatically) once all its requests have completed, and let the old code be collected by the GC.
It's a neat idea though. Do you know of any frameworks that would make the idea easier to implement? It sounds loosely similar to the Unix availability concept of passing off the listening socket to a new instance of the process, while allowing the old processes to linger until they finish serving their requests.
All other things equal, I favor architectural approaches where I can take any machine offline safely and effortlessly since I tend to need to handle that case anyway for availability reasons.
According to the FAQ, it doesn't create multiple class loaders, it just extends the exiting ones.
Not if you leave the message classes alone.
> Do you know of any frameworks that would make the idea easier to implement?
No, just careful design (JRebel is for development rather than production, as I understand, but I could be wrong) Tooting my own horn (again) but Quasar does hot code swapping for actors (including state preservation)[1]. This is possible (as it is in Erlang) because actors encapsulate their own state.
[1]: http://docs.paralleluniverse.co/quasar/#hot-code-swapping
Look the erlang syntax is not that hard.