No More Callbacks: 10,000 Actors, 10,000 Threads, 10,000 Spaceships

It sounds like you're saying that it wasn't faster? Can you explain a bit more about the motivation if so?

Any thoughts on whether we should have user-space threading in the JVM today? Maybe it could be implemented in RoboVM as an experiment (rather than wrestling with the JVM immediately)?

These people people got up to 2M concurrent TCP connections

http://blog.whatsapp.com/index.php/2012/01/1-million-is-so-2...

And on top of it, it is using isolated heaps. That is beautiful I think. Completely concurrent GC is beautiful.

I know Erlang syntax is not to for everyone's taste (I do like it though). There is also Elixir (http://elixir-lang.org/). But the underlying BEAM VM is an awesome piece of technology.

> The thought was to have a billion threads on a SPARCStation 10

There are some green-thread C libraries I've been playing with, like Protothreads ( http://dunkels.com/adam/pt/index.html ) and libconcurrency ( https://code.google.com/p/libconcurrency/ ) I think they use setjump/longjump trick. Some use setcontext()/getcontext() POSIX call.

I still like Erlang VM best.

That's 3 years ago. I'm sure more can be handled now, with more memory and more cores.

Bundled Netty with a distributed system like Hazelcast, you can easily scale out to more machines.

Akka doesn't have lightweight threads at all. You implement a message-handling method, but it must not block on, say, a DB call, lest it block the entire thread it runs in. An Akka actor simply must not issue a DB call: it's as simple as that.

With Quasar things are different: you pull messages rather than implement a callback; you can block: on IO, DB, lock or anything else. The programming then is not only simpler, but also more powerful. For example, Quasar supports selective receive - just like Erlang.

Because actor is an establish paradigm and that has been around for a while. I haven't heard Hickey's talk on "agents" but based on your description, how are agents radically different?

What stops actors from reading the snapshots of the world? They can 1) subscribe to "world" actor and get publication when it changes or 2) if database is immutable (I guess you are hinting at Datomic or Clojure's datastructures here?) an actor can also call a function. Remember actors in practice are there to help isolate concurrency contexts. Reading truly immutable data is safe so actors could just periodically read this immutable data (just think of the database a function). It would be awkward having to process messages from mailbox, timing out, then reading world state, process world state, going back to processing messages. Etc. I like 1) better.

> the actor has to define its behavior.

How does an agent bypass defining its behavior? Doesn't an agent have a piece of code that specifies what that agent does.

> and send transformation functions on the data of that specific agent, which will be then processed by the agent's thread in an ordered way.

So this basically centralizes the state of all the agents in one central location that is an immutable database? Hmm interesting. It is a different way of looking it at it I guess. Each actor usually handles its own internal state privately. I guess we also assume that there is something underneath that constantly distributes all these incrementing tree of states across a whole system. I don't know I would rather think of actors explicitly choosing to send their state to a system half way across the world rather than rely on another layer of distributing state. Maybe it is just a matter of a mental model here...

Having said that, I do think actors have some advantages over agents when it comes to fault tolerance. Actors better isolate and communicate faults.

Threads are the WorseIsBetter approach to concurrency; they're incredibly simple to implement, but that just means that the difficulties are pushed on to the users (ie. developers using the framework/library).

Threads may be a good idea for code which has no 'design flaws' and is not 'too complex', but as we all know everything has bugs and everything is more complex than it seems. The arguments in favour of higher-level concurrency models are basically the same as for tests and version control: if you don't use them, you're making a dangerous gamble which may cause a large price down the road.

Concurrency models like callbacks and actors can make dangerous things more difficult; if we use the callback examples from the article:

> It’s hard for a programmer to reason about which line of code executes on which thread, and passing information from one callback to another is cumbersome as well.

Of course, this is the point of callbacks. The callback model tells us to reason using function arguments and function calls, so of course we can't map lines of code to threads, since neither lines of code or thread have any place in a callback model. Likewise for passing data between callbacks; the problem with threads is that everything is shared all of the time, which makes it incredibly difficult to enforce invariants. When using callbacks, everything is local by default and transfering data between threads requires explicit channels, eg. free variables.

In the actor model the safety comes from messages having no ordering or latency guarantees, so we can't assume that our data is always up to date.

With higher-level concurrency models we end up screaming at our IDEs as we try to contort our code to fit the paradigm. This is how it should be, since this means nothing's gone wrong.

With low-level concurrency models, the machine gladly accepts our dangerously broken code, the number of interleavings is so huge that our tests never hit an error case (or more likely, some of the bugs are so obscure that it never occurred to us to test them). Six months later the application explodes and as we sift through the pieces we find the true extent of the problem, and discover that subtley corrupt output has permeated through every aspect of the business and we can't anything that's been done since that code went live.

You are mixing the two concepts. The distinction is between threads/actors vs callbacks. Not threads and callbacks vs actors.

If you disregard isolated heaps and memory an actor is just a thread plus a queue. Other threads write to the queue and the threads gets messages from queue and executes them.

The real distinction is callbacks vs threads and actors thought. And I agree with original point. Callback-based concurrency is more complicated and more challenging to write than threaded-based concurrency.

> Also, in lots of languages multi-core != concurrent. You can have 10,000 actors using a single core. In fact writing a scheduler that can efficiently distribute actors between different cores is probably where the complexity Doron Rajwan refers to lies.

What do you mean by multi-core? Languages don't come with cores, hardware does. Do you mean that CPU-bound units of concurrency (threads, actors, processes, co-routines) can be dispatched onto multiple CPUs if those exist? Yeah some languages (or more precisely their runtimes and libraries) can't do that. Like Python has the GIL so CPU bound threads can't work. But threads work great for IO bound threads.

But have you given any thought to the critical and urgent problem of running 10,000 Actors, 10,000 Threads, and 10,000 Spaceships?

Scala gives you no advantage here. None of its concurrency constructs really require Scala. There is no reason not to implement them in Java and use them in any JVM language. More specifically, Quasar actors are more general and powerful than Scala actors because they run in true lightweight threads and can block. Also, a lot of people don't like Scala.

Comments "...10,000 Co-routines ... "

;-)

In Python with greenlets, state can leak between green threads through module globals and other module state.

Erlang is a different matter. We love Erlang. But the JVM ecosystem is not only two or three orders of magnitude bigger, but the JVM serves other requirements as well like excellent performance (performance is not Erlang's strongest suit, and more than a few Erlang projects require C code to meet performance requirements).

Some projects will be best served by Erlang, but many will benefit from Erlang's capabilities on the JVM.

In short – Erlang is awesome, the JVM is awesome, extremely popular and very successful. Why not combine their strengths? We already have a full Erlang implementation for the JVM, and we think that Pulsar (Quasar's Clojure API) really brings together the best of both Erlang and Clojure.

There are many technical advantages to using the JVM, too. Because it has really good low-level concurrency constructs, you can implement state-of-the-art concurrent data structures in Java. This is downright impossible in Erlang, as in all pure functional languages. This is as it should be, because these languages work at a higher level. The problem is that BEAM, Erlang's VM, operates at the same level, too, so if you want to write a concurrent DS for Erlang you'll need to do that in C. It's a lot harder than it may seem, because many of these data structures require a good GC, and the BEAM the GC can't help because it only manages process-private heaps.

In fact, the code is very naive, and that's our main point. Even naive code can scale well with this approach. We care more about scaling than sheer performance.

So your calculation is wrong, as we're not trying to approach a theoretical limit (which would require optimizing the algorithm) but to demonstrate scaling of a naive algorithm. For example, instead of a single spatial join, each spaceship queries its surroundings: this is asymptotically (n^2 vs n) worse than a single join.

So it's not synchronous in the sense that it's running everything sequentially.