Comment by EvanMiller | Hacker News Reader

Comment by EvanMiller | Hacker News Reader

Parent Full thread

The Erlang VM is a successful implementation of N:M. It addresses some but not all of the issues you mentioned.

* IIRC each new process consumes 284 bytes so it's not "spending lots of memory on creating stacks for your green threads"

* Synchronization is a non-issue as everything is message-passing.

* Disk I/O is blocking, but the VM has dedicated disk threads.

* Dealing with 3rd party code is a problem, which is why relatively few libraries exist for Erlang. However, there is some hope now that NIFs (native interface functions, i.e. C code) can interact with the scheduler, i.e. report how much time has been used and return control if necessary.

You are right that "Building a correct highly concurrent scheduler is no easy task." The Erlang scheduler was not an easy task, and has received a ton of work from extremely talented engineers over the course of decades. Definitely worth a look to see how N:M can be made to work well.

C integration is the killer for N:M threading. C libraries do blocking IO, are not wrapped into a VM, are not compiled for segmented stacks, use thread-local storage, etc. You can work around this problems, which increases complexity. For close C integration, you are basically forced to adopt Cs view of threading.

This is why the R17 release of Erlang might get "dirty schedulers" for this kind of work. Essentially it gives you background thread pools where such work can be carried out.

Currently, you can call C functions directly, but it is not meant for long-running tasks which can block and so on.

If E17 includes this AND maps I might cry (':

You can create threads with enif_thread_create and build a queue to handle long running tasks (or a pool of threads), and use enif_send to return the result. Not quite as hacky as it sounds, since afaik port drivers rely on message passing to return values to the VM as well. Maybe not ideal, but better than naively bumping reductions. I haven't played with enif_timeslice yet, so I'm not sure how it compares (it's newer than NIF threads).

Reply on news.ycombinator.com