Hacker News Reader

Comment by dev_dull | Hacker News Reader

> Sending messages between Erlang processes was not as cheap as we expected, and the reduction cost — Erlang unit of work used for process scheduling — was also quite high.

This is really surprising to me, and definitely something the elixir team should look at optimizing. Sending messages should be extremely fast.

View on HN

There are ways to speed up message passing. It's not slow by any means but weird things become your bottle neck when you have 5m concurrent users. For instance you have to build you OTP tree extra layers deep or spawn hundreds of supervisors because just routine calls to the master process to add / remove children begins to become bottle neck.

In my implementation, only 1/2 a million concurrent IOT devices, I use routing tables that narrow down a process to a node and registry cluster (https://hexdocs.pm/elixir/master/Registry.html) , and from that registry it fans out to 1 of 100 supervisors for that worker type per node.

The high reduction cost is deliberate. Reductions in BEAM are an arbitrary value that certain operations in ERTS assigned. Sending to a remote node has a high reduction cost, which means that the more sends you do, the more your process has to be rescheduled to do remote sends. We worked around this with manifold, by limiting the number of remote sends we do to fan-out messages, and transforming them into local sends on the receiving nodes, which has a much cheaper "reduction" cost.

I have no insight into how Erlang performs this process, but I'd assume Erlang is performing sending messages almost as fast as it's possible, as it's a key part of the platform and has had two decades to perfect it. Most likely this is a limitation of the paradigm (of the Actor model) and not the implementation.

I believe the bottleneck the author was trying to overcome here is how the Erlang VM moves processes to the back of the run queue once they hit a predetermined number of operations (preemptive scheduling) and not of the Actor Model.

If the message is sent to a remote host then perhaps the send_no_suspend function could help. I know it is used in core parts of erlang telcom systems with several millions of users to work around performance issues

http://erlang.org/doc/man/erlang.html#send_nosuspend-2

That's something that the BEAM group would work on, although jose has contributed some patches to the compiler.

Reply on news.ycombinator.com