Tell HN: I'm writing an Erlang recipes book, are you interested?

I got it and I like it quite a bit. It is a good introduction book for practical Erlang use, since you learn to do things the "OTP way" right off the bat.

It has examples on how to build network servers (TCP RPC server, HTTP server, a caching service, setting up a distributed system with multiple nodes, resource discovery, logging, release managing, interfacing with C code and many other useful things).

What you need to do then is look into a few different options.

The first one is distributed Applications. This is part of the standard OTP stuff. What this does is declare a node as a master, and then failover nodes for your OTP application. When your master goes down, the other nodes take over it. Whenever the master is back, the failover nodes drop their applications and restarts it on the master. This loses all of the state if it hasn't been kept in a safe place.

Another one is to look at gen_leader. gen_leader is not part of OTP, but has been written by Ulf Wiger and is available on github or through agner. A quick description of it would simply be to say that it's a boosted gen_server that will find a master of itself somewhere and automatically replicate its data everywhere else. If the master dies, a leader election occurs and a new one takes over.

A third option is to have a masterless concept. This is what happens in Riak. Data has to be shared in clever ways and code organized to support such a design. Nobody's a master, everyone's a follower. You add redundancy here and there, push important state to a safe spot, etc. In the standard library, modules such as 'global' handles this kind of thing.

Supervisors are not exactly made to cross node boundaries. They protect your local processes. If you need something more complex to keep your state alive, this usually happens at a higher level for OTP.

Heart does solve it in a way, yes. Any Master/Slave pattern where a masterless slave becomes a master also solves it. The complex cases I had in mind happen when you dynamically add more workers and try to set up crazier links (A is the master of B, B is the master of C, C is the master of A), although there are few practical uses of that. You'd have to try to shoot yourself in the foot I guess. I see it as a bit hard to provide a generic OTP pattern to do that kind of thing past the standard takeover/failover setup (and heart for VMs)

No, this doesn't break the async philosophy. The synchronous sequence of calls is done by a question/reply made out of 2 asynchronous calls. It's just that simple to make something synchronous, as opposed to turning something synchronous asynchronous (by using a queue).

By doing this you are not limiting the queue to the size of 1. If you have N producers, you can have more than N elements in the queue, but the idea is that each producer can only enter them one by one, with the agreement of the consumer. Nothing would stop the consumer from accepting a thousand of them before waiting before replying, therefore slowing down the producer. The idea is just to limit the rate at which you accept queries by tying it to the speed of the consumer.

If this is not enough, you can look at things like the jobs tool (https://github.com/esl/jobs) to work with scheduling.

By the way, you're mistaken on the idea of message sending always succeeding. The idea is that you will never error out because of trying to send a message (send and pray), not that the message will make it to destination. If you want any guarantee that the message made it, then you need that acknowledgement async message being sent back to you.

If you want a message queue bounded to length N, then make the messaging protocol such that the sending process sends up to N messages without an ack.