Caramel: An OCaml for the Erlang VM

You're absolutely right: with Erlang you have to spend a great deal of effort planning for, implementing, and testing code that can work correctly across nodes with different versions of the code and/or versions of Erlang itself.

I think the open question is whether strong, _static_ type system can handle it at all.

If you follow the philosophy of Erlang, you can take shortcuts to strongly thinking about it.

On the server side: what happens if a process geta a message it doesn't handle? Either it will crash (and be restarted by supervision, hopefully), it will receive and discard it (possibly logging it), or it will ignore it and leave it in the message queue (not great for long term memory use).

On the client side: what happens if the process sends a message that is ignored or crashes the receiver? The client will timeout, and won't know if the work was done or not; best to bubble up the error, perhaps by crashing (and maybe restarting by supervision).

If you're ok with these 'worst cases', you don't have to proactively deep think. The proper deployment strategy is clear: deploy servers that handle new messages first, then deploy clients that send new messages, but if you mess the order, you'll be within your failure model. It gets more tricky when there's a difference in data types and a piece of data could be used by old code, then new code, then old code, of course.

Of course, the actual worst cases are more fun, but less likely to happen. You could find a bug that crashes BEAM or something; I don't think that would be easily found by sending a unexpected message, but maybe (depends on what you do with unexpected messages).

But if your static typing just says yeah I dunno, it could crash, I don't think you've gained much from the exercise. I'm probably biased against static typing, but if it can model the system and provide useful insights that are accurate on the system, that sounds helpful; if it can't model the system though...

As I understand it, there are real questions about the type theory needed to correctly express what the BEAM does, and they have yet to be answered sufficiently. Nonetheless, a lot of people clearly want static types in their BEAM applications, and there are certainly parts of any application that don't cross the network. I'm interested in efforts like Caramel and Gleam mostly because of my interest in BEAM as a general-purpose platform, and because I want to see if they come up with solutions to the type theory problems.

I think what the parent is saying isn't that most production Erlang apps aren't clustered; it's that most production Erlang apps aren't clustered using the distribution protocol. Instead, they're clustered at the application layer, with explicit wire protocols like gRPC. You can upgrade nodes within such clusters just fine, because the ABI of the wire protocol is an explicit part of the application, something that can be stabilized separately from the app itself, rather than being an implicit property of what ERTS version you're using, or about how the application's processes communicate internally.

Note that even in the context of Ericsson, Erlang's distribution protocol was never designed for the use-case of horizontal N-node clustering. It was designed for static-role clustering — where each node is like an organ in your body, with a name and a specific function relative to the "system" that is your body. For example, "master" and "warm standby" roles. (By analogy to Kubernetes, the Erlang nodes of a distributed-OTP architecture are closest to being like the sibling containers of a single k8s pod. Except that Erlang's "containers" can be running on separate machines while still being part of the same "pod".)

If you want to scale such a system (a "pod" of nodes), you are supposed to spawn N copies of the entire distribution set; and then those N system-copies will coordinate with other such clones not via the distribution protocol, but via explicit coordination protocols. (Sometimes this explicit coordination protocol is designed to use the distribution protocol as a carrier — this is one of the reasons that Erlang supports making manual distribution-protocol connections between nodes, rather than forcing a connected topology on you. But such connections carefully avoid passing arbitrary RPC data across them, instead usually having an explicit "peer server" on both ends, where the servers speak a limited and ABI-stable term protocol to one another.)

Yes, the Erlang distribution protocol has been repeatedly taken beyond its design tolerances to accomplish horizontal scaling. CouchDB does this, for example. But many Erlang applications that you might think of as doing this, are actually much closer to static "organs in a body" architecture than you'd think, despite claims of scalability. Riak and Ejabberd, for two examples, both treat their nodes very much like static organs, rather than a fleet.

> Kubernetes does not solve the issue, it merely has strategies for how to upgrade nodes. The issue is that...

I understand the issue, Gleam/Caramel/et al. (static typing layers on BEAM) are not trying to solve this issue.

> So my assumption here is that they're putting ocaml on the BEAM to utilize OTP and clustered apps, correct?

That's actually not my assumption. I'm assuming that the point of Caramel is to get sound static typing for (some of) the code that we are deploying on the BEAM. It doesn't necessarily have to be for clustered apps. As I said, many deploys of even Erlang and Elixir today are not using clustered nodes. They don't care about those capabilities. They use Kubernetes to manage nodes and state.

> instead it seems like you'd be forced to spin up entirely new clusters and then direct traffic to the new cluster to bypass this problem.

This is exactly what Kubernetes (or Nomad etc.) will let you avoid--they can spin up and spin down nodes within the same cluster.