Decentralized cluster membership in Rust

Thanks a lot.

This is correct yes. Rumor-mongering is more efficient both in term of detecting/propagating a node failure and in term of network overhead.

We stopped using SWIM because it is too hard to get right, and because scuttlebutt allows all nodes to expose a bunch of meta-information about themselves.

It took a lot of time and effort to Hashicorp to patch memberlist into what it is today.

- In term of message overhead there is not much difference. one details is that each node keep incrementing his heartbeat counter continuously, making it a state change that need to be propagated.

- slower here depends on the number of node selected to gossip with. if you choose a long gossip period yes, but in real system the gossip period is short enough to not notice it. So slower means compared to Rumor-mongering style. Not as in having an inconsistent cluster state.

-(Thanks) I will probably amend this as a note: A seed node can be any node in the cluster we know is reliable and will almost always be there; kind of like the ACE within your cluster, they can be many.

-Garbage collection :"... We mitigate this by periodically garbage collecting obsolete nodes."

The source itself is commercial AGPL v3.0. I'm guessing that this was an accident so I opened https://github.com/quickwit-oss/chitchat/issues/42 to clarify.

Awesome!!!

Unfortunately, there exist two different distributed protocols named scuttlebutt. The one referenced here is a gossip algorithm, the one you mention is a social network protocol.

They appeal to different enough spheres and scuttlebutt is such a good name I can't fault whoever moved second on that one.

No it's not.

Thanks a lot, you can always file a issue on https://github.com/quickwit-oss/chitchat. We can improve the documentation, add more example and mostly learn from your use case.

Usually that's modeled in the application layer. In scuttlebutt you'd store a higher-level state as one of the node-specific parameters. The gossip protocol doesn't need to know that you know the node knows it's about to die, though there are optimizations to be made there.

Despite what the sibling commenter claims, in a production system you absolutely need that distinction -- not in terms of the reliability of the low-level protocol, but in terms of the application features you want to expose. A node about to be removed needs a chance to drain active connections.

Then you have to differentiate between accidental and intentional death. Much more robust not to make that distinction.

Depending on the need, it can be implemented on client side as well as on chitchat. We actually thought of this feature as a way for a node to normaly exclude itself from the cluster (maybe for application update). We are still evaluating how necessary this is because a node crash or normal shutdown can just work. Future experience will surely tell us.