Make your cluster SWIM (2020)
bartoszsypytkowski.com
bartoszsypytkowski.com
A popular implementation of SWIM you can use with just about any system (if you can handle another runtime dep) is HashiCorp's Serf. Serf does some interesting stuff on top of SWIM; it does full-node sync with TCP to speed up convergence, for instance, and, most notably, implements a network telemetry system on top of it to estimate all-points RTTs for everyone in the cluster.
For any Someone Who Isn't Me who is also curious: it stands for "Scalable Weakly-consistent Infection-style Process Group Membership Protocol"