Serial, Parallel, and Quorum Latencies

Thanks for making this. This is such a cool tool.

Would be awesome to see the tradeoff between network latency and network throughput, assuming that network throughput is not unlimited.

For example, a primary with a 1Gbps NIC may see less favorable latency profiles due to queueing theory if it's replicating to backups in parallel but also ingesting data at a high rate.

This is where matching outgoing bandwidth 1-to-1 with incoming bandwidth, and spreading the primary's load more evenly across all replicas for better stability, but still with 3/5 quorum acks for grey failure tail-tolerance, might show slightly higher latencies, but with less of a latency cliff and significantly higher throughput, saturating the line rate.

We devised this technique for TigerBeetle, and it works extremely well together with Flexible Paxos, since you only pay the serial latency cost for the first synchronous replication hop from primary to next backup (subsequent serial hops between backups can then be async in terms of Flexible Paxos), and this is also ideal in terms of Viewstamped Replication since the next backup will also be the next primary and already have the latest log:

https://github.com/coilhq/tigerbeetle/blob/main/src/vsr/repl...