Thank you so much for this comment. I have come to a similar conclusion recently -- as long as the P100 latency is acceptable (e.g. all requests are served within 1 seconds), the only thing that matters is the TPS.
Back in 2016, I stumbled upon "How NOT to Measure Latency" by Gil Tene, and it really opened my eyes, especially the part about coordinated omission from benchmark tools.
Of course, after a few days, I also learnt that Gil was behind the Azul Zing JVM with the pauseless C4 garbage collector, and thus it makes sense for him to emphasize on measuring latency correctly.
At around the same time, GCP also boasted about having "consistent single-digit millisecond latency" for its BigTable offering, with Cassandra being the obvious target to attack. I was sold.
Then came Scylla, with a focus on maintaining low tail latency while running on fewer larger machines. I tested one of the early version with cassandra-stress, and the result was worse than cassandra. But I continued to follow Scylla blog posts with great interest.
Recently, I saw https://www.p99conf.io/, and something "clicked" when I read that it was sponsored by Scylla. Suddenly the hype of P99 seems to be wearing off. The video from Gil above is still correct, but I think it is applicable more to exceptional cases, e.g.
- the JVM enters full GC and do no meaningful work for seconds/minutes
- the InnoDB engine stalls for seconds/minutes due to a million different reasons
For normal cases, I haven't found the P95/P99/P99.9 metrics to be that useful. Instead, something like PostgreSQL/MySQL slow log threshold, where anything that exceeds a soft P100 target is logged, seems to be more useful.
Back to the article. If we only concern about TPS, under the "real-life" workload with Gaussian distribution, Scylla beats Cassandra by 2X. So that's it, 2X better performance on one hand, Apache vs. AGPL on the other.
(Of course that's an oversimplification. Scylla's shard-per-core architecture also allows it to avoid some silly single-threaded bottlenecks in Cassandra, but nobody wants to talk about those.)