A gentle reminder that FoundationDB exists and has this nailed down really well. They are just bad at marketing, so it's not in fashion. But do check it out if you want a distributed database with strict serializable semantics, that works.
I am guessing maybe it's Apple's corporate secrecy that is the issue. Apple likely has a massive deployment of this tech.
Secondary indexes in "distributed strongly consistent" systems is what ruins performance: because each index is +1 write to another "index table" (... +1 raft quorum check!).
I don't think FoundationDB has "secondary indexes" to begin with, so one may never run into the +1 write per index issue.. it's just a layer on top of TiKV (equivalent to RocksDB in CockroachDB).
Can't secondary indexes just be implemented in your state machine?
I.e. your state machine handles the insert and also writes a secondary index at the same time. Every state machine could do this identically off the log.
Secondary indexes are implemented as an additional smaller table in most databases (Index -> Primary Key), which requires a write, too.
Don't take my word for it- run some benchmarks yourself to see the performance cliffs appear in these "distributed strongly consistent" databases when you have a handful of indexes.
It's why many require 10x the hardware to scale past the performance of a single node postgres / mysql.
> Calling FoundationDB "just a layer on top of TiKV" is… well… :-)
FoundationDB recommends running TiDB ontop to get an SQL layer...
There are some well supported document interfaces like the official record layer or Tigris though.
You might be mixing up projects here - TiDB/TiKV are completely separate projects from FoundationDB. I don't think FoundationDB has a supported SQL layer.
> Don't take my word for it- run some benchmarks yourself to see the performance cliffs appear in these "distributed strongly consistent" databases when you have a handful of indexes.
I'm not saying secondary indexes aren't a performance cliff. I'm saying I can't think of a reason they must require additional quorum on top of the initial message that would require a secondary index update.
Since you said:
> Secondary indexes in "distributed strongly consistent" systems is what ruins performance: because each index is +1 write to another "index table" (... +1 raft quorum check!).
This depends on the specifics of how your database works, of course. I’m not sure this would involve multiple consensus quorums in FDB for instance, given that I believe it instead relies on a centralised timestamp service.
Calling FoundationDB "just a layer on top of TiKV" is… well… :-)