It's always about trade-offs. Citus has ACID transactions within nodes, and ACD transactions across nodes. It does not have distributed snapshot isolation, but that means it also avoids the latency and concurrency penalties that come with distributed snapshot isolation.
Because clients talk to Citus using the synchronous postgres protocol, latency is one of the most important factors in performance. A single connection can never do more than 1/latency transactions per second. While you could use more connections, that is often complex, not always possible (e.g. long transaction blocks), and also comes with a significant performance penalty (fewer prepared statements, high SSL overhead, ...). It's important to optimise for latency.
We find the Citus transaction model is sufficient for most purposes. If you think of the typical balance transfer example, then there is no way to double spend or overspend. Distributed transactions are atomic and locks in postgres will serialise distributed transactions that update the same rows and ensure constraints are preserved. Importantly, Citus has distributed deadlock detection, which allows generous concurrency at almost no cost to the application (apart from maybe having to reorder transaction blocks).
Postgres supports different modes of replication, including synchronous replication to avoid losing any data on node failure. Many users actually prefer asynchronous replication, to avoid the extra round-trip and the broader performance implications that would have. It's also the reason we use streaming replication over other replication mechanisms that might give higher availability. Streaming replication supports higher concurrency and thus gives better throughput.
At the end of the day, people use Citus because a single node postgres (or MySQL) database is not performant enough to keep up with their workload. Performance is key. Citus is not a silver bullet for every postgres performance problem, but it has two particular strengths:
Minimal pain scaling out for multi-tenant applications that can shard by tenant.
- Any SQL query is supported when filtering by partition column, including CTEs, window functions, complex joins, etc,
- N times more CPU, memory and I/O bandwidth to process queries, and 1/N as much data to read
- High throughput, low-latency transactions, including UPDATE/DELETE with subqueries
- Citus Warp can do live migration from postgres (e.g. RDS) using logical decoding
- Major ORMs are supported after small data model changes
Superpowers for real-time analytics apps (e.g. dashboard).
- Bulk loading, up to millions of rows/sec
- Massively parallel SQL with subqueries containing outer joins, lateral joins, etc. across all shards when joining by partition column
- Massively parallel roll-ups (again with joins) using INSERT...SELECT
- Massively parallel DELETE and UPDATE
- Integration with other Postgres extensions, such as HLL, PostGIS, pg_partman
- Advanced index types, including GIN, GiST, partial indexes
- Table partitioning (for bulk delete, faster scans)
Citus has a laser-sharp focus on addressing these use cases and makes certain trade-offs because of that.