Spanner vs. Calvin: distributed consistency at scale
fauna.com
fauna.com
In Spanner's design, on the other hand, a transaction can only be blocked by another one on which it actually conflicts. It will have worse average performance, but better tail latency as things scale.
Perhaps it is just a nit, but the blog post is somewhat inaccurate when it says that Spanner uses the commit timestamp to order transactions. It uses locks to order the transactions, then holds the locks for extra time to ensure that TrueTime order agrees with the lock-based order.
Lest it seem like I'm not actually a fan of the log-based approach, let me point out a way in which Calvin crushes Spanner: write contention. So long as the Calvin transactions can encode the application logic, they can support extremely high write rates on contended objects in the database. Spanner's 2PL, on the other hand, has single-object object update rates visible to the naked eye.
I have a real problem understanding why Google (and now FaunaDB) are so clueless and/or inconsistent about software licensing. Google seemed to finally get it with Kubernetes, which has grown into a real and vibrant ecosystem; I will (and do) happily use Google's hosted Kubernetes product, while I know, at the same time, that I am not locked into a single vendor. With Spanner (and FaunaDB), all I get is lock-in. (The cynic in me suspects that Google's open-sourcing of Kubernetes is a cunning strategy to dominate the cloud by commoditizing container orchestration, and thus undermining competitors such as Amazon. After all, it's not like Google is open-sourcing Cloud Storage or BigQuery either.)
The fact that I can't run these products locally on my dev machine is another major downside unrelated to the philosophical aspects of free software.
I'm sure Google and Fauna will make truckloads of money on enterprises that are willing to bet on them, of course. Personally, I am holding out for CockroachDB.
Most companies we talk to are more worried about being locked into specific infrastructure than specific software, since they would pay for the enterprise or hosted version of the open source database anyway. FaunaDB solves that problem by not being tied to any specific cloud.
There is a developer edition of FaunaDB on the way that you will be able to run locally for free.
You can't run a full spanner cluster unless you have the Google Hardware. Spanner needs to have an accurate clock with a known error bound. This makes it difficult to open source without open sourcing a bunch of hardware.
That's simply not true; there are many consumer-grade GPS clocks out there well within the bounds that spanner operates (iirc skew bounded at +- 5ms).
Not only that, but you can bet that if spanner were open-sourced, you'd see additional pressure to lower the cost of that hardware.
Not that the answer is any more nefarious: They don't open source it because it extensively uses google-only code, and open sourcing spanner would entail open sourcing many, many other parts of code, which is expensive. It'd be much easier to have something like the dynamo db local front to sqlite3 like amazon offers.
Spanner is the software, plus the network, plus the hardware.
(work on Google Cloud)
For transaction resolution it's a bit easier if you are able to assume more about the storage layer's semantics.
For example, if you store versioned values for some bounded period of time (ala MVCC), you can go to other replicas for the version required to resolve a transaction, removing the restriction that transaction resolution must proceed in lock-step across all nodes, and allows transaction reads to route to live peers assuming they have the required version of each read dependency.
Anyone worried about that statement. Ok so they are rare, but what happens when they do occur? File corruption is rare as well but in our large deployment over thousands of servers I see it happen pretty often.
But in general, Google's network is very different than other networks. With their resources they can provide guarantees and capacities other run-of-the-mill data centers can't. So others probably shouldn't listen and assume it applies to them as well.
Reliable enough, at least, that trading off perfect resiliency in the face of partitions is worth it to gain strong consistency semantics.
IMO behavior in the face of partitions is a bit of a red herring. The latency tradeoffs required in a geo-replicated CP vs AP system is much more relevant.
I think the post needs to be more precise about this. Modern networks are asynchronous and so they are essentially always partitioned. And this is important, because later, when snapshot and transactional read latencies are discussed, they are not exactly "very low", not on "AP" systems level. Because "CP" systems need consensus they can't just send requests to all of the replicas and wait for the first reply or even read one locally, "very low" there is more about talking to a remote node "low", with all of the problems associated with it, like poor tail latency, etc.
The fairness, isolation, and throughput (FIT) tradeoffs seem more interesting to me than CAP.
Daniel's post is relevant to more than just FaunaDB though.
Did not know that...why?
Cloud Spanner fully supports interactive read-write transactions.
I'm not sure what the source of the confusion here is. Maybe Daniel is using a new definition of "client-side interactive transactions" that I'm unfamiliar with. :)
Maybe that's no longer the case?
Could you be referring to the fact that we make you do writes via a mutation API rather than DML? Obviously that has no impact on interactivity...
It would be cool to know why Spanner is like that.
> Updates are also supported using SQL data manipulation statements, with extensions to support updating fields inside protocol buffers and to deal with repeated structure inside protocol buffers.
(FWIW, CockroachDB does not have this limitation - transactions can read their own uncommitted writes, just like in other databases)
What other production implementations of Calvin are out there?
My brief summary comparison. VoltDB is a bit less general in some key ways. It tends to have the same performance no matter how much contention there is, which is rare. It's also getting pretty mature, with lots of integrations and hard corners sanded off.
It also typically has much lower latency than these systems, both theoretically and practically.
VoltDev here. Huh? We added this feature in 2011 and read the Calvin paper sometime later IIRC.
If you heard it from Ning or myself then it's probably what happened.
That said, we <3 determinism at VoltDB and rely on it to achieve what we achieve.
Distributed consistency as discussed in this blog post is not really about scaling write throughput (which is the problem solved by horizontal/vertical partitioning), but rather about keeping replicas in sync with each other, seeing the same state. The usual suspects, MySQL and Postgres, don't really have any sort of replication that's totally synchronous (although they're getting there[1, 2]).
As for scaling writes, I agree that chopping up your Postgres database is the safest choice for now, but it's generally true that this is hacky, and kind of a pain to architect your application around it. Solutions like Spanner and FaunaDB aim to free the application developer from this burden, without having to give up the large feature set of relational databases by using a NoSQL database.
[1] https://dev.mysql.com/doc/refman/5.7/en/group-replication.ht...
[2] http://paquier.xyz/postgresql-2/postgres-10-quorum-sync/