Comment by _benedict | Hacker News Reader

Comment by _benedict | Hacker News Reader

Parent Full thread

I believe Fauna inherits some scalability issues from Calvin, in that given NxN communication is necessary for transactions to be processed, a catastrophic failure in one shard can bring the entire database offline. Is that correct?

It may not happen in practice today, but as your clusters grow your exposure to such a failure is increased, and the NxN communication overhead grows does it not?

It's certainly more scalable than other systems offering this isolation today (besides Spanner), but algorithmically at least I believe the proposal we are developing for Cassandra is strictly more scalable, as the cost and failure exposure grow only with the transaction scope, not the cluster.

Not to ding FaunaDB, though. It probably is the most scalable database offering this level of isolation that is deployable on your own hardware - assuming it is? I know it is primarily a managed service, like Spanner.

I'm also not aware how large any real world clusters have gotten with FaunaDB in practice as yet. Do you have any data on that?

> I believe Fauna inherits some scalability issues from Calvin, in that given NxN communication is necessary for transactions to be processed, a catastrophic failure in one shard can bring the entire database offline. Is that correct?

No, that's not correct. FaunaDB is inspired by Calvin, but is not a direct implementation of Calvin. Transactions are applied independently on each storage host, though checking OCC locks may involve peers within a single replica.

> assuming it is

We used to offer on-premises delivery, but we are strictly a managed service now.

> I'm also not aware how large any real world clusters have gotten with FaunaDB in practice as yet. Do you have any data on that?

I'm not at liberty to say.

> Transactions are applied independently on each storage host

I’m not talking about transaction application, but obtaining your slot in the global transaction log. How does a replica know it isn’t missing a transaction from some other shard if that shard is offline come its turn to declare transactions in the log? It must at least receive a message saying no transactions involving it were declared by that shard, no?

This is pretty core to the Calvin approach, unless I misunderstand it.

> I'm not at liberty to say.

I think scalability is something that is a function of both theoretical expectations and practical demonstration.

> How does a replica know it isn’t missing a transaction from some other shard if that shard is offline come its turn to declare transactions in the log? It must at least receive a message saying no transactions involving it were declared by that shard, no?

A transaction is committed to the log with sufficient information necessary for the log (and each storage host) to know whether any particular host is involved in that transaction. There's no need for a message to prove a host's absence from that transaction - it has all it needs in the log to determine that it isn't involved. There is some complexity around how that information maps to the cluster's physical topology, but hosts which aren't involved in a transaction don't process that transaction and need not check with any other host to know that fact.

[ETA] That information is derived from the read set when constructing the transaction to propose to the log. Logical partitions that weren't a part of that read set don't need to know that they weren't involved - that fact is obvious from their absence.

> I think scalability is something that is a function of both theoretical expectations and practical demonstration.

I agree with you, though as a commercial product operated as a service, that practical demonstration is naturally limited in this forum.

You misunderstand my point. A replica may of course safely assume it is not involved in a transaction it hasn’t witnessed so long as it has a “complete” log (as in, the portion involving it).

> that fact is obvious from their absence.

Yes, but absence from what? There must be some message that contains the relevant portion of the log declared by each other shard. If that shard is offline, so it cannot declare its transactions at all, how does the system continue? This absence is one step back from the one you are discussing.

If a shard’s leader is offline, a new leader must be elected before its slot comes around for processing - and until this happens all transactions in later slots must wait, as it might have declared a transaction that interferes with those a later slot would declare.

If no leader can be elected (because a majority of replicas are offline) then the entire log stops, no?

It's true: I've had trouble determining what exactly your point is, as you seem to think you know more about FaunaDB's architecture than you do.

You're under the mistaken impression that we have the same data/log architecture as described in the Calvin paper, which requires that every host talk with every other host. That is not true of FaunaDB.

What you describe is a very important distinction from Calvin, as this mechanism for deriving a global log is core to its design. Have you published this new approach in any public venue?

If not, we’re now in an unfortunate situation where neither your practical nor theoretical capabilities are public knowledge.

> every host talk with every other host

Only every shard

> you seem to think you know more about FaunaDB's architecture than you do

You are perhaps being uncharitable. The only public statements I am aware of about Fauna's capabilities relate to its Calvin heritage. I have only been asking if my understanding is correct regarding both Calvin and its application to Fauna. However, none of the prior issues you responded to were problems with the original Calvin paper either, at least by my reading.

> What you describe is a very important distinction from Calvin, as this mechanism for deriving a global log is core to its design.

The salient aspect of the paper is how it derives the log logically, not how that is mechanically implemented in a real system outside of academia.

> However, none of the prior issues you responded to were problems with the original Calvin paper either, at least by my reading.

I wasn't clear what exactly you do (or don't) know of the architecture which would lead you to believe this NxN messaging story, as that relates directly to your contention regarding the scalability (or not) of the architecture. That much is clear to me now.

LMGTFY:

https://fauna.com/blog/consistency-without-clocks-faunadb-tr...

Thanks

> for scalability, this insertion process happens in batches, and the log itself is replicated and partitioned

This is the link's total exposition on this topic, but it is consistent with my reading of the Calvin paper.

If any log partition becomes unavailable, the log in its entirety becomes unavailable until that partition recovers[1].

Anyway, it feels like we’ve wasted enough of each others’ time without making much progress. Might as well leave it there.

[1] ... and every log partition must communicate with every replica, hence NxN (or MxN)

Reply on news.ycombinator.com