freels

Thanks for the shoutout. At some point if you find yourself with some spare time you can check out our new FQL version. It's closer to JS in terms of syntax now, but still a small, relatively functional language.

One might say uncomfortably slow.

The NoSQL movement overstated how important the A in CAP was. Relatively few systems need _all_ nodes to be available, compared to those that benefit from strong consistency, especially those that live primarily in datacenters/the cloud.

But this strict notion of Availability (all nodes must be available), was conflated with being available at all, leading to CP systems being disfavored.

When we introduced Fauna, it took quite some time (and a Jepsen report) to convince others that building a CP system without exotic hardware was possible, and that in practice, access to multi-region strongly consistent replication is a far better availability experience than the typical single region deployment topology which is still most common today.

I'm clearly biased, but at least in my experience, while this is technically true, you're still dealing with XML (and now JSON) shoehorned into a tuple-based context. In other words, there is still a (lossy) translation layer, it just happens to be in the RDBMS rather than in-app.

Fauna's advantage here is that this way of structuring queries is deeply integrated with the language (and underlying wire protocol) itself. For example, Fauna's response format supports returning independently iterable result sets (supported by cursor-based pagination under the hood), allowing you to lazily populate the result graph in your app based on further user interaction.

If your current stack is working for you, that's great. The main advantages of Fauna vs say MySQL are going to be:

- Fauna is distributed and multi-region and therefore more resilient to hardware or regional outages (for example we barely noticed the last AWS us-east outage, except for the fact that it affected customer traffic to Fauna).

- You gain a lot of flexibility in terms of where and how you deploy your compute layer. Fauna works very well in concert with serverless platforms or edge-based compute like Cloudflare Workers. It's also possible to connect directly from your client/front-end, using Fauna to enforce end-user permissions.

- Even if you know SQL, it's worth checking out FQL. Simple queries in SQL are also easy in FQL, but more importantly, FQL gives you much greater control over the shape your query result, meaning you don't need an ORM to reconstruct your object graph. If you have ever used GraphQL, the experience is similar. Or you can see a few examples and comparisons with SQL on our FQL product page: https://fauna.com/fql

Getting the best of both worlds is what we're trying to achieve with Fauna, at least for OLTP use-cases. I'd be curious to hear what challenges with you ran into with other NoSQL databases, though.

While, making it easy (and possible in the first place) to write procedural code directly in a transaction is a core part of FQL's design, it is certainly possible to implement safe read-modify-write via FQL, it's just less efficient.

They don’t bother and just use search: https://futurism.com/the-byte/gen-z-kids-file-systems

There’s no catch. No database is going to work for every use case, but we’ve built FaunaDB to be general purpose, and suitable for many of them.

Fauna, Inc. | Remote | Full-Time Opportunities | https://fauna.com/company

At Fauna, we’re building the serverless database for modern apps built on JAMstack. If you want to work on systems and challenges related to serverless databases, GraphQL, and JAMstack, Fauna is hiring!

All Positions: https://fauna.com/careers

Highlighted positions:

VP of Engineering - US REMOTE: https://hire.withgoogle.com/public/jobs/faunacom/view/P_AAAA...

Software Engineers-Scala/Java - REMOTE: https://hire.withgoogle.com/public/jobs/faunacom/view/P_AAAA...

Calvin has been an elegant protocol to work with in practice, and has pretty radically simplified FaunaDB's implementation of transactions compared to classic 2PC. Writes are committed in one global communication exchange, read isolation is pretty straightforward, and not requiring transaction recovery cuts out a significant amount of complexity which tends to be overlooked.

In talking with others, my best guess as to why we've seen relatively few implementations of it in the wild is that it is just less well understood compared to 2PC, so misconceptions propagate. The original paper focuses on how it works, rather than how to apply it in detail to generic transaction processing, which perhaps is a shame in hindsight considering that is where most of the confusion lies, IMHO.

For example, there is no reason stemming from Calvin that FaunaDB cannot provide full SQL-style session transactions. We chose not to implement them because they aren't a good fit for the core use-cases the system currently targets. Specifically, interactive transactions are too chatty for client-server interactions over the internet where link latency dominates an application's perceived speed: Instead, FaunaDB's interface encourages packing as much logic into as few requests as possible. (But I suppose that's a topic for another comment thread.)

Yes, I was not trying to imply the bugs Jepsen found haven’t been fixed.

However the possibility of causal reverse is a characteristic of CRDB’s architecture, so you can still observe it in the latest version.

Isn’t that just describing reintroducing waiting out the uncertainty window in order gain back external consistency? I’m not sure that ends up being substantially different from TrueTime.

I don’t know of an article describing it offhand. The issue arises where each transaction occurs on a different node or set of nodes, and the causal link between them is external to the dbms. In this case, it’s possible for each transaction to independently commit without the necessary message exchanges to ensure their HLC based timestamps correspond with their causal order. In other words since the client doesn’t participate in the HLC scheme, it breaks the causal chain and can allow an HLC-based system to assign out of order timestamps.

This is hinted at in the HLC paper but it’s not very clear: Section 2 limits the definition of “happens before” to only apply to events which occur on the same node or after a message is received from another node.

Cockroach Labs themselves freely admit CRDB allows for causal reverse anomalies [1], and Kyle reproduced it in his Jepsen analysis [2].

HLCs don't prevent causal reverse, because they don’t guarantee message exchanges synchronously happen across nodes between transactions. Which most HLC implementations piggy back on to properly advance the HLC on each node.

1: https://www.cockroachlabs.com/blog/consistency-model/

2: https://jepsen.io/analyses/cockroachdb-beta-20160829

As others have commented, most widely deployed DBMSs provide deadlock-free serializability. FaunaDB is a commercially available implementation of Calvin that makes deadlock free serializable transactions performant in a global environment. The technology exists, and I think your concerns are unwarranted. The consequences of performance issues are far less severe than data correctness bugs, so preserving correctness is a much better starting point.

This advice is highly implementation specific and unnecessarily dangerous. For read only transactions, a well designed system will be able to provide SNAPSHOT isolation with little negligible perf impact compared to READ UNCOMMITTED. When combined with SERIALIZABLE write transactions, reads at SNAPSHOT are equivalent to SERIALIZABLE.

For transactions which must write, the entire point of this article is that anything less than SERIALIZABLE leaves you subject to anomalies whose affects on your data can be very difficult to predict beyond trivial scenarios.

If you must lower consistency for performance's sake, then so be it, but I would strongly argue that should be the exception, not the rule, and not be a decision made lightly.

Serializable Snapshot an implementation strategy for providing Serializable isolation. In PostgreSQL if you ask for Serializable you get SSI.

Sorry I can't let this go unchallenged. Again, you are inventing an architectural deficiency where the is none. The log in Calvin is partitioned and does not require all transactions to go through a single physical consensus leader. There is no single node in a Calvin cluster which must handle the entire global stream of transactions. The Calvin paper itself extensively covers how this works in detail: http://cs.yale.edu/homes/thomson/publications/calvin-sigmod1...

This take on Calvin is inaccurate:

Under contention, a Calvin-based system will behave similarly to others which use optimistic locking schemes for Serializable isolation such as Postgres, or YB itself. There are advantages to the Calvin approach as well. For example, under Calvin, the system doesn't have to write out speculative intents to all data replicas in order to detect contention: The only writes to data storage are successful ones. The original paper only describes this briefly, but you can read about how FaunaDB has implemented it in more detail: https://fauna.com/blog/consistency-without-clocks-faunadb-tr...

It's also not a stretch to see how the protocol described in that post can be extended to support session transactions: Rather than executing a transaction per request, the transaction context is maintained for the life of the session and then dispatched to Calvin on commit. (This is in fact how we are implementing it in our SQL API feature.)

I would instead say that one of the more significant differences between Calvin and Spanner is the latter's much stricter requirements it places on its hardware (i.e. clock accuracy) in order to maintain its correctness guarantees; a weakness its variants also share.

We are always trying to reduce the learning curve of FQL, but we've gotten a lot of good feedback from those who do take the plunge. That being said, I'd love to provide SQL support eventually, but there were a few reasons why we don't support it as the primary query language for FaunaDB:

Our core use-case is OLTP, and we wanted to address the shortcomings of SQL in this context, such as query performance being highly unpredictable in the presence or not of indexes, whims of the optimizer, etc.

SQL is just not great as an application-level interface: Tables are not a natural fit for many data models (the classic impedance mismatch problem), programmatic composition is difficult. We want to obviate the necessity of an ORM library on top Fauna.

SQL the language is not great for writing complex transactions in, and session transactions require a lot of back and forth between the client and database. We wanted to make it easy to write queries which can encode as much business logic as possible, hence FQL's semantics are a lot closer to a regular programming language.

It does handle contention well, we haven't emphasized that point well enough yet. Writes never contend on conflicting reads, and serialized reads never contend at all.

Accurate clocks are not enough... to really get the benefits that Spanner alone enjoys, you have to have a TrueTime equivalent service available, and it has to be rock solid. As well once your system is sensitive to clock skews in the milliseconds, you start having to care about things like the leap-second policies of your clock sources. All in all, the resiliency tradeoffs are a significant downside to relying on clock synchronization, which is why we did not pursue a transaction protocol dependent on it.

No, in FaunaDB, a replica is a group of nodes, within which each node only stores a part of the dataset. One thing that's different from other systems is we make replicas a first-class operational unit to think about.

The cluster topology which was tested was a 3 replica cluster with 3 nodes each. Each node contained 1/3 of the dataset.

It's definitely recommended, depending on what you're trying to get out of the result. Happy to help out with query patterns on our community slack https://publicslack.com/slacks/fauna-community/invites/new

Just because failure can happen doesn't mean it's frequent. For the record AWS does this to protect against correlated failure. At their scale, they assume they have some percentage of local nodes down at any given time, so have designed the system to tolerate both network partitions and local failure at the same time.

Unless you have specifics, I’m not sure this comment stands up to reality. Modern networks are pretty reliable in my experience. The state of the art in consensus reduces learning to one round trip. Calvin further eliminates all but one global RT in distributed transaction commits.

Additionally, FaunaDB gives you the tools to work within a global environment with _safe tradeoffs_. For example, reads at SI/Serializable can be fast and uncoordinated. You choose per-query.

The latest released version, 2.6.0, has most of the fixes as outlined in the post & report. Minor/ergonomic fixes are forthcoming.

If you’re running on cloud, we do the upgrade work for you.

The constraint that all observers see T1 before T2 because T1 is acknowledged before T2 begins is the “in real time” component of a strict serializability guarantee. Mere serializability does not guarantee this, as unintuitive as that sounds.

Jepsen.io has a good explainer: https://jepsen.io/consistency/models/serializable

It is 3 transactions. Alice deposits money into her savings acount, then subsequently withdraws money from her checking account. In the meantime, some concurrent process transactionally reads the balances of both accounts.

Since there is no internal causality tying transactions 1 and 2 together, transaction 3 is allowed to arbitrarily order them without violating serializability.

I think the answer to your second question comes down to what anomalies you are trying to protect against. In order to prevent causal reverse, all read-write transactions must be strictly serializable, but it is safe to reorder read-only ones.

Usually 2PC is used to run transactions across multiple shard which each have a subset of the total dataset. Though it doesn't provide you with any extra reliability.

There's a tradeoff in sync vs async replication... a node which you synchronously replicate to won't serve stale reads, but then it becomes part of the critical write path. Conversely when reading from a async node it's very possible to get a stale read. (Alternatively, systems like Cassandra allow you to read from multiple nodes and return the latest result).

Note, I don't believe it's necessary for _all_ transactions to be strictly serializable, but I would make the argument that all read-write transactions should be in order to prevent the anomaly in the article. In other circumstances it's reasonable to make the tradeoff and accept potentially stale data.