TiDB – a global scale distributed DB

The VoltDB post is also wrong in many of its particulars, but I’ve gotten into that argument on this site way too many times by now (and last I checked John agreed that some of its claims were overly hasty).

If there’s a particular question about FoundationDB’s performance characteristics that you’re curious about, I can probably give you a straight answer provided that it was once public information.

And I still can't respond to equally uninformed speculation about the FDB transaction, because that was confidential.

But I can say that the article's technical thesis about the design space is wrong. A layered design (and, really, all databases are layered, so we just mean a design where distributed concurrency control is below the layer of the data model) is good for performance, exactly because it makes it practical in engineering terms to use the right data model for the job, often getting asymptotic wins rather than tiny constant factor ones. Row and column stores, hierarchical document or "table family" stores, conventional indexes, compressed bitmaps, spatial indexes, etc can all be reasonably efficiently mapped to K/V pairs and then take advantage of the same fast distributed storage and concurrency control. You can use them for different indexes in the same database or the same table. You can, just to throw out an example, store recently changed rows in a row store while transactionally moving them into a column representation in the background, and combining both seamlessly for reads. And you can have things that aren't a good fit for the relational model, like queues, graphs, and full text search indexes, in the same database and operate on them transactionally. You can still co-locate compute with the data, and you can actually scale indexes because they don't have to be co-located with the data as most databases wind up doing. And let's not forget that you have at least a chance to actually get the hard distributed systems part right, as so few products do, because it is not hopelessly entangled with your data model and execution engine.

To address some specific speculations from the article: you don't have to read metadata every transaction; you can just cache it in memory and just ask the distributed store to make sure it hasn't changed in conflict with the transaction (via a tiny read of a serial number, for example, or a usually-empty range read of a log of metadata changes). A key/value store designed for layering needs to be able to go full speed with real, cross-node, multi-key transactions rather than focus on a single node "fast path" (like, if I recall correctly, VoltDB does). It's also highly recommended to be able to do concurrency control on arbitrary ranges rather than just individual keys, both for performance and serializability reasons. Indexes, as I've already mentioned, are vastly more scalable in this type of design than most others. And you get to decide whether you want to read from them serializably or not. One way you can do push down (which in the modern world of 10GBE networking is way more important for OLAP workloads than OLTP, but let's be perfectionist) is to co-locate SQL and K/V store nodes and to let the higher layer see where data is located in the lower layer so that it can partition queries (at least approximately) to one of its own co-located nodes. You keep all the nice abstractions and the statelessness of the SQL nodes, and you can read data locally (a nice optimization would be to have a super fast shared memory path for doing this in the database driver).

There is some cost for every abstraction. But because human ingenuity is finite, successful abstractions on net greatly increase what we can do. We don't write everything in machine code, or build new custom networking hardware and protocol stacks for every piece of data we need to send, even though there might be some microoptimization opportunities if we did.

The real performance problem that any new SQL database is going to face is that its query planner isn't exactly the same as the one people are trying to switch from. So if the customer's random web app has 1000 queries, and 995 of them are faster on the new database and 5 of them are worse, guess what happens? The 1000 queries by selection bias ran acceptably on the old database, but the 5 worse ones can be arbitrarily worse, because the badness of query planning in such a rich query model isn't really bounded. So the application is now brokenly slow. So there is going to be a long, long game of whack-a-mole, and users' experience of the performance is going to tend to be negative no matter how good or bad it really is. In my view, the ideal database interface for scalable production "OLTP" applications would be as powerful as SQL, but do less magic in the query planner - it would make it more explicit in the query how challenging the execution plan is supposed to be, so that you don't have queries that explode into unscalability. Pushing a little more work onto developers in order to save the skins of the ops people when the cardinality of some table changes.

Sorry, /rant. I don't know anything about the engineering of TiDB or TiKV and can't comment on them. But I really strongly believe that a layered architecture is the right one both for databases and for lots of things that people don't think of as databases but that wind up having to solve all the same concurrency control and fault tolerance problems.