Comment by xyzzy_plugh | Hacker News Reader

Comment by xyzzy_plugh | Hacker News Reader

Parent Full thread

Thanks for the detailed response.

This is what I was getting at, mostly:

> Any fragment which is missing the "ok" file, is ignored by the reader [3].

As a client, how do I detect this? Say I just wrote some data and now I want to generate a new total. Could my data be silently missing?

> TileDB handles corrupt or incomplete fragments by erroring out on the read. In the future, we could offer a retry mechanism for failed reads, but the important thing is we will never return corrupt or invalid results.

But now I have to handle this everywhere I interact with TileDB. Most users don't expect essentially random errors. Retries are a stop-gap, as it can take arbitrary time for consistency to converge. I've observed O(hours) regularly.

> Incomplete fragments can happen because of S3's eventual consistency; it is possible for the atomic ok file to show up before an object in the fragment "folder".

Indeed! And from what I understand, mostly from conversation with the Iceberg team last year, they solve this entirely by using a central ACID service backed by RDBMS (which, aside, I find very amusing!! They never listen to Stonebraker...).

Indeed, as far as I can tell, the only way to remove consistency issues is to: - store an index to objects somewhere central - never list the bucket (or speculatively stat, etc.)

Until either a) you support a central index or b) S3 has a better consistency model (not that DynamoDB garbage Hadoop uses) I would be very reluctant to use TileDB if it wasn't using my own disks.

I think for small stuff, it'll work great, but at scale you're going to hit some roadblocks. I recently encountered splitting a bucket into to many buckets and prefixing UUIDs at the root of the bucket to reduce consistency problems caused by S3's rebalancing, and it ended up being a very expensive marginal improvement.

Folks, apologies, but I think we got a bit side tracked here, TileDB does not suffer from the consistency issues mentioned above.

Here is how TileDB performs a new (potentially concurrent with other reads and writes) write:

- It creates a fragment folder (or "prefix" of a set of objects on S3 - there are no "folders" on S3) which is timestamped and carries a unique UUID. This fragment is self-contained and represents the entire write (e.g., all cells and all attribute values)

- It writes all data objects under the fragment prefix. Note that TileDB never updates, it always writes new immutable objects.

- After all the PUT requests succeed for the data objects, it creates an empty "ok" object.

Here is how TileDB performs a (potentially concurrent with other reads and writes) read:

- It lists the array prefix to get the ok objects

- There are two cases:

1. The ok object is not there for some fragment. That fragment is completely ignored.

2. The ok object is there. Since TileDB writes the ok object last, all the data objects it wrote have been committed and are all visible with GET requests. TileDB reads the data objects only with GET requests (not ListObject requests). Due to S3’s read-after-write consistency model (https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction...), all those objects will be available for reading (now on all S3 regions) with GET and there will be no errors.

Therefore, TileDB follows the eventual consistency model of S3 without any errors and surprises. The user doesn't need to handle anything. Our customers have been using TileDB in production for a long time, storing hundreds of TBs of data on S3, and no consistency issue has ever come up.

Summarizing, what xyzzy_plugh is raising here is that TileDB does not have ACID guarantees. And that is true (we never claimed the contrary) and intentional. We are building a transactional layer outside of the storage engine. The reason is that this transactional layer indeed needs to be a constantly running distributed service, whereas we want the TileDB storage engine to be embeddable and used without performance regression even by applications that do not need ACID (that is, the majority of our data science applications).

Reply on news.ycombinator.com