Comment by ipsum2 | Hacker News Reader

Comment by ipsum2 | Hacker News Reader

I don't know if the creators of TileDB visit HN, but if they do - it would be nice to see a code example that showcases what makes TileDB unique. Right now, I don't see any benefit versus serializing a data frame in hdf5 and storing it in Redis or cloud blob storage.

(Reading https://tiledb.com/developer, the Github, or the blog - none of it shows me anything amazing)

Edit: browsing around the site, I finally found out what tileDB offers, and I'm still confused (too much fluff language) To save people time:

https://docs.tiledb.com/main/handling-dataframes

Multi-column slicing: By defining any subset of the columns as dimensions and due to TileDB's tiling flexibility, you can increase the pruning effectiveness of multi-column slicing, thus leading to better overall read performance. Essentially, a TileDB array acts as a primary multi-dimensional index on the columns selected as the array dimensions.

Data updates and versioning: TileDB offers rapid, parallel, cloud-optimized updates. All the update logic is pushed into the storage engine and is completely transparent to the user. TileDB also exposes useful time traveling functionality, such as reading arrays at time snapshots, effectively implementing data versioning built into a single embeddable library.

Partitioning: TileDB enables balanced partitioning, without limiting each partition to single column values. Moreover, we will soon expose API functions for dynamically selecting different partitioning schemes (e.g., on different subsets of columns with different orders), without the need for reorganizing/rewriting the array.

Sorting: All sorting is taken care of by TileDB internally with multi-threading.

Stavros from TileDB here. TL;DR, it's all about fast slicing on multiple columns while supporting updates, locally or in the cloud.

Suppose you serialize your dataframe in HDF5 or Redis. Let your dataframe have schema (Date, Stock, Price). Assume this dataframe is 1 TB long and stored on S3, GCS or Azure (as they are cheap). How would you be able to efficiently perform an average query on Price for a specific Date range and Stock symbol? With HDF5 you would have to download 1 TB (no notion of "fast slicing on variable predicates") and apply the predicates locally. If you stored the dataframe in Parquet (a better choice for this use case), then you would be able to build some logic in your code that uses the Parquet metadata/indexes and prune a lot of unnecessary information (as Spark does). However, Parquet is "one-dimensional", i.e., your pruning would be efficient on Date, but not on Stock (you'd have to "partition" your Parquet files with Spark or Hive and things could get quite complicated). Most importantly, you wouldn't be able to update the Parquet files; you would have to generate new files and build a catalog on top (or use services like Delta Lake) to manage your Parquet files. And this is an extremely cumbersome task.

TileDB abstracts everything for you, while allowing you to slice fast on any number of columns. You just define Date and Stock as "dimensions", and slicing on both those columns becomes uber efficient locally or in the cloud. Effectively, you turn this dataframe into a sparse 2D array. Updates and time traveling are handled by TileDB. You get to use Spark, Dask, MariaDB and PrestoDB as you did before, but there is no need for Hive, Delta Lake or any other cataloging service. Thank you for pointing out the confusion though. We just launched and we have tons of examples coming up.

Stavros thanks for the explanation, how does TileDB avoid downloading the entire matrix and do the slicing (locally)? Are we achieving this by breaking down a big matrix to a set of smaller ones? so that you only down the subset of that the current query need? If this is the case, what is the current measure we have to avoid mismatch on metadata (e.g. some error while uploading them to S3) that links them together? thanks

Efficient slicing happens because of "tiling", hence the name TileDB. A tile is similar to an HDF5 or Zarr "chunk", or more loosely to a Parquet page. Although totally configurable, tiling is handled solely by TileDB, the user doesn't need to know about it. A tile is the atomic unit of IO and compression. TileDB maintains all the necessary metadata and indexing built into its format and, given a query, it knows how to fetch only the tiles that might include results. The tiles are decompressed in your memory and filtered further for the actual results. The dense array case is rather straightforward. The sparse case is a big differentiator in TileDB and it is quite challenging, especially in the presence of updates. TileDB handles the sparse case via bulk-loaded R-trees for multi-dimensional indexing, and via an LSM-tree-like approach with immutable objects that allows time traveling.

Concerning your point on potential errors occurring on S3, this is addressed by TileDB's immutable object approach. If an error occurs upon some write, there will be no array corruption. Happy to discuss about this topic on a separate thread.

Some related docs:

https://docs.tiledb.com/main/performance-tips/choosing-tilin...

https://docs.tiledb.com/main/basic-concepts/tile-filters#til...

https://docs.tiledb.com/main/basic-concepts/definitions/frag...

I heard about TileDB because one of the authors commented at https://news.ycombinator.com/item?id=22423632 . That discussion includes what you mentioned, plus a mention that it integrates with MariaDB, PrestoDB and Spark, including the ability to use an embedded MariaDB.

Thanks for the post!

Here is a quick demo of dataframe i/o: reading to/from a pandas dataframe, and then creating and slicing a sparse array with heterogeneous dimensions (int and string):

https://gist.github.com/ihnorton/1544bb169b4ec8cd385e9c3e756...

This demo doesn't run properly. Eg in the 2nd code block I got: "AttributeError: module 'tiledb' has no attribute 'from_csv'". I received this message in the latest TileDB Docker image.

There seems to be a problem running the example under python 3.6 used in the docker container. Investigating. Please ping us (tiledb.com for contacts) and I will let you know when the docker image is updated.

Fixed in the latest image update (tiledb/tiledb:2.0.1). Thanks for the report!

Reply on news.ycombinator.com