Stavros from TileDB, Inc. here: TileDB could be useful for storing your training data (in some storage backend of your choice) as well as your intermediate data (as davedx pointed out). But this would make sense only if your data are truly large and cannot fit in main memory. In fact, we are looking forward to seamlessly integrating with systems like TensorFlow, but we would rather wait until we can bring some value to applications with very large storage requirements.
Stavros from TileDB, Inc. here: Currently, TileDB is a storage manager, so it offers only efficient slicing. Support for aggregation queries (e.g., similar to what you can do with Pandas) is in our roadmap. We plan to work on it as soon as we ship the Python bindings.
In-core algorithms require the entire array(s) in main memory to perform some computation. Out-of-core algorithms are typically block-based and stream the array blocks from persistent storage to main memory on demand, working on parts of the array(s) at a time, thus minimizing the memory requirements. If this is done asynchronously and carefully, for some CPU-bound algorithms you may be able to completely hide the storage-to-memory cost, thus saving memory without losing performance.
The way the code is currently architected allows us to add support for pretty much any compressor (compatible with our MIT license) with minimal effort. So, please do send us suggestions about your preferred compressor and we will add it pretty quickly. Thanks!
Stavros from TileDB, Inc. here: We have elevated the Python/NumPy bindings as our top priority and have already started development. We will try to ship them asap. :)
Stavros from TileDB, Inc. here: This comment gives us the opportunity to explain a bit the history of the company and our thesis. TileDB was my research project while I was working as a full-time researcher at Intel Labs and was stationed at MIT. This was part of a collaboration between Intel Labs and MIT, called Intel Science and Technology Center (ISTC) for Big Data. Intel funded MIT (and several other universities) for 5 years, while mandating that any tech produced in the center should be open-source with a non-restrictive license (like our MIT license), and without any strings attached. So ISTC was for pure research. We created TileDB, Inc. so that we can continue working on TileDB beyond the graceful termination of the ISTC, since we believed that the tech (and future vision) can significantly contribute to the scientific community (there was already some early evidence through our work with the Broad Institute on genomics). The company will continue to respect and contribute to the open-source community, while trying to bring together very talented people to transition the tech from a research project to a production-ready software.
Stavros form TileDB, Inc. here: Another area of great interest to us is point cloud data, which are essentially 3D points in a super sparse space. In fact, any application dealing with spatio-temporal data (which are 2D/3D/4D data that tend to be sparse and skewed) can take advantage of TileDB.
Stavros from TileDB, Inc. here: Arrow employs a columnar format to store objects like data frames. TileDB is also columnar (so you can sub-select attributes and perform analytics with very similar optimizations to Arrow), but the first-class citizens are (the more general) dense and sparse multi-dimensional arrays. Moreover, TileDB focuses on optimizing for the persistent storage backend, so that it can handle out-of-core analytics on massive datasets that cannot fit in main memory (e.g., genomics), or offer the same performance using less RAM (leading to cost savings in the cloud). Nevertheless, we are quite fond of Arrow, so we hope to work together at some point and integrate as seamlessly as possible.