Hacker News Reader

How is this different/the same as the DAT project (https://datproject.org/ and https://github.com/datproject)?

Dat is a distributed transport layer for raw data. Quilt is a centralized (your infrastructure or ours) transport and consumption layer for virtualized data. As such we'll be able to, for example, run efficient queries across all of Quilt, allow users to import data the same way (no data prep scripts) across a variety of platforms, etc.

I recently encountered https://data.world/datanerd/inc-5000-2016-the-full-list and was impressed by the combo of a pandas-friendly client library & centralized online tier with bells & whistles. Feels closer to that direction.

There's obviously friction in this space -- this weekend I'm playing with databricks spark notebooks and public data on S3 and the data prep will still be annoying -- so I'm looking forward to design innovation in it.

I'd like to chat more about this. Quilt uses S3 and I think we could make the process of getting data into Databricks much simpler. Drop me a line if you'd like to discuss aneesh at quiltdata dot io.

Reply on news.ycombinator.com