NoDB: Efficient Query Execution on Raw Data Files

The whole premise of the paper is that data can be analyzed in situ. That means loading from its original place in the original format without any previous transformations. This is in contrast to the traditional approach of database systems that the data has to be loaded first into a database.

This paper describes a way how unprocessed unindexed data can be efficiently used to answer queries using a database system. The novelty of this approach is mostly that they build a index on the fly that can be reused later and examining the idea of directly using the raw files. For efficient loading of CSV files which is also mentioned in the NoDB paper, I think those details are better described in a later paper from TU München[1] that examines this aspect in more detail.

HDT for RDF or Parquet[2] and ORCFiles[3] in the Big Data space, are (binary) formats where the data is already processed and stored in a more efficient format than plain old text CSV files. Creating these files can already be compared to just loading the data into a database. The only difference that format used for data storage is open and can be used by many systems. So its a completely different setting.

Still its an interesting thought to make databases aware of the indexed information in those file formats besides CSV so that these can also be directly used without loading.

[1] http://www.vldb.org/pvldb/vol6/p1702-muehlbauer.pdf

[2] https://github.com/Parquet/parquet-format

[3] https://cwiki.apache.org/confluence/display/Hive/LanguageMan...

http://csvkit.readthedocs.org/en/0.9.1/scripts/csvsql.html

Here's an example of how to download the SF health inspection data, unzip it, build the SQLite schema (the dataset contains multiple tables), and then insert the CSVs into their respective tables, and query them, all from the command line.

csvsql handles the inference of schema from the CSV data and the import work; the SQL commands can all obviously be done from just echoing into SQLite directly. But csv2sql will also let you create a CSV in memory and run a SQL query for those times when you just need to quickly filter the data. Here's an example:

http://2015.padjo.org/tutorials/mapping/077-ok-schools-quake...

Tremendously useful tool. I can say without hyperbole that it has changed the scope of how I do data science, even though it literally does nothing that I could not do via SQlite and Python alone (csvkit is basically a command line wrapper written in Python around csv, SQLAlchemy, and so forth)