Hacker News Reader

Comment by gavinray | Hacker News Reader

What's the difference between this, and Datasette which has been in development multiple years by the Django co-creator?

Sorry if it's a dumb question but I skimmed the docs and sample and it looks similar + didn't see any mention:

https://github.com/simonw/datasette

View on HN

I looked into Datasette before starting ROAPI. From a product/use-case point of view, to me Datasette focuses more on quick and easy ad-hoc data exploration type of work. ROAPI focuses more production ready online serving of static datasets. So I would expect users to use ROAPI to power micro-services in production with high QPS.

From a technical design point of view, ROAPI authors owns the full stack end to end from query parsing, data format parsing to query execution because I am also a maintainer of Apache arrow and it's sub-project datafusion. The whole project is built with Rust end to end from scratch. Datasette is mostly a wrapper around sqlite. It translates user actions into SQL queries, then execute them on sqlite. In ROAPI, we work at a lower level. We translate REST APIs, GraphQL and SQLs into datafusion logical plans and execute them. Datafusion is also a analytical compute engine optimized for columnar data, so it will be a lot faster for OLAP workload, while sqlite is optimized for OLTP. I also plan to add other type of query capabilities like nearest neighbor vector search for ML applications, etc.

Thank you for a genuine, well-thought answer. Much appreciated.

I'm not in the Data Science space so I only know of Datasette, but maybe worth copy-pasting that on a FAQ page under "How does ROAPI compare to X?" to avoid repeating it.

yeah, that's a good idea. thanks for the suggestion :)

Similar question: how does it compare to using ClickHouse with http interface (which supports a number of output formats including Arrow and ArrowStream).

It does take a few more commands to start a ClickHouse server, create table and load data in... a friendly loader program could probably get pretty close to Roapi interface.

Both Roapi and Datafusion look very cool! Excited to learn about them.

I think these two systems explore different design spaces, the biggest difference I would say is Roapi can apply more read optimizations by exploiting the fact that it doesn't need to support frequent online updates from the client. Most of the datasets it serves will be static. For data-sources that supports streaming updates like delta tables, the update frequency will be much lower than what clickhouse supports.

This is not a concern for ClickHouse.

You can load data into MergeTree table that support streaming data ingestion or into Memory table (that is read/write as well).

Also you can run queries on the data without any preprocessing (with `file`, `url` table functions or similar).

datasette is only sqlite, ROAPI serves up CSV, Parquet storage etc.,

Datasette serves JSON, CSV, SQLite, and a bunch of others as well. Don't think it handles Parquet tho

  https://docs.datasette.io/en/stable/ecosystem.html#sqlite-utils

"Insert data into a SQLite database from JSON, CSV or TSV, automatically creating tables with the correct schema or altering existing tables to add missing columns."

There's a bunch of plugins (mix of official and user-developed) that might add other formats too, I'm not 100% sure:

https://datasette.io/plugins?sort=downloads-this-week

Reply on news.ycombinator.com