Architecture: https://evidence.dev/blog/why-we-built-usql/
1. Query SQL databases, APIs, or local data (eg CSV)
2. Compile all the data sources into Parquet files
3. DuckDB-WASM in the browser that allows you to aggregate across sources
4. Users write code in DuckDB SQL and Markdown, enriched with viz components (built in Svelte)
Some things that we have learned in the process:
- It can be pretty performant up to about 20M rows of data in the parquet files,
- Above a certain level, for speed, it's helpful to sort your data in your parquet files to take advantage of DuckDB's predicate pushdown
- It's helpful to map DB types into a smaller set of Arrow types when you convert to Parquet - otherwise you have to consider a lot of different cases in the browser when you render in JS
- DuckDB-WASM still has some rough edges, though is improving fast