There are 2 big benefits to working with Parquet files in DuckDB, and both relate to speed!
DuckDB can query parquet right where it sits, so there is no need to insert it into the db first. This is typically much faster. Also, DuckDB's engine is columnar (SQLite is row based), so it can do faster analytical queries using that format. I have seen 20-100x speed improvements over SQLite in analytical workloads.
Happy to answer any questions!
Considering DuckDB's and Parquet's features, if DuckDB manages to support the emerging GeoParquet standard someday, I believe it could be a nice alternative to the typical SQLite + Spatialite combo, and GeoPackage as a whole.
DuckDB excels at single machine compute where everything fits in memory or is streamable (data can be local or on S3) — it’s lightweight and vectorized. I use it in Jupyter notebooks and in Python code.
But it may not be the right tool if you need distributed compute over a very large dataset.
I’m really interested in what the limits of DuckDB and Parquet are.
Can you give me an idea of what size you mean by “a very large dataset”
DuckDB itself doesn’t have any baked in limits. If it fits in memory, single machine compute is usually faster than distributed compute — and DuckDB is faster than Pandas, and definitely faster than local Spark.