Hacker News Reader

1egg0myegg0on Open Source SQL Parsers

Since DuckDB is so portable, it can be a good choice for adding a SQL parser to a new system. Have a look here for an example that uses the parser separately from the execution engine: https://github.com/duckdb/duckdb/blob/master/examples/standa...

1egg0myegg0on Directly running DuckDB queries on data stored in SQLite files

It is just scanning the data (in parallel), not copying it into DuckDB!

1egg0myegg0on Grist – Open core alternative to Airtable and Google Sheets

Does this support live updates as other users make edits (without refreshing the page)?

1egg0myegg0on Polars: Fast DataFrame library for Rust and Python

If you wouldn't mind, please update DuckDB as well!

1egg0myegg0on Polars: Fast DataFrame library for Rust and Python

Polars would be 10-100x faster, but so would DuckDB!

1egg0myegg0on DTable a new distributed table implementation in Julia using Dagger.jl

Have you thought about interfacing with DuckDB for out of core processing?

https://github.com/kimmolinna/DuckDB.jl

1egg0myegg0on DuckDB quacks Arrow: A zero-copy data integration between Arrow and DuckDB

I should also add that there is a duckdb fdw, so you could have DuckDB read from your parquet files and do faster transformations before you pull your data into Postgres!

https://github.com/alitrack/duckdb_fdw

1egg0myegg0on DuckDB quacks Arrow: A zero-copy data integration between Arrow and DuckDB

I am not very familiar with the Postgres Parquet FDW, but here is an educated guess!

Postgres is a row store engine rather than a column store, so I believe there will need to be quite a lot of translation for Postgres to be able to process parquet data (DuckDB and parquet are both columnar). My hypothesis is that DuckDB would be significantly faster! However, feel free to benchmark things!

1egg0myegg0on Fast Moving Holistic Aggregates

DuckDB has a few options for custom functions! I'm not sure exactly how optimized they are relative to what you have in mind, but I believe there is still quite a lot of query optimization that you get out of the box.

The first option is to create a macro: https://duckdb.org/docs/sql/statements/create_macro

The second is Python-specific, but you can have UDF's straight from Python! It's on the roadmap to scale this to the other supported languages. https://github.com/duckdb/duckdb/pull/1569

In general, I also think that SQL is still abstracting a lot of the complexity you would have to handle in an imperative language. These holistic moving averages are actually a great example of why I love SQL! I just learn the WINDOW functions syntax once and it works for many queries.

1egg0myegg0on DuckDB-Wasm: Efficient analytical SQL in the browser

Yes! DuckDB (not WASM DuckDB) has a JDBC connector that works with DBeaver.

1egg0myegg0on DuckDB-Wasm: Efficient analytical SQL in the browser

If you head over to the shell demo, you can run a query like the one below!

https://shell.duckdb.org/

select * from 'https://shell.duckdb.org/data/tpch/0_01/parquet/orders.parqu...' limit 10;

1egg0myegg0on DuckDB-Wasm: Efficient analytical SQL in the browser

NIIIICE! Data twitter was pretty excited about that cool SQLite trick - now you can turn it up a notch!

1egg0myegg0on DuckDB-Wasm: Efficient analytical SQL in the browser

Check out this post for some comparisons with Pandas.

https://duckdb.org/2021/05/14/sql-on-pandas.html

DuckDB is often faster than Pandas, and it can handle larger than memory data. Plus, if you already know SQL, you don't have to become a Pandas expert to be productive in Python data munging. Pandas is still good, but now you can mix and match with SQL!