Hacker News Reader

Comment by zX41ZdbW | Hacker News Reader

DuckDB is obviously faster than SQLite on analytical queries, from 10 to 11 000 times on a benchmark [1]. But in comparison to SQLite, DuckDB cannot run all the queries due to worse memory management. It cannot run 12 out of 43 queries.

[1]: https://benchmark.clickhouse.com/#eyJzeXN0ZW0iOnsiQXRoZW5hIC...

View on HN

I’m a big fan of DuckDB. However it is true that memory usage is not optimized right now — I’ve had trouble with very large groupbys even with pragmas on memory limits (but I’m talking huge limits like 250 GB RAM). I’ve had to remodel and break down my problem to work around these limitations.

I believe it’s on the DuckDB roadmap to get better at memory use but apart from that it’s very very good at what it does — ie complex analytical queries. It vectorizes ops so everything is fast and efficient.

I'm surprised duckdb failed to run simple count/groupby queries. Is it simply OOM? If so, do you know how much more it would take to complete those queries?

My wild guess is lack of some functions, e.g., regexp, but need to take a deep look.

It's likely out of memory issues with many many distinct values in an aggregated column, or with a very large join intermediate. The joins are now able to handle larger than RAM, and larger than RAM intermediates in a group by is on the roadmap!

Looking at the failed queries, yes, OOM might be one big factor due to aggregations (there is no join in these queries though).

Reply on news.ycombinator.com