Hacker News Reader

Comment by mastax | Hacker News Reader

I really like the DuckDB philosophy, and the maintainers were really responsive and helpful with my docs questions. Didn't end up using it because the compressed database files were much larger than SQLite for my data (weird use case, admittedly), and the .NET libraries were poor. I look forward to trying it out again, though.

View on HN

> the compressed database files were much larger than SQLite for my data

Just curious, what format did you use?

If it's Parquet, it does column compression via Snappy by default (which more optimized for speed than size).

You can change the compression to ZSTD or GZIP for higher compression ratios.

https://duckdb.org/docs/data/parquet

I tried all of the available compression formats in parquet, natively in duckdb, and compressing raw duckdb files as a whole. IIRC just GZIPing SQLite files beat them all. The SQLite files were much smaller uncompressed (~220mb vs 800mb) which was probably most of the difference.

Can you still query a gzipped sqlite file? What’s the sense in gzipping it if not?

Do other formats (Snappy?) allow this?

You can query snappy parquet files directly.

You can also query gzip and zstd parquet files directly.

> ...the .NET libraries were poor.

DuckDB's Node bindings leave a lot to be desired, as well. WASM bindings have a different API surface, too. From my reading of the code and docs, it looks like duckdb 3p bindings for anything other than python is in dire need fit and polish.

We're still on a journey to explore what APIs work best with JavaScript but the differences between WASM and Node are on purpose. DuckDB-Wasm has isolated wasm heap memory and runs in a separate web worker. That's why we serialize everything as Arrow IPC buffer and pass the ArrayBuffer through the workers message API as transferrable. On Node.js, we can interact with DuckDB much more easily and don't want to pay the price for the IPC stream every time. The truth is, we tried quite a few different APIs and this turned out the be a good trade off between efficiency and convenience.

But I'm happy to discuss this further if you have any suggestions.

Reply on news.ycombinator.com