Hacker News Reader

The use case for DuckDB is the that the vast majority of analytics aren't on "big data", but also aren't "small-ish". If your data fits on a single machine, DuckDB will allow you to query it at great speed. I disagree that the speed "doesn't significantly matter".

There is work being done on a dataframe API frontend for DuckDB, if you prefer that interface.

> I do not fully get the speed argument

The way I see the speed improvements is this: fast processes run faster, moderate-to-long processes (that are still awkward and large in Python, but not enough to justify the shift to Spark) now run in noticeably faster time, and the threshold for “what I need to use Spark for, shifts a long way up the scale. That is, you can now do more, with the same amount of compute.

Some more thought in the same direction: https://dataengineeringcentral.substack.com/p/whats-all-the-...