Hacker News Reader

Comment by hiyer | Hacker News Reader

There are benchmarks here - https://github.com/Eventual-Inc/Daft?tab=readme-ov-file#benc.... Seems to outperform Dask by a fair bit.

View on HN

These benchmarks seem strange, it looks like some sort of database test suite? And compared to the performance that is listed on the benchmarks own page[1], the Daft results look abysmal? Like Daft takes several hours to complete ten queries and the results the benchmark authors point to can do around a million queries per hour... oh, and the software they use for that is MS-SQL.

[1]: https://www.tpc.org/tpch/results/tpch_advanced_sort5.asp?PRI...

Hello! Daft developer here. The benchmarks we performed aren’t directly comparable to the benchmarks on TPC-H’s own page because of differences in hardware, storage etc.

For hardware, we were using AWS i3.2xlarge machines in a distributed cluster. And on the storage side we are reading Parquet files over the network from AWS S3. This is most representative of how users run query engines like Daft.

The TPC-H benchmarks are usually performed on databases which have pre-ingested the data into a single-node server-grade machine that’s running the database.

Note that Daft isn’t really a “database”, because we don’t have proprietary storage. Part of the appeal of using query engines like Daft and Spark is to able to read data “at rest” (as Parquet, CSV, JSON etc). However this will definitely be slower than a database which has pre-ingested the data into indexed storage and proprietary formats!

Hope that helps explain the discrepancies!

I’m sure you mean slow = high latency, but I do hope you’re «high throughput», like spark/…?

The QphH metric reported for TPC-H[1] is _not_ queries-per-hour, it's a composite metric:

> The performance metric reported by TPC-H is called the TPC-H Composite Query-per-Hour Performance Metric (QphH@Size), and reflects multiple aspects of the capability of the system to process queries.

so it's not directly comparable to seconds/query reported by Daft.

[1]: https://www.tpc.org/tpch/default5.asp

Reply on news.ycombinator.com