Hacker News Reader

Comment by malisper | Hacker News Reader

Yep! The new version of pgrust supports batch based execution and a columnar format. I'm curious how you got δx to perform that well? From what I've seen a columnar layout only gets you part of the way and really good parallelism and really fast hash tables seem to make up a significant portion of why Clickhouse is faster.

View on HN

Yeah, spent a lot of time on parallelism, vectorizing, pipelining, filter push-downs, bloom filters, all the tricks out there. It's really fun to make pretty steady progress on this.

pg_mooncake (now effectively abandoned due to being acquired by Databricks, but still up at https://github.com/Mooncake-Labs/pg_mooncake) pulled the DuckDB engine into Postgres wholesale, if I remember right.

pg_lake also uses DuckDB but keeps it external, routing through Postgres and managing Iceberg tables (but not the data itself) there (https://github.com/Snowflake-Labs/pg_lake).

Both of these were neck and neck with ClickHouse last time I tried them.

Actually δx is faster than the "duckdb embedded in postgres" options: https://benchmark.clickhouse.com/#system=+_etx|_b|_i)|dula|pnc&type=-&machine=-6t|ca2|6ax|g4e|6ale|3al&cluster_size=-&opensource=-&hardware=+c&tuned=+n&metric=combined&queries=-

Plus all the normal Postgres features work as expected: physical/logical replication, crash recovery, pg_dump/pg_restore, etc.

That isn't what the data shows, but we don't need to discuss it further. My reply was to the person interested in learning from other Postgres OLAP designs. This stuff is all pretty immature though and I wouldn't actually build on it outside of some very narrow, well-understood workloads.

Are you using Datafusion or parts of it for query engine

Reply on news.ycombinator.com