Hacker News Reader

Comment by tudorg | Hacker News Reader

> - Is ~300x faster than Postgres on analytical workloads. Right now it's 2x slower than Clickhouse on clickbench and I think it's possible to get faster than Clickhouse

That sounds like you are storing the data in a columnar format? Or do you do both row and columnar?

In a somewhat similar (yet also quite different) effort, I've been working on δx, a Postgres extension that compresses the data in a columnar format stored in normal Postgres tables (so replication, crash recovery, pg_dump, etc. still work normally). https://github.com/xataio/deltax

It is currently about 30-40% slower than ClickHouse (single node, ofc). The PR to add it to clickbench was just accepted, so you can see the comparison here: https://benchmark.clickhouse.com/#system=+liH|_etx|gQ|saB&ty...

View on HN

Yep! The new version of pgrust supports batch based execution and a columnar format. I'm curious how you got δx to perform that well? From what I've seen a columnar layout only gets you part of the way and really good parallelism and really fast hash tables seem to make up a significant portion of why Clickhouse is faster.

Yeah, spent a lot of time on parallelism, vectorizing, pipelining, filter push-downs, bloom filters, all the tricks out there. It's really fun to make pretty steady progress on this.

pg_mooncake (now effectively abandoned due to being acquired by Databricks, but still up at https://github.com/Mooncake-Labs/pg_mooncake) pulled the DuckDB engine into Postgres wholesale, if I remember right.

pg_lake also uses DuckDB but keeps it external, routing through Postgres and managing Iceberg tables (but not the data itself) there (https://github.com/Snowflake-Labs/pg_lake).

Both of these were neck and neck with ClickHouse last time I tried them.

Actually δx is faster than the "duckdb embedded in postgres" options: https://benchmark.clickhouse.com/#system=+_etx|_b|_i)|dula|pnc&type=-&machine=-6t|ca2|6ax|g4e|6ale|3al&cluster_size=-&opensource=-&hardware=+c&tuned=+n&metric=combined&queries=-

Plus all the normal Postgres features work as expected: physical/logical replication, crash recovery, pg_dump/pg_restore, etc.

That isn't what the data shows, but we don't need to discuss it further. My reply was to the person interested in learning from other Postgres OLAP designs. This stuff is all pretty immature though and I wouldn't actually build on it outside of some very narrow, well-understood workloads.

Are you using Datafusion or parts of it for query engine

Reply on news.ycombinator.com