> To enhance query execution speed, we embedded DuckDB as the execution engine for columnstore queries
So is it Postgres or DuckDB that cracked the analytics top ?
So is it Postgres or DuckDB that cracked the analytics top ?
Fast analytic databases need two key things: columnar storage and a vectorized execution engine. We introduce a columnstore table access method in Postgres with data stored in Parquet) and execute queries on those tables using DuckDB.
By leveraging DuckDB's execution engine, we avoid reinventing vectorized query execution while keeping everything managed through Postgres.
More on our architecture: https://www.mooncake.dev/blog/how-we-built-pgmooncake
We blogged about this: https://www.mooncake.dev/blog/duckdb-parquet
That doesn’t seem like something that makes sense to in DuckDB.
> pg_mooncake leverages Postgres to store table metadata, eliminating the need to query an external catalog (as DuckDB would require when scanning external Iceberg or Delta Lake tables). Instead, it directly queries the Postgres table to identify which Parquet files to scan, reducing overhead.
> Additionally, pg_mooncake stores detailed Parquet metadata for each file in Postgres, including column statistics for every row group. These statistics, such as MIN/MAX values for each column, are aggregated to optimize scans. When querying a columnstore table, pg_mooncake uses these statistics to filter out data files and row groups that cannot produce matches.
That was always the culprit for me compared to AWS/Athena and BigQuery. They are dirt cheap on analytics workloads when you can parallelize the calculations to 100 CPUs without really paying any extra.
With postgres you are stuck with linear cost for scaling up number of CPUs, so everything is slow anyway.
Good point! Normally for postgres extension it won't be solvable, but for mooncake it is actually not the case!
The core idea of mooncake is to built upon open columnar format + substitutable vectorized engine, while natively integrate with Postgres.
So right now it is using duckdb within postgres to run the query, but we can and we will support ad-hoc using other 'stateless engines' like Athena, StarRocks or even spark to run a big query.
> Fast analytic databases need two key things: columnar storage and a vectorized execution engine. We introduce a columnstore table access method in Postgres with data stored in Parquet) and execute queries on those tables using DuckDB.
It might not even be possible to install x on some hosted pg servers.
It is great that you can expand pg and that you and others have but I don't rate it as high as a buil-in solution.