Hacker News Reader

Comment by nitinreddy88 | Hacker News Reader

I wish if there's a way we can hook DuckDB as Extension to Postgres itself. Currently there's no proper columnar database/extension exist in PG. In most of the use cases, one doesn't really require dedicated "analytics" only DB and sometimes everything has to be done through "Primary DB" (typically in enterprise products world).

By hooking DuckDB as Extension, we get best of both of worlds from PG and DuckDB while retaining the persistence and analytical capabilities without hosting yet another DB infrastructure.

View on HN

+1. I checked Citus, hydra, etc, and nothing comes close, all has its restrictions or gotchas.

I also really want to have a LSM based storage engine, but noone implemented it yet. Yugabyte built a postgres compatible database, but it has some management-related stuff not opensource... Cockroach similarly...

Check out https://github.com/paradedb/paradedb/tree/dev/pg_analytics, we're shipping this week for column-based storage in Postgres. It's not LSM, though, but Parquet/Arrow

Seems like someone has tried it:

https://medium.com/@ahuarte/loading-parquet-in-postgresql-vi...

Now we need to write parquet files from Postgres. Its essentially what do we need to do for end to end solution with PG by leveraging extensions

This can be done using duckdb_fdw. It needs one additional copy though.

You insert into the duckdb table and then export from duckdb using duckdb_execute.

https://github.com/adjust/parquet_fdw

It doesn't support writing data. One has to hookup worker to write data efficiently to parquet on regular basis or some fashion using PG. Its like bits and pieces everywhere but didnt create end to end usecase

pg_analytics looks promising here https://github.com/paradedb/paradedb/tree/dev/pg_analytics

Wouldn’t you still miss the part about storing tables column based in your database rather than row based? Would be nice to have “create table” or “create materialized” view with columnar storage option.

Hey! I'm one of the makers of pg_analytics/ParadeDB.

We store tables column-based. You can do CREATE TABLE (...) USING deltalake; when using ParadeDB and it will be stored column-based (and soon, with support for also querying cloud object stores!)

Looks interesting, but exactly what is missing when just adding duckdb to Postgres.

Why did you go with delta lake over iceberg?

DuckDB is in-memory. It makes sense as a in-memory columnar cache. We went with DataFusion to benefit from the deep integration it has into the ecosystem with Arrow, Parquet, etc.

As for Delta Lake, we would've loved to do Iceberg but the Rust crate for Iceberg is very new, and doesn't have enough functionalities yet. We're tracking the project and planning to contribute, so that we can add support for Iceberg eventually

Reply on news.ycombinator.com