vira28

Aside from the cost, my major motivation is to keep the infrastructure simple. The data is already there in Postgres, so I didn't want to add another data warehouse. I have also shared my thoughts on where this is heading https://viggy28.dev/article/postgres-gateway-drug/

Thanks for the kind words!

Short answer: yes, column-level schema changes sync to Iceberg automatically[0].

Logical replication (pgoutput in v1) doesn't actually stream DDL statements. Instead, Postgres emits a fresh Relation message describing the table's current column layout right before the next change to that table. So we diff that against the last layout we knew and infer what changed.

From there we evolve the Iceberg schema in place: flush any buffered rows under the old schema first, then write a new metadata version with the change. What's handled today:

  - ADD COLUMN — new field ID allocated; the column's Postgres DEFAULT is carried into Iceberg's initial-default/write-default, so existing rows read back correctly
  - DROP COLUMN — removed from the current schema, existing data files untouched
  - Type widening — int4→int8, float4→float8 (the changes Iceberg considers compatible)
  - REPLICA IDENTITY changes

[0] https://github.com/viggy28/streambed/pull/21

Welcome. Would love to hear your experience. Feel free to share here or in the repo. Fully open source.

Should be fairly doable using binlog-based producer https://github.com/go-mysql-org/go-mysql.

Both projects are relevant. Curious, what kinda pushdown capabilities that you were looking for?

Hello, I checked ingestr repo, and it is in my bookmark. Small world.

Agree, CDC is like Death by a thousand cuts. I believe Debezium has a Java library.

My initial need was Postgres compatibilty. Wanted to give an endpoint that BI and dashboard teams can use to query as if they are querying a Postgres replica. Added more context here https://news.ycombinator.com/item?id=48350820

Author here. For context, I was the tech lead for the Postgres team at Cloudflare, and this came directly out of a challenge I kept hitting there: BI and dashboard teams needed to run long-running analytical queries, and the answer was always to spin up another bespoke read replica or stand up an ETL dump into an analytical database and query that.

So the question I started with was: what's the fewest components I could get away with? That led to the architecture here — Streambed connects to Postgres as a logical replication subscriber (same mechanism as a read replica) and streams WAL changes straight into Apache Iceberg on S3, queryable from psql via an embedded DuckDB. There are a lot of edge cases to handle, and it's very much early days.

Welcome any feedback.

If I have to guess on why they are providing voluntary exit option, because they have a lot more folks from Europe.

Did exactly that for the actual filing — Python, mentioned in the post. The 23 numbers were a probe, not the goal: I wanted to understand how it works.

Curious, why cursor for this? VSCode or pretty much pure open source IDE's have CC integration. Or am i missing something?

I am on phone. Sorry if it’s already discussed.

How’s it different from Parallel Web Systems?

Rappo | https://buildrappo.com | Founding Applied AI Engineer | Remote or Onsite SF

Hi, we have been working on a personal assistant for knowledge professionals. A lot of challenges around building an advanced RAG, improving reasoning, and custom models

Learn more about the role: https://wellfound.com/jobs/3382533-senior-ai-ml-engineer or email vignesh@buildrappo.com

PS: I am a solo founder, and we are generous with equity

How does this compare to Letta?

Rappo | https://buildrappo.com | Founding Fullstack Engineers | Remote (Preferably India) Hi, I am the founder of Rappo (agentic GTM Chief of Staff). We observed that technical startups spend way too much time in the early stages figuring out GTM.

Everyone says early PMF is crucial, but there is no streamlined support there.

Learn more about the role: https://wellfound.com/jobs/3133503-lead-founding-full-stack-...

PS: I am a solo founder, and we allocated generous equity for advisors and early hires.

Building Rappo (http://buildrappo.com/founders), an agentic GTM for devtools and infra startups to land their first customer.

It’s one of the area where Postgres docs are light.

I don’t remember they have a similar doc for setting up HA.

Rappo | https://buildrappo.com | Founding Fullstack Engineers | Remote (Preferably India)

Hi, I am the founder of Rappo (a network for GTM). We observed that technical startups spend way too much time in the early stages figuring out GTM.

Everyone is like PMF is crucial, but there is little support/help around in getting there.

Learn more about the role: https://wellfound.com/jobs/3133503-lead-founding-full-stack-...

PS: I am a solo founder, and we allocated generous equity for advisors and early hires.

This is amazing overview of the current challenges.

As a database practitioner and a founder of a database startup, I would be curious to see how you approach these challenges and address them. Also, how you make it economically sustainable too.

How should I think about banc wrt digits?

A platform for technical founders to accelerate their journey to PMF http://buildrappo.com/founders

BTW, thanks for your awesome work with Pigsty :pray:

This is excellent. One of the things that many folks mentioned [1] as a challenge in self-hosting databases.

https://news.ycombinator.com/item?id=27671376

I don't know about the support in core but there is this extension https://github.com/sraoss/pg_ivm

Container manager (omni_containers[0]) is a bridge (aka migration solution) for existing applications or services. There are already a lot of applications running using containers and with omni_containers one can lift and shift. Hopefully, once they migrate fully to Omnigres, they will no longer need (at least, that's the hope)

0: https://github.com/omnigres/omnigres/tree/master/extensions/...

Omnigres | Founding Engineer | SF Bay Area HQ | REMOTE

At Omnigres, our north star is to enable developers to laser-focus on business needs instead of fighting technological challenges. We're fighting the complexity and inefficiencies of contemporary stacks by removing them instead of hiding them.

At the core, we are turning Postgres into an Application Runtime. Why? Because we believe that code and data are inseparable in pretty much all of the line-of-business application systems. Turns out, when done this way, applications work a lot faster, require a lot less maintenance, and are simply easier to write.

Our foundation is open source and is available at https://github.com/omnigres/omnigres

We're backed by some great early-stage VCs and looking to onboard people who can move quickly, learn on the go, and maintain the focus on the goals. Another way to look at it is that we want to meet other pragmatic idealists.

Email: founders@omnigres.com

Interesting approach.

We have been seeing this trend (or pendulum swing) of pushing SQL and simplicity. I am not saying this solution is simple (will leave that for discussion).

Tangential, if this kinda stuff is interesting, folks might be interested in what Omnigres (disclaimer: my employer) is doing with the python integration in Postgres

https://docs.omnigres.org/omni_python/intro/

would love to chat. Sent an email. Mine is vignesh@omnigres.com

Emailed you. Would love to chat. vignesh@omnigres.com

Omnigres | Founding Engineer | SF Bay Area HQ | REMOTE At Omnigres, our north star is to enable developers to laser-focus on business needs instead of fighting technological challenges.

We're fighting the complexity and inefficiencies of contemporary stacks by removing them instead of hiding them.

At the core, we are turning Postgres into an Application Runtime. Why? Because we believe that code and data are inseparable in pretty much all of the line-of-business application systems. Turns out, when done this way, applications work a lot faster, require a lot less maintenance and are simply easier to write.

Our foundation is open source and is available at https://github.com/omnigres/omnigres

We're backed by some great early-stage VCs and looking to onboard people who can move quickly, learn on the go and maintain the focus on the goals. Another way to look at it: we want to meet other pragmatic idealists.

Email: founders@omnigres.com

Haven't played with the product itself, but I have known Yev for some time now. Sure he and the team will do something meaningful in this space. All the best.