Hacker News Reader

Comment by saisrirampur | Hacker News Reader

But why? I’m skeptical of the idea of unifying storage just because it sounds “elegant” or “cool”. It’s not obvious to me how a single storage engine can compete with purpose-built OLTP and OLAP systems like Postgres and ClickHouse, without significant tradeoffs.

You also mention removing CDC pipelines. I’m curious if the materialization (conversion across formats) can catchup to an OLTP workload that is heavy (50K+ tps), which is pretty common these days. Also CDC if done right and with care can be magical for users and stays native to the OLTP/OLAP data-store.

Third, data Lakes and open formats are suitable for Data Warehousing / Data analyst use-cases than real-time customer facing apps. Sure, you might work on changing that, which is what you are upto, but you’ll always run into tradeoffs, which will make it hard to unleash the best performance, much needed for the latter category.

View on HN

Conversion is async. The whole point is to never deal with CDC which is error prone and taxing Postgres with occupying a replication slot and burning memory and cpu in the OLTP system.

Taxing Postgres is one thing, which can be overcome with ways like using standbys. There could other more native ways (than unifying storage), which you’ll hear about in a few weeks. Also I don’t fully agree on logical replication taxing Postgres, if the client is built with care and precision.

In regards to error prone and speed (lag, latency at real-world scale), I wish the blog went into more detail and gave evidence than talk theory.

> CDC which is error prone

Just have superior CDC :)

Exactly! Why unifying storage, which opens up a can of trade-offs.

If you product is CDC based (peerdb) you don’t want storage to support this :)

This architecture is better for OLTP because all maintenance operations are moved to storage AND it has all other benefits such as LTAP that emerge from having a scalable storage.

;) sounding good on paper vs how it works in practice (supporting demanding real-time OLTP/OLAP workloads) are completely different ball games.

Separately, I understand taking care of it at storage level, but still don’t get “unifying storage” or “zero copy”.

Anyways, I’ll stop now. Good to see all the innovation happening on converging OLTP/OLAP front. Each with a different approach and perspective. :)

this is all so funny, gl to everyone

Did you even read the blog? Or are you just throwing shade because you are working on a competitive product based on CDC?

Oh no, I did read the blog. Not throwing shade at anyone here—the blog is great. It just doesn’t provide real-world evidence, and it opens up a bunch of technical questions that I’m trying to understand. that’s exactly what HN is for. :)

zhou here :)

Moonlink was arguably one of the best ZeroETL(mirroring) solution, and there's a reason we built LTAP instead (not just because it's cool and we can do it)

To me, two big issues with CDC/mirroring are: 1. For ad-hoc querying, CDC is usually not there when you need them, and it is too expensive to maintain for all tables if you don't query them often. 2. For true data pipeline, a simple mirror is not enough, you will end up building all kinds of transformation and it essentially becomes ETL like spark.

One small issue is CDC + merge into columnstore means huge write amplification if you want fresh read on Analytics side, it is absurd especially for lakehouse tables (I have seen 100X for many OLTP workloads).

The only solution I found that we can trust agents creating, managing, running OLTP & OLAP queries is LTAP.

CDC _never_ goes away, that's for sure. Maybe just for replication to a logical replica, but you'll often need a sync from one system to another -- it just ends up happening.

Reply on news.ycombinator.com