Hacker News Reader

I just want a recognized standard format for write ahead logs. Start with replicating data between OLTP and OLAP databases with minimal glue code, and start moving other systems to a similar structure, like Kafka, then new things we haven’t thought of yet.

View on HN

The structure for the write head logs needs to different between systems. For Postgres, the WAL is a record of writes with new blocks. It can't be used without knowing the Postgres disk format. I don't think it can be used to construct logical changes.

Using a standard format, converting things into logical data, would be significantly slower. It is important that WAL be fast because it is the bottleneck in transactions. It would make more sense to have a separate change streaming service.

> I don't think it can be used to construct logical changes.

It can: https://www.postgresql.org/docs/current/logicaldecoding.html

It's not entirely from the WAL though, some catalog accesses are necessary for metadata (shape and name of tables etc).

Reply on news.ycombinator.com