Hacker News Reader

gpaulbagettion [dead]

Wrote this after watching a client's query costs balloon from table layout decisions made early and never revisited. Mostly about partitioning/clustering tradeoffs people skip when moving fast.

gpaulbagettion What drives BigQuery costs, and what doesn't

Follow-up to a piece we did on data org for cost, this one's narrower: which BigQuery cost levers actually move the needle vs. which ones are folklor

gpaulbagettion MySQL CDC to BigQuery: what periodic syncs miss, and how binlog avoids it

hey thank you for your feedback! The description was giving the wrong idea indeed. I've just updated the article :)

gpaulbagettion MySQL CDC to BigQuery: what periodic syncs miss, and how binlog avoids it

Hey! Glad to hear about Bruin, competition always pushes us to make our product better for our customers :)

Frankly, I am not a big fan of streaming either. But CDC and streaming are two different things. Streaming database replication is generally built on CDC, but you can use CDC with micro-batches too (which is what we do). I wrote about this in another article: https://www.erathos.com/en/blog/cursor-based-sync-vs-change-...

Imo, the key reason for offering CDC is that most databases aren't designed around "how can we extract data efficiently", therefore they often lack a reliable "updated_at" column for us to use as a cursor.

gpaulbagettion MySQL CDC to BigQuery: what periodic syncs miss, and how binlog avoids it

Wrote this after seeing the same failure mode too many times: a MySQL sync job that looks fine in the dashboard but has been silently dropping deletes and intermediate updates for months, because it's comparing snapshots instead of reading the binlog. Tried to lay out exactly what has to be true on the MySQL side (row image, binlog_row_value_options, server-id, retention) for CDC to actually be complete, not just "eventually consistent." Happy to go deeper on any of it in the comments.