Hacker News Reader

Comment by rileymichael | Hacker News Reader

Perfect timing — I was just looking into similar tools.

If I want to do a bulk copy (say, nightly) with various transformations but not continually stream after, is that supported / a use case that’d be a good fit for your tool?

View on HN

It looks with what is currently implemented you'd have to drop the tables from the destination if you want repeated copies, probably not quite what you want. Close enough to your use case that it still might be worth testing it out though.

yeah that’s not an immediate deal breaker for me, I’m essentially looking for pgdump/restore + transformations. I’ll give it a look and see how it performs

Thank you for giving this a spin. That's correct, today you'd need to drop the table before another sync with pg_flo. That said, I have given delta syncs some thought, and also looking into a control plane that can make some of these things easier. Would love to hear your feedback.

I think this also might be of interest to you - just one time copies and apply transformations - https://github.com/shayonj/pg_flo/issues/6. I will look into shipping that very soon.

Oh awesome, yeah that’d work perfect then. I’d just prep beforehand and run the one-time copy.

amazing! Will plan a release by Tuesday latest. You can experiment with `--copy-and-stream ` in the meantime [1]

[1] https://github.com/shayonj/pg_flo?tab=readme-ov-file#streami...

Heya! v0.0.4 is now out with `--copy` support, which will stream all the data from the table and the transformation/routing will get applied before the changes land in destination.

[1] https://github.com/shayonj/pg_flo?tab=readme-ov-file#streami...

[2] https://github.com/shayonj/pg_flo/releases

Awesome turn around — I’ll give it a go this week!

Reply on news.ycombinator.com