Ask HN: Have you ever set up two-way data syncs? What did you use?

6 points·vinarun··3 comments

Hey HN, asking for help from my fellow data engineering community! Between what two systems have you ever needed a two-way data sync?

For example, I’ve seen some devs wanting it to connect databases to tools like Airtable. But are there other tools / use cases that you’ve seen? I’ve heard of Mulesoft and Piesync but those seem overly expensive for these use cases

View on HN

The main advice with two way data syncing is to ensure that you've mapped out the data flows and make sure that it's clear which system needs to be updated to update data. I once worked on a PHP monolith that worked with a vendor .net system where because the data flows were incorrect, would routinely overwrite data changes, especially for fixed/updated changes. We eventually did a large project which fixed the data flows and cut the number of complaints to 20-30% of the original volume.

How you set up these syncs really depends on the ingest/egress options for the systems in question. Are you pulling data directly from a db? Do you need apis? Etc

Theory of Composing Protocols : https://arxiv.org/ftp/arxiv/papers/2203/2203.02461.pdf

search engine topic "two-way data syncing" brings up quite a few listings (with additional search subject area suggestions such as software & best practices)

lot of differences between 'real time' and 'off line' data syncing. aka banking business hours changes vs. nightly book keeping.

Reply on news.ycombinator.com