Comment by kernel_concern | Hacker News Reader

Comment by kernel_concern | Hacker News Reader

Parent Full thread

I work on the syncing team at Hightouch. These are great questions and also good feedback on how we could be clearer when describing the problems we need to solve.

1. We also support the case you describe, in which an ML model processes data and then updates properties in a destination. However, customers still get a lot of value out keeping downstream systems synced with their warehouse tables. For instance, you can define which people you want to receive different campaigns and make sure that's consistent across all your ad platforms. You can also use it for simple projects like easily keeping Airtable in sync with a Postgres database.

2. Some people have warehouse tables with many billions of rows.

3. If you have a billion rows, you need to hit a very high rows per second number in order to run a sync in a feasible amount of time. Also, we have an event collection product, which allows customers to feed events into Hightouch in realtime, and a personalization API product, which allows customers to hit an API and get a low-latency response for how a given user's experience should be personalized. Making sure that the new data flowing into the events API is processed, and data is ready in the personalization API for fast fetching, needs to be fast.

4. It's true that syncing often implies some bi-directionality. In this case we think about "syncing" the destination system state to that present in the source system. It's nice because you can use the source system as the source of truth and trust that any edits you make will be reflected elsewhere. Possibly across many destinations.

Is this helpful?

1. > However, customers still get a lot of value out keeping downstream systems synced with their warehouse tables. For instance, you can define which people you want to receive different campaigns and make sure that's consistent across all your ad platforms.

Ah thank you that's a very helpful example for understanding your product.

2. > Some people have warehouse tables with many billions of rows.

I don't doubt customers have multi billion row tables in their CDWs but I guess I'm not seeing why you would need to send billions of rows to FB (or any other downstream system that isn't an OLAP database) rather than some much smaller payload distilled from that data via an ML model or SQL query. I admittedly have never run a FB ad campaign but Meta has 3.35 billion daily active users [1] across all of their products. If they are sending billions of rows to FB, do your customers have individualized ad campaigns for every single FB user? Perhaps I'm just not familiar enough with the state of modern digital advertising.

3. > If you have a billion rows, you need to hit a very high rows per second number in order to run a sync in a feasible amount of time.

I wonder why do you have to send billions of rows to FB every time? Surely you send it once for initial setup and then incrementally sync smaller deltas? And presumably your customer is OK with the initial setup being slower. Unless your customer is doing billions of writes to their CDW in between syncs?

Thanks for taking the time to explain!

[1] https://s21.q4cdn.com/399680738/files/doc_financials/2024/q4... page 10

You're right that typically the day to day delta we need to sync is much smaller. However, customers often want to change something for a large fraction of their dataset, requiring a large update. Also we support a our Personalization API product, for which customers do frequently want to refresh what personalizations they're showing to all of their users. For this we do need to be regularly syncing all the data.

Reply on news.ycombinator.com