Why local state is a fundamental primitive in stream processing | Hacker News Reader

Why local state is a fundamental primitive in stream processing

radar.oreilly.com

I wish to solve the problem of sync data from mobile to a master database, and is about invoices, orders and that kind of data. Here, is necessary to preserve the order of the data actions. The data must return back to each client, and the clients could be offline from time to time. In some cases for a couple of days. I have total control on the solution, so I could move to any direction, but my only hard requirement is that the data in the client is on sqlite.

I struggle to build a reliable solution, but I understand that a log based could be a good idea. Wonder what way could lead me closer to the solution?

So you need to process your events linearly, but you don't need to store them linearly.

Here's a way you could achieve this:

- keep sqlite

- use Pouchdb on top of it to sync your data to your server, typically Couchdb

- Since you know the order of events, tag each of them with an application-level sequence number

- On your server, when you receive an event, check all events that happened before, in order. For each event, if it has not been processed, process it

But I need to duplicate the data to pouch, then move it to couch.

My question is how do the sync myself. How build the log so it be usefull to this task? To be clear, how is the algorithm for this? Any hint or open source example?

Shameless plug: at WebAction we have an enterprise-grade (exactly once processing, scalable, built-in streaming data integration) SQL-based streaming platform. It's clear real-time analytics is headed in this direction and we have some interesting problems to solve.

If anyone is interested in chatting about it, feel free to email me john@webaction.com

Is anyone using Samza outside of LinkedIn? I've been interested in it for a while, and I'd love to hear your experiences.

Reply on news.ycombinator.com