Ask HN: WebSocket data architecture strategies: idempodent vs. incremental data? | Hacker News Reader

Ask HN: WebSocket data architecture strategies: idempodent vs. incremental data?

3 points2upmedia8 comments

How have you guys architected your data over websockets? Do you use certain strategies to reduce latency with less data over the wire and/or strategies to ensure local memory doesn't exceed a certain amount?

View on HN

I think you could find some inspiration in cryptocurrency exchanges. Most of them expose public websockets for prices, order books, trades, etc. They're high volume with lots of subscribers.

For example, check out the docs for the Binance websockets (they use both snapshot and delta messages): https://github.com/binance/binance-spot-api-docs/blob/master...

If you could tell me a little more about the data format, data volumes, number of subscribers, and how you get the data on the backend, I can try to give you some more concrete advice.

If I understand the question: you don’t have to choose between idempotence and efficient incremental updates if you can give your application data a revision number. Send your incremental update(delta) with a reference to the app data’s last revision. Increment the revision number after applying each delta and reject any delta that doesn’t reference the current version on the server, they’ll only be applied at most once, so they’re idempotent.

There are fancier versions of this scheme that can be devised based on the business rules for your application.

Happen to know of any places where I could find high level data architecture concepts like that? Or did you know that just by experience?

I’m searching now and I see multiple books on designing distributed systems. They’re probably worth looking at, but I haven’t read any of them so I can’t make a specific recommendation.

I’ve learned a lot from reading about the architecture of existing distributed systems e.g CouchDb, Git, and Bitcoin.

Awesome. Thanks.

Wow. Never heard of that one. Very useful thanks.

How are you currently doing this?

did you try going for a 3rd party provider like Firebase, Ably or Pubnub?

Planning stages, so no 3rd parties yet. Considering strategies because I need live connections with as low latency as possible so I'm probably going to be managing the websocket connections myself on the server side, unless there's something else that will give me the same benefits and nearly no latency.

Reply on news.ycombinator.com