Hacker News Reader

Comment by ta93754829 | Hacker News Reader

One of the problems I think I see with event sourcing is its inability to scale. You have to guarantee the order of events, right? How do you do that in a large scale distributed system with eventual consistency, without incurring an insane synchronization time penalty? I'd genuinely love to hear if you have a good solution for this, because if you do I have a use case I need it for, so this ain't a troll comment!

View on HN

You generally only have to guarantee the ordering of _some_ events--not all of them. Ideally you want a way to partition your data such that ordering is only important per partition. For example, when I worked in derivatives trading, we only needed to guarantee ordering by product class, but events could be generally unordered given multiple product classes, as long as each individual one was ordered. Similarly, the telemetry systems I work on now guarantee ordering per sensor (duh) and per sensor grouping, but not across _all_ sensors on the vehicle. Partitioning makes for easy scaling (just make a new partition yay).

This depends on the nature of your scaling issue. Are you having trouble with consuming events? Or with adding all of your events whilst trying to ensure that they are ordered?

For the former, you might use something like [ZooKeeper](https://www.elastic.co/blog/found-zookeeper-king-of-coordina...) in your consumers to control parallelisation.

For the latter, you order by time of insertion. I think you might have some contraints you haven't mentioned that may be the real problem you are trying to solve. What is distributed?

https://stackoverflow.com/questions/41082938/event-sourcing-...

that basically says Vector Clock, which is a good solution for where it's usually applied, but I think this is different. Having everything bottlenecks through your vector clock isn't going to be feasible.

Exact order of events should be guaranteed for one DDD aggregate, it is doable with simple optimistic locking. So unless your system consists of one aggregate - you should not have scalability issues.

I guess, whatever a Kafka cluster does, that's how you do it.

Reply on news.ycombinator.com