Comment by criticaltinker | Hacker News Reader

Comment by criticaltinker | Hacker News Reader

Parent Full thread

> Reversal is the most straightforward when the event is cast in the form of a difference. An example of this would be "add $10 to Martin's account" as opposed to "set Martin's account to $110". In the former case I can reverse by just subtracting $10, but in the latter case I don't have enough information to recreate the past value of the account.

I've used the event sourcing pattern with great success in the context of an exceptionally complex web application for my day job. The front-end would create an 'event' for every action the user took. This allowed us to seamlessly implement client-side undo/redo functionality with surprisingly little effort, a market leading feature our users really appreciated.

One key insight we had was the necessity of capturing the state change as a difference, as Martin explains. This turned out to be tremendously powerful. We were able to build a variety of other features on top including detailed analytics of how users were using our app, and session replay for training and debugging, among other things.

The article also says:

> ... all the capabilities of reversing events can be done instead by reverting to a past snapshot and replaying the event stream. As a result reversal is never absolutely needed for functionality. However it may make a big difference to efficiency...

When I implemented undo/redo, I found that replaying from snapshots was fast enough. I wasn't working on a database-backed app though.

Implementing reversal is extra work and bug surface, so it should be subject to cost/benefit analysis.

I find fuzz testing works great for things like this. Make a function which, given everything you know about a user's state, randomly generates an action that user could take (and the expected result). Then you can run your user model forwards and backwards through randomly chosen actions. If you support undo, check that if you play any change forward then backward, the resulting state is unchanged.

I usually pair that with a "check" function which just verifies that all the invariants I expected hold true. Eg, check the user has a non-negative balance.

It sounds complex, but you get massive bang for buck from code like this. 100 lines of fuzzing code can happily find a sea of obscure bugs.

And if your system isn't using event sourcing (where this sort of testing is relatively simple to add), you can also do model-based testing.

I like this example as a demonstration from FsCheck using F# (for the test program) and C# (for the program under test): https://fscheck.github.io/FsCheck//StatefulTesting.html

If you have any examples of this I'd love to see them. Are you just doing a random pick of X possible actions N times?

Sure - here's a simple fuzzer from a rope (fancy string) library I wrote a few years ago back when I was still learning rust. (Don't judge me!):

https://github.com/josephg/skiplistrs/blob/master/tests/test...

In a loop it simply randomly decides whether to insert, delete or replace some text, and after any action checks that the state is valid (In this case via a call to check2()).

You can go way deeper with this sort of thing if you want, with more complex models, random item generation and simplifiers to help pare down problems to simple test cases. And you can have more complex state - for example, where you also track interaction between multiple items. For example, if your state is a few users, and you also track one user paying another user and verify the total balance across all users is unchanged.

But you don't need to go deep for fuzz testing to be worthwhile. Even something as simple as this little loop is a remarkably effective bug finder.

We're using jqwik's stateful testing to achieve this, https://jqwik.net/docs/current/user-guide.html#stateful-test...

It's essentially as you described.

I had the impression idempotency was the way to go in distributed systems and not differences.

Those two concepts are not mutually exclusive. Each change can be idempotent by marking some identifying information about the state it was trying to update.

    increase balance by 10 (it is currently 100)
    balance is now 110
    increase balance by 10 (it is currently 100) <- duplicate event!
    event has already been applied
    balance is now 110

It can definitely break down at some point (maybe the user really did want to increase by 10 two times, they just clicked twice really fast?). But idempotency and change-based flow can live together.

This is where CQRS comes in. Every unique action has its own command id, so if a user clicks submit twice, the same command id is being sent twice and after the first success all future ones don't affect anymore change, they just return automatic success since that command id succeeded in the past.

Combine this with versioning, where you assert an event stream version on write, and now you're able to affect change on a known state that cannot have changed (and if it did change, your command would fail and cause no persisted changes server-side).

I'm well aware that there's no absolute standards for this, but I've built a number of CQRS systems, and I didn't ever have idempotency IDs as a core tenant of the system.

What literature, or framework or ecosystem holds this to be a hard-and-fast requirement of CQRS?

Personally I've solved this problem similar to Git, modelling CQRS on top of Merkle trees so your system at least knows if your incoming edit (command|commit) is being applied on HEAD (no changes to the model since the command started) or to HEAD+n (needing to re-run some command validations, and "rebase" the incoming command).

I don't think there is any "standard" that strictly governs the implementation of a CQRS system. We just need to be aware of the nuances of the fault-tolerance aspect in distributed systems:

- idempotency is ONLY a hard-requirement if the events are transmitted via lossy channels such as network. ID is one out of several ways to ensure idempotency. If you're implementing CQRS for your distributed system then it's needed - otherwise, if the events aren't lossy and can be processed atomically, you don't need to ensure idempotency. think implementing CQRS for your user interface where events are just transmitted between different objects in memory rather than via network

That's generating an idempotency ID on the fly, isn't it?

That means commands have to be stored as well (at least for some time) in a non-eventual-consistent way?

If a command is successfully converted/persisted into an event, that event should contain the command's ID. When the command is replayed, the domain aggregate/event stream will contain that command ID, and will therefore reject/ignore the replayed command.

So just to get it right: 1. it means that there is only one instance that processes commands and 2. this instance checks all command ids if they appeared before (so keeps track of all ids in an efficient way)?

> it means that there is only one instance that processes commands

Not necessarily. Depends on how you shard the aggregate/streams. Since aggregates _by definition_ maintain their invariants, you can have literally a shard for every aggregate. Overkill, but it illustrates the point. Since there's no state shared between aggregates, you can process each aggregate/stream on its own instance.

> this instance checks all command ids if they appeared before (so keeps track of all ids in an efficient way)

Yep. If you trust the client and don't mind losing old commands, this isn't too hard - use a monotonically increasing number (e.g. ms since the epoch) as part of the commandId. Any command with an id less than the commandId in the aggregate will be rejected as "out of date". Note that this serves as an optimistic concurrency check.

If you don't trust the client (or you don't want to lose old commands issued on a possibly out of date view) and just use GUIDs as commandIds, then yes you'll need to keep track of some ids - but not necessarily all. Do you really need commandIds past 1000? (Depends on your domain.) You'll need to "roll up" the events into some summary view anyway to check/maintain business invariants, so keeping, say, the last 100 commandIds as part of that summary view is simple.

Thank you, that makes sense and a lot more clear to me.

Now I only wish there would be a good alternatives for when you want to have multiple instances per aggregate (even if you may have to give up certain guarantees).

Interesting... just to be clear, an _individual stream of user events_ is an aggregate. E.g. a user Bob Smith's events are independent of another user like Jane Doe. It isn't the case that the entire "User Table" is a single aggregate - each row in it is an aggregate.

If you're trying to do multimaster db storage, like a phone app syncing with some cloud server, where an aggregate resides in two separate locations, this is where event sourcing shines. It's just like git, right, and so you can do all the stuff that git can do - merge event streams, rebase events... it's actually the reason why my pet project switched from an RDBMS to event sourcing. Related video: https://skillsmatter.com/skillscasts/1980-cqrs-not-just-for-...

You can run multiple instances, and use something like distributed locks to avoid contention causing failures.

Exactly. When you are affecting a change on a stream, it has to read the events anyways, so it will be able to tell if the command id is already associated with the stream since each event has the command id in its metadata.

> It can definitely break down at some point (maybe the user really did want to increase by 10 two times, they just clicked twice really fast?)

Well, that is exactly what OP meant - in the context of a distributed system it is not "at some point" it is just a given.

What if this is concurrent with "decrease balance by 10"? If the decrease occurs between the two increases, then the increases won't be detected as duplicates.

It’s my understanding that these systems don’t use the value, they’ll instead reference some kind of “state id” that refers to the state of the system as the result is some previous, accepted, change. This way, even if a decrease races the duplicate increase, the duplicate is still correctly detected.

The two are not necessarily mutually exclusive.

    Balance is 100
    Operation #1 says add 10
    Balance is 110
    Operation #1 says add 10
    Balance is still 110

If you can create unique IDs for your operations, idempotence is really easy to implement.

from 10000ft away that has the feeling of op-based CRDTs

Event sourcing can be represented as a chronological list of changes, replayed so that any list index represents a state.

How CRDTs work is different in different implementations. Some CRDTs use a timestamp to order all changes, thus making it essentially identical to an event sourced list.

Timestamp based ordering may not actually be the most ideal, especially for low trust environments where the timestamp can be spoofed.

Some apps might not last-write wins. In this case, timestamp largely becomes meaningless because there has to be some other way to resolve conflicts due to multiple clients making changes to the same piece of data. Some apps want one clear winner, other apps might want the two inputs to be merged, others (like git) explicitly ask the developer to resolve the conflict.

In this way CRDTs can diverge significantly from an ordered list of changes.

More or less.

Reply on news.ycombinator.com