Why Payments Engineers Should Avoid State Machines

The usual analogy used here are double-entry ledgers. I’m not sure what state machines, Marco Polo, and google maps have to do with it.

Although I suppose you could look at your aggregates as state machines if you’re using them to validate commands before adding new events to the log.

It says to not hide the events that drive the state machinery: make both the events and state machines available (to both servers and clients), so everyone can reconstruct history when necessary, using the agreed upon events and state machinery.

Which makes sense.

As it turns out, this is what Holochain does to build reliable distributed systems at scale out of unreliable components (payment systems, or anything else).

Every event stored / cached by every App DHT participant is verified by ensuring it’s valid according to the verified shared “DNA” of the App (any state machines and other arbitrary validation code). Misbehaving clients / servers self-incriminate (all events are signed and non-repudiable), and are ejected from the App by all correctly operating nodes.

By induction, a node can then reason that every prior event in the App DHT is valid, regardless of system scale — because an attacker would have to subvert all nodes to avoid having just one valid node notice and report their error to everyone.

But it also has a very valid point about events. Or rather accounting records.

Because ultimately it's a payment system, and what you're doing is updating the accounting records across at least three parties: the user, the payment service, and your own records. You do so by sending messages, and it's a good idea to keep track of the content of those messages. At which point it starts to look a lot more like an event system.

In some cases recording may be a legal requirement. French NF525, for example, mandates securely recording every receipt.

https://www2.ikosoft.com/en-gb/all-knowledge-about-the-nf525...

I've found bitemporal databases a good way of thinking about most crud-style applications, including payment systems. Basically, you keep 2 timestamps: When something supposedly happened, and when you learned about it. This means you leave a logbook of every action around. By replaying the logbook ordered on happened timestamp, you arrive at the correct current state. You can only append to the logbook, never modify an existing field. You use a state machine to gather the state from the logbook, and probably cache it in another table

So e.g. on day 1 you do a payment. happened=learned=day 1, paid. On day 2 you find out this was an error: happened=day1, learned=day2, undo payment.

I don't follow. Of course a state machine cannot reconstruct the past. But you can keep a little auditing framework/logic alongside the state machine to keep track of the event history.

So I don't really follow how you can leap from "A state machine cannot reconstruct the past" to "must avoid state machines." There is no logical connection between the premise and the conclusion!

https://news.ycombinator.com/item?id=41639763

Also, state machines are very good at storing state transitions, AKA events. So I'm not sure what the author is going on about saying that you can't reconstruct how you got into a particular state. People do it all the time.

As far as scalability is concerned, you can't push it out to the client because the client has no legal authority. Payments, which is part of the banking system, relies on centralized services and processes having legal authority and authorization to make withdraws and deposits.

Since someone having 10 years of experience in payments should know this stuff, I'm thinking maybe the author poorly conveyed their thoughts on the matter.

Does keeping a log of state changes, an addition to the current state, transform a state machine into an event-driven system?

> You’re getting scalability by forcing the client to accept more responsibility.

How? Believing that a client is never buggy or malicious is just asking for problems. Alternatively, if "responsibility" here means simply that a client should be able to replay a history of events received from the server on its own, then this problem reduces to the question I asked paragraph above.

But that's just nonsense. There is absolutely nothing intrinsic preventing a state machine knowing how it arrived at a state, in fact if you use an AWS state machine with lambda step functions you can look up any past execution of a state machine and see all the steps it took to arrive where it did.

Furthermore, there's nothing intrinsic about event pipelines that makes them know all the actions taken up arrive somewhere. Not only that, but it's extremely common problem in large event architectures that teams and actions are so distributed you don't actually know who all your consumers are.

The problem and the solution discussed is maintaining history, which has nothing to do with state machines nor event-driven systems, which in turn are not mutually exclusive.

I like to use nouns and verbs to illustrate a difference here. I think people initially approach state machines with noun nodes and verb edges. In practice, verbs are where most of the time is ("waiting for", "attempting to"), so you can also have a state machine where the nodes and edges are inverted. Some systems work better one way or the other.

I would use a state machine, and log the transitions (“events”, if you insist).