Cell-based architecture for resilient payment systems
americanexpress.io
americanexpress.io
Just kidding!
I find the idea quite good, and have to assume that the amount of payment fails they experience due to partitions/outages isn't very high and that the post-payment reconciliation and reclamation process gives them the liberty to rank availability a bit higher than correctness.
One thing that looked a bit shaky was the interplay between the global transaction router's state of knowing which cells can handle a particular payment and the asynchronous distribution of the "failover data", which I presume it needs to know to route correctly. To me that seems to create a window where it might route to the wrong cell due to an outdated routing state.
It also doesn't go into the HA setup of the global transaction router itself.
But still, I kind of like the design.
But if the router sends to the wrong cell the cell will either send it back to be rerouted or it will fail and the router will try again (or report back the failure so upstream can try again I assume)
But what if the cell doesn't know that, and it's holding, for example, a stale account number?
When talking to a local merchant about why they dont take amex the fee was not mentioned. The merchant said that visa and mastercard settle at the end of the day and they get their money but amex it was sometimes 4 or 5 days before they actually got their payment.
This isn't about payment technologies, it's not about isolating transactions, it's about scaling the middle layer. What's worse it's not even explained what middle layer does.
No info on how routing works, no info on data synchronization.
Folks just learning Kubernetes and write extremely abstract stuff.
Granted. It works really well in practice. It should be noted we haven't actually had the world war the Internet was designed to survive. So we're not entirely clear on the semantics of operations in unusual and unexpected configurations. I would expect DNS to be the first shoe to drop there.
The router needs to be shard-aware. It needs to know what data is where based on the request coming in so that it can route accurately. A GLB is DNS. It cannot be shard-aware because all it knows is the FQDN being resolved.
It can be a "router" if all the router needs to know is to resolve to the nearest data center or the nearest CDN. But at that point I have to ask the question - why does one need a cell-based architecture and can't it just be geo-redundant active-active failover across regions.
In any sense, the architecture itself isn't novel or new. It's documented here: https://docs.aws.amazon.com/wellarchitected/latest/reducing-.... It's the go to model if you're running a cloud.
One can have GLBs that do routing. So long as the tenant-to-cell routing tables are consistent, it works fine. And those mappings tend not to change frequently.
that said: still a nice write up, learning about some of the architectural choices that AMEX makes is definitely insightful (and relavent/useful to what i am working on right now as well!)
As long as event orderings are unimportant, or self resolvable this works really well.
e.g. if Events A,B arrive, but A+B => C and B+A => C, then as long as you durably record A, B, the end state is the same.
I'm not sure why "reroute" is a message instead of a response, I would expect it to be a failure response pruning the control flow. With the GTR being "default retry".
There's a lot of learned experience in that doc. Reading between the lines, both the logging and configuration systems have caused global outages (or near misses). Nifty to read.
This style of architecture fits with a "no global changes" and "never lose it all" approach to fault tolerance, accepting that there will be visits from "Mr. Cock-up" [1].
Very nice writeup.
Because of the title I was expecting to read about doing payments with a distributed network, like a terrorist cell network, or something like Hawala. Not (as I infer from other comments) Amex using multiple independent systems.
This is why I find it best to declare a card stolen right before expiration or after.
We use a cellular architecture to help constrain the blast radius of a modular monolith. Each one of our customers lives in exactly 1 cell. Any kind of cross-customer BI/reporting happens through a data warehouse.
https://news.ycombinator.com/item?id=32023863
https://wso2.com/engineering-platform/developer-platform/doc...
Some CICS regions, a DB2 and a couple of VSAMs and that's it.
The system I work on has such a property and the only real infra style approach is sync replication before responding to a caller and a delayed replica for delete/drop protections (say with a 2hr or more window).
Should also defend for this in your code (be able to reply from your initiation systems also etc)
Last week it required me to take pictures of my face from multiple angles to regain membership privileges. I suspect this may be part Palantir data collection and part Peter Thiel dating service.
They email you an alert. You click "this is okay." Card still doesn't work, you call. Rep asks ID questions, still not enough. Says he has to text a QR code linking to some wonky app that videos your face, no other option.
You can't even login to pay the card off and cancel without going through the process.
This reads to me like an attempt to patch a system that's already fucked beyond belief while pretending you're doing "engineering".
Fancy implementation of a retry loop attempting to minimize downtime.
I worked as an SRE at a well-known monitoring company that used a similar architecture. It worked extremely well, and aside from one software SPOF (which still had a blast radius limited to that cell), we had very few large-scale production incidents compared to everywhere else I’ve worked at.
Even if there was a physical hardware failure (at the time, it ran on-prem, but it’s not like AWS is immune to this), every service modulo the aforementioned SPOF had redundancy, so we would have the datacenter techs replace that blade, which would provision itself and rejoin, zero downtime, just a temporary loss of redundancy. Even then, if we felt it necessary, we could shift customers into a different cell, though that did cause a brief outage for them, which would be coordinated ahead of time.
Any sufficiently complicated concurrent program in another language contains an ad hoc informally-specified bug-ridden slow implementation of half of Erlang.