Transactions in a Microservice World
wso2.com
wso2.com
"Can entirely encapsulate a transaction" is a good rule of thumb for designing your microservice boundaries.
Once that transaction has completed within the microservice, you can have other services then reliably carry out follow-on actions (maybe via an at-least-once delivery queue such as Kafka).
If the thing can encapsulate an entire transaction, then it's taking care of an entire business vertical. What I do agree, it's one of the best ways to partition your services (just after not doing it). But calling this a "microservice" only serves to confuse people.
It's like finishing a migration by renaming your initial project with the new name of the target project: everyone is happy, the software is still working, the various political stakeholders now feel the warm embrace of ownership from the old bad domain to their turf, and you saved years of reinventing the wheel to end up in the same place.
Almost everywhere you go, you see the same mistakes. Almost every one can be found in Sam Newman's 2nd edition. The conclusion I draw is that a very large number of developers ran with it without understanding the pros and cons. Services are only worth it if your organization benefits more from the pros. Many orgs don't, so it makes no sense for them to adopt it over monolithic architecture that is way easier to implement transactions with.
Designing "transactions" across services and databases is an insane endeavour. It's best if each service is split on a line that it can deal with incomplete states in other services.
I think that the word micro is indeed the issue. Once they stop being micro and just become services, it pretty much becomes a Service Oriented Architecture. So why keep calling it microservices?
Before that, we had "SOA", that was coined to mean exactly breaking your system into full verticals. But then the term was hijacked by some very loud companies selling tools for microservices, and now if you go search for it you will mostly find stuff like how to set distributed transactions.
Overall, humanity currently has a pretty hard time communicating, due to the sheer number and size of bad actors actively and purposefully polluting our communication infrastructure. We get to feel the effects when talking about software too.
You should never have a microservice that doesn't own something unique and the best ones are interesting enough to have a proper owner.
They proceed to create 5 services in python aws lambdas: ui proxy, credit card processor, insurance gateway, email service.
The flow would be: the user clicks buys, it sends the the ui proxy, which triggers a redirect to a credit card ui, the client pays, the backend sends a request to the insurance gateway, and if all goes well, an email is sent with the policy.
Ofc, we didnt receive a policy document from the insurance gateway, we'd generate it ourselves on success, the insurance gateway only saying "ok". The first few weeks, the team missed some failure condition on the insurance gateway, and would charge the client and send a policy pdf by email, but the client was NOT actually insured. It impacted 7 clients a day going to trips thinking they were insured but were NOT.
Imagine the face of the business when they realized there was no transactional boundary over the whole process: one micro service could fail and the others could succeed and god knows what happens then. The change was to build one giant procedural thing that does 1 step at a time and finishes once all is good. One service, one transaction, one state that matters. The architect was eventually fired, finally.
If you go monolith, you’ll of course have the advantage of having one place that orchestrates everything, the place where you start the transaction and follow up with credit card / insurance / email.
But if you are forced to deal with micro services you can roll a poor man’s transaction in the form of a state machine for the whole process backed in the db. Ui proxy receives click, credit card service authorizes money, insurance generates, credit card service settles transaction, email is sent.
The good thing about this whole affair is that you can very clearly see where something fails and why, because failure is not a special case, but part of the normal operation of the service. Also gets you the whole split up into micro services thing where different teams can handle different services. Bad thing of course is its massively more complex, so most probably is not justified.
Transactions are great, but when they rollback, they rollback everything. Now if you want to actually understand and track what went wrong, you need to log stuff somewhere else, which in itself can also fail.
Your database and the credit card company's are separate, so they may retain a record even if you lost the data.
A multiple business step flow should be called something different, to avoid confusion with well established database terminology.
Every business I've ever worked for has figured out they have peanut butter down, and they have chocolate pretty well squared away, and wouldn't it be great if we made peanut butter cups? We have all the bits we just need to mash them together.
Business loves, loves, loves creating new features by combining existing ones. It's easier than actually inventing anything new. So whatever transactional boundaries you created two years ago? Those are creaking under the weight of feature factory work coming in from customers, business folks and the C-suite.
We had full transaction isolation and it just ran thousands of connections and transactions per second, gloriously, without any issues. We had a micro currency (kinks), first 1080p pay per minute live viewing, and a whole slew of happy customers with .gov/.mil email addresses.
Kids these days with their 'microservices'... psssschhhaaa...
I wish everyone many microservices and cutting edge frontend frameworks!
While Microservices themselves can/should absolutely use transactions internally, they should be coarse grained actions. A call into a microservice should either fail and complete, but never leave a system in a partial state.
As a trivial example, consider any interaction with a third party. Once you're into that territory, you're talking logs, asynchronous retry queues, compensating operations, etc.
Sometimes the surface area of your company's product gets so large that, in order to scale, you interact with other teams as if they were 3rd parties.
Easier would be to admit that the microservices idea was the biggest fad in the history of computing and milked by consultancies to no end while compounding the complexity of software development to unprecedented levels, needlessly.
People just lose their mind when you say the microservice.
1. We got into this microservices fad and built this really granular service architecture which we're so proud of but is actually a pile of steaming shit and we're in denial.
2. We forgot to consider that a service boundary is a transaction and/or failure domain and now we have this distributed operation that has to be transactional because it's crapping itself 50 times a day and clients are complaining that something is inconsistent somewhere.
3. Read Martin Fowler's blog and this blog article while having a post-work shit in the bathroom.
4. Come out with a half baked understanding of distributed transactions which is good enough for your ego to think is a really good business outcome.
5. Design another fucking microservice and correlation system for transaction registration.
6. Hand it over to some junior and senior developers who half-ass the whole thing because they don't understand your inane ramblings and have no idea how to test it properly.
7. Throw it into production and shout SUCCESS as loud as possible over the sounds of creaking, grinding and explosions.
How this should have turned out...
1. Actually realise that you're only doing about 5 million transactions a day and that's hardly anything.
2. Stuff the whole shebang behind a monolith with a single call, single transaction and in one storage and failure domain.
3. Pay for a slightly fatter DB instance.
4. Go back to developing things with a better ROI.
Apologies for the bitterness but it is realistic.
So as one might expect - various automated reports and actions routinely fail with deadlocks and timeouts, any change anywhere has a pretty big chance of screwing some other distant part of another system, all services need full unrestricted access to all databases, and off course nothing can really be tested because you need to run practically every database and service on the same machine to even try to test.
Naturally the pace of work is glacial, and bugs are blamed on individual intellectual failing of lesser tenured developers not having the experience and the vast system trivia knowledge of “lifers” who secured their guaranteed employment this way.
You might even be those people for all i know.
We went through a few growing pains when we had some big reports, but the first pass just moved them to a ~100 line batch reporting system, that solved it for another 2 years. Then we hit growing pains again but by that time had a shiny new data warehouse and just moved them to use the data warehouse instead.
There's truth to both sides of this, but optimising for where most engineering is happening, keeping developers productive, and simple solutions that everyone understands, have always paid off well in my experience.
But maybe posting a snarky meme about it isn’t constructive.
At scale microservices do help, but you get into unnecessary problems when you don't have scale and use microservices.
As for lifers, everyone I work with knows how many days until retirement. Sad sad people. I am merely hopping between frying pans until I can slide out into academia and get old and rot somewhere hopefully with tenure.
step 5: partition your database and setup sharding and use logical replication to replicate columnar storage on redshift for long term analystics.
whatever is past step 6, you probably wont' get there
Except someone wrote their own and it's buggy.
One of these days I’m going to write a follow up blog post called “You (probably) don’t need sagas.” About how at my current company I was able to remove almost all of the sagas we were using because it turns out they were never needed in the first place.
I also publish updates about related blog posts.
The "Classic Funds Transfer" diagram at the top is the signal that you don't need to read this article and the author probably shouldn't have written it.
Your Coffee Shop Doesn't Use Two-Phase Commit:
https://www.enterpriseintegrationpatterns.com/docs/IEEE_Soft...
This version was published the same year the term 'microservice' was coined, but the author blogged about it the previous fall.
Actually I’m annoyed that a course in distributed computing is an elective at good schools and may not even exist at others. I like concurrency, but I’ve been surprised over and over again to find nobody else can or will step up and it’s because I’m the only one who was classically trained. It’s just ridiculous at this point.
If you're dealing with money, you should at a minimum have some kind of ledger, and derive the current state from that.
Calling an SQL transaction "Classic funds transfer" is a bit of a disservice to all the banking that happened before databases. Mesopotamians were recording transactions on clay tablets 5000 years ago.
Using incorrect and invalid models of real world systems and using them as exemplars of technical concepts, even as a short hand, is a sign of a not well written document.
It gets worse, because the "level up" to distributed transactions builds on the model by trying to show two banks engaged in a distributed transaction using 2PC to handle a funds transfer. This doesn't exist. It's not a thing. It never has been. There's nothing "classical" about this example.
It exists in the same canon as "a manager is a person. an employee is a person..." examples to explain data modeling.
Using these "examples" leads readers (learners) to walk away with incorrect understanding that they will likely spend a ton of time having to unlearn and overcome.
Good luck with your Chesterton's Fence adventure boys. You're gonna need it.