So we ended up with great puzzles like how to link an order service with an inventory service to check if there's enough inventory to fulfill an order, all through a central event bus. Then of course the order service has to update the inventory if the order is confirmed. But of course, the order service also has to talk to the pricing service, and since the price of products varies by day, they'd have to remember the price at the time the order was made. Whose responsibility is that? No idea, that's someone else's service.
If you want microservices to work you shouldn't have anything central in them, and you should avoid async as much as possible.
One-way async dataflows work much better. That way each microservice is up with all the (potentially delayed) data it needs to respond to requests.
Typically one-way or two-way is not a choice you can make as an engineer, most processes in your business require two-way, because they are initiated by a user and user wants a definitive response.
The "potentially delayed" approach is very tempting for engineers, but it should be exception rather than the rule.
It
- dilutes responsibilities between services (who is responsible for delays? is service A producing too many messages or is service B too slow to process them?)
- makes your SLA vague (message was processed 3 days later, do we treat it as downtime or not?)
- requires more infrastructure & processes (every service has a queue, dead-letter queue, and a process to deal with dead letters)
- requires a ton monitoring overhead (what delay is acceptable? how do we even measure delay? what if you have different SLA for different messages? we'll have a monitor per message type?)
- introduces a lot of unnecessary complexity and rules (how do you deal with TOCTOU, e.g. admin deactivates a user, but by the time the message gets processed he's no longer an admin)
- ruins user experience (we received your payment information, but we won't immediately tell you that it's wrong).
Despite it's downsides, potentially delayed approach can be a fine tradeoff when it saves you 7-8 digits per year. Most companies never reach this phase.
By abstracting content the only thing that needed to change for new types of content was the content service and clients. Abstracting recipients which can be users/groups, meant that the only service that needed to care about this detail was the one that replicates sent to inboxes in the receiving service. Because of the use of content ids and user/group ids, this is all small idempotent/immutable metadata. The system was complex (yet became manageable over time) and onboarding onto each service was immediate.
I think few have seen well-bounded microservices' contexts leading to the idea that it's a bad distributed monolith. Also worth remembering that advantages of microservices 'done right' is large scaling of developers and isolate failures.
I've advised engineers to rewrite code before, mostly for resume building to help jumping ship to somewhere better. Basically, use the current job as a way to train on the stack the company you really want to work for is using.
If a company doesn't have a stock-based comp and sticks to prevailing wages/CoL there's no incentive to help the company succeed. Especially with non-technical management, the best thing to do for ICs is often to aggressively rewrite (in the language that's trendy right now and that better companies use) and get promoted by showing off a significant output. Then jump to a better company having spent a year training on their stack at the previous company's expenses.
It also creates an invisible bias towards worse companies in your next position, because smart companies are wary about this kind of behavior, and will be savvy to filter out candidates that have displayed it.
Great work happens at the intersection of employee's and company's interests!
Why not simply align the two? Compare the Xerox Alto with the Macintosh...
so naturally it doesn't matter (in long term) what you should learn to get a first job, change jobs, change domains - prioritize ROI the tool would give you in future, a job market, and your sanity :)