Domain-Oriented Microservice Architecture (2020)
uber.com
uber.com
Serious question: why though? I can't think of a backend system where a single regression could bring the whole thing down. If something blows it up, it rarely takes the whole application down. What's this scenario exactly? Maybe I'm only thinking of certain types of applications, but a single failure somewhere needn't bring the whole thing down.
> Poor separation of concerns. It was difficult to maintain good separations of concerns with a huge code base. In an exponential growth environment, expediency sometimes led to poor boundaries between logic and components.
I kind of get this...The idea is that if the other area of code simply isn't accessible to you, you're forced to keep the concerns separate? Seems like this could be dealt with by applying more discipline and just making sure to design things to be more modular and.. separate. This is the hallmark of any good code base. Do we need an entirely different architecture, with all the pitfalls that microservices entail, because we can't seem to keep things clean and separate?
I really am asking these questions in good faith. I won't hide the fact that I'm skeptical of the microservices fashion (and I know that Uber is at least trying to be more thoughtful here), but I am totally open to being convinced otherwise. I'm at this very moment gearing up to begin a new large project and am wondering what design makes the most sense and whether there's a place for separate services so I'm curious to hear what people think. The one area I'm thinking about specifically is the fact that difference parts of a monolith might have different scaling requirements and if you have a single application then everything has to scale out or in as a unit. But it seems like this doesn't require a separate code base necessarily...just the ability to run different parts of a single code base as its own applications or unit of deployment, no?
This old post from Michael DeHaan comes to mind as I'm thinking about this https://archive.ph/YMRYx
Serious question: why though? I can't think of a backend system where a single regression could bring the whole thing down. If something blows it up, it rarely takes the whole application down. What's this scenario exactly? Maybe I'm only thinking of certain types of applications, but a single failure somewhere needn't bring the whole thing down.
A crash in a stateless distributed tier (e.g. web servers) will not take the whole system down.
But a crash while processing any request in a stateful non-distributed multithreaded system, can bring it down. It's much harder to make stateful systems resilient and fault tolerant. Usually it's solved by moving the state somewhere else (e.g. databases, message queues, or Kafka-style AOLs).
Exception: monolithic stateful systems using BEAM/OTP/Erlang/Elixir. Here the blast radius is per Erlang process, so in the worst case only a single request/user/session should be affected. There are some classes of errors (e.g. out-of-memory, overloaded process message queues, or crash in native code library) which can bring down an entire Erlang node. This can be mitigated by running a cluster of several Erlang nodes (which is supported out-of-the box by BEAM VM and OTP).
150 nodes is a lot. I love the beam.
I personally only run max 10 nodes Erlang cluster (on physical servers) in production, and ~ 20 on AWS for dev/testing workloads.
But I think it's better to have a federation of smaller clusters, or separate cluster per datacenter/AZ.
Erlang distribution isn't a silver bullet, though. But for 80% of cases it's enough.
---
[1] Scaling Erlang cluster to 10,000 nodes
https://codesync.global/media/maxim-fedorov-scaling-erlang-c...
BTW: I expirienced the entier us-east region going down just in time when I was demoing my product to a VC. Turns out it was a bug in an AWS Management Console for that region.
Yes, because code is written by humans, and humans make mistakes. "Just have discipline and write good code" doesn't scale, especially in, as Uber's case, quickly growing organizations where large numbers of people new to the codebase are contributing.
IMO a monolith is always the starting point until your team gets big enough to justify breaking it up.
> Seems like this could be dealt with by applying more discipline and just making sure to design things to be more modular and.. separate
This really depends on the team size. As you get above, say, 30 engineers contributing to your codebase daily communicating those logical boundaries becomes nigh impossible without significant operational overhead. You either accept the chaos and let your codebase become a Big Ball Of Mud (most common) or you introduce so much process overhead that your team paradoxically gets slower the more people you add. I'm sure there are exceptions but it's rare.
> Do we need an entirely different architecture, with all the pitfalls that microservices entail, because we can't seem to keep things clean and separate?
Microservices are probably not the tool to reach for no. Several "fat" services separated along some kind of bounded context within your domain? I've seen that work great as a next step in a monolithic architecture evolution. Teams can have the autonomy they need to make decisions quickly, separation of concerns is enforced by the network. The extra overhead incurred isn't untenable since you only have a few services instead of a hundred.
IME about half of the microservice architectures Ive worked on have managed to make sure that the concerns and network boundaries cut across completely different places.
At least this mistake in a "monolith" (hate that word) is easier to undo.
This misstanke can be done there just as well as in monoliths.
Suppose you have a web based CRUD that processes HTTP requests. To do this, it connects to a DB. There is one thread pool for HTTP requests, there's another thread pool for DB connections. Suppose there's an issue that causes a DB connection to hang (either a really slow query, or an issue with the DB thread pooling solution itself). All of the sudden, the DB thread pool is slowly getting exhausted, until no more queries can run because everything is hanging. Soon after, the HTTP request thread pool fills up, because requests are waiting for processing but none is being done. Users either see their requests frozen, or, if you're packaging both the API and the front end resources in the same deployment unit, soon nobody can even load the static resources of the page. I've seen this happen with a Java web app that used an old DB pooling solution that caused a prod outage years ago. Had to rip it out and replace it with something else.
That said, the same could happen with stuff like scheduled processes conflicting, memory exhaustion, CPU utilization, disk usage and so on. Now, there shouldn't be anything preventing you from running multiple instances of monolithic codebases in parallel, though sometimes systems aren't written with that in mind (e.g. no DB queues/RabbitMQ for messaging/scheduled processes, no assumption of multiple instances accessing the same data, in-memory caches instead of Redis and so on) and you end up with a singleton application instead, which is worse than just a monolith. Either way, planning for components/systems failing and being able to recover from that is almost always a good idea (health checks, automated restarts, some horizontal scaling, uptime notifications and alerting, some instrumentation like APM), unless you're really short on time.
None of that means that you should start with microservices though, unless your whole team is really comfortable with that. Monoliths are okay, as are modular monoliths (same codebase, deploy different features for app instances with feature flags, like FEATURE_ENABLE_API, FEATURE_ENABLE_SCHEDULED_PROCESSES etc.).
I guess another benefit is micro services make it easier for teams to work in whatever language/tool they're comfortable with.
I think that's it really. Better defined ownership could be done with modules. Testability could be done with modules. Most of the usual arguments could just be answered basically with well defined code, and when the interfaces are all async then it even means even those modules could have micro services as their underlying implementation.
Modularity in terms of code, and modularity in terms of deployment lifecycle.
As you say most dependency problems can be handled at the level of the code, depending on how you structure the builds etc. ( personally I think automatically pulling in dependencies is a mistake - it should be slightly painful to add a new dependency ).
For me, the reason for microservice like deployment is having a separately lifecycle - being able to start,stop, update separately from other services.
Another aspect which I don't think is talked about enough, is that every deployment has a copy of the code ( in whatever executable form ) - this means I can deploy version 1 of code A in service X and version 2 in service Y at the same time if I need to - ie I can incrementally update the system.
So one way to think about what should be deployed together ( or not ) is thinking not about the code per se - but also the natural deployment cycle.
Being able to experiment with languages and tools (as you mention) is also a pretty big advantage. HTTP offers a universal communications interface beyond language-level APIs and ABIs. When we think about an entire department with many teams with a diversity of expertise, at least we can guarantee everyone understands HTTP.
A pipeline I work on right now ingests large amounts of data from an FTP site and dumps it all onto a Kafka topic which is handled by another process on the consuming end. If this were a monolith it would run faster, but any errors or crashes would either require reprocessing the entire file (not really possible given its size) or keeping state ... which is what we are using Kafka for. We can restart pieces of the pipeline at any time and they just resume where they left off. Having a monolith has no overwhelming advantage to us in this use case.
I say this not to argue one way of another for services, just to say that I think your example is just Kafka, or more generally event driven architecture. One thing that defines services is a request/response messaging pattern. In other words a service makes a request and then they wait for a response, like http. What you’re talking about resembles a service because the producer/consumer are separate processes, but they don’t really have much else in common with services, due to a different messaging pattern.
My point was that I can build a monolith or micro service around Kafka and they end up being equivalent but the multi process approach works better for my needs. The original question was asking what advantages a separate process would have. I gave one.
In the name is the word "service", which means a process that waits and responds to a client's request. This act of request followed by response is fundamentally synchronous.
OASIS defines a service as "a mechanism to enable access to one or more capabilities, where the access is provided using a prescribed interface and is exercised consistent with constraints and policies as specified by the service description". [0]
I see nothing that requires synchronous communication only. Virtually every large micro-service architecture I have ever seen communicates through queues asynchronously.
[0] https://en.wikipedia.org/wiki/Service_(systems_architecture)
It’s super old tech but that doesn’t mean that what’s new is automatically better. And in this case something like small talk sort of seems like it’s from the future when compared to the hodgepodge of tools that are sort of all duck taped together that form the basis of modern infrastructure.
I think in a way our ideas as modern developers have been held back by the present day status quo. In a large way that’s in part because it’s many of our jobs to build these systems.
Sorry, I know it’s a bit of a rant, and I don’t mean any offense at all. I’ve worked on platform before, so maybe I’m projecting, but I really do think it’s good or even necessary to question the present day wisdom.
For me, the funny thing is I've found that it can be just as easy for a single broken microservice to bring down everything else, and it can be a huge pain figuring out where such a problem originated from because the wrong services are erroring out.
Not necessarily the One-Architecture-To-Rule-Them-All (none is), but as you've mentioned a "new LARGE project", it might be a good choice.
See https://scs-architecture.org and https://manuel-kiessling-net.translate.goog/2016/01/02/die-a... for more information.
A single regression in your isEven microservice can also take the whole system down. You can mitigate this by slowly rolling out new versions to try and catch regressions early so you can revert.
>Risky, expensive deployments. These were painful and time consuming to perform with the frequent need for rollbacks.
Code that is going to be deployed all needs to be deployed. Painful deploys are going to happen with microservices too.
>Poor separation of concerns. It was difficult to maintain good separations of concerns with a huge code base. In an exponential growth environment, expediency sometimes led to poor boundaries between logic and components.
Making libraries will boundaries without having to do an expensive network call.
>Inefficient execution. These issues combined made it difficult for teams to execute autonomously or independently.
Monoliths let you work independently too. You go in and change something and now it is changed. No part of monoliths make it harder as a single person or team to make changes.
Also keep in mind Uber ships Android and iPhone apps which are monoliths. They don't force you to install an app for entering a location, an app for making a payment, an app for looking up pricing, etc.
If you have one then you're doing it wrong and don't get the most basic thing about microservice architecture. Microservices must be business-aligned and as independent as possible.
> They don't force you to install an app for entering a location, an app for making a payment, an app for looking up pricing
They probably do. I bet their Android app has independent dynamic feature modules that are loaded on demand.
No. Microservices as the name suggests are micro and their functionality can be pretty limited. If you don't have any trivial microservices then you are doing it wrong. You may be objecting because of how inefficient it seems to put a function like that over a network call, but that just makes the performance downside of the microservice architecture clearer.