Microservices
grug wonder why big brain take hardest problem, factoring system correctly, and introduce network call too
seem very confusing to grug
Microservices
grug wonder why big brain take hardest problem, factoring system correctly, and introduce network call too
seem very confusing to grug
"We want to modernize how we access our FOO_TABLE for SCALE_REASONS by moving it to DynamoDB out of MySQL - unfortunately, 32 of our 59 teams are directly accessing the FOO_TABLE or directly accessing private methods on our classes. Due to competing priorities, those teams cannot do the work to move to using our FOO_SERVICE and they can't change their query method to use a sharded table. To scale our FOO_TABLE will now be a multi-quarter effort providing the ability for teams to slow roll their update. After a year or two, we should be able to retire the old method that is on fire right now. In the meanwhile, enjoy oncall."
Compare this to a microservice: Team realizes their table wont scale, but their data is provided via API. They plan and execute the migration next sprint. Users of the API report that it is now much faster.
Bologna. Choosing clear abstractions is an enabler of focus, but that doesn’t necessarily imply those abstractions are a network call away.
We have an org wide push to get more linting and more checks in place. The damage is done and now we have a multi-quarter effort to re-organize all our code.
This _can_ be enforced via well designed modules. I've just not seen that succeed. Anywhere. Microservices are a pain for smaller teams and you have to have CI and observability and your pains shift and are different. But for stepping on eachother? I've found microservices to be a super power for velocity in these cases. Can microservices be a shitshow? Absolutely, esp. when they share data stores or have circular dependencies. They also allow teams to be uncoupled assuming they don't break their API.
Well, that's different.
My experience is that "leadership" often finds quality to be expensive and unnecessary overhead.
That's one reason that I stayed at a company that took Quality seriously. It introduced many issues that folks, hereabouts would find unbearable, but they consistently shipped some of the highest-Quality (and expensive) kit in the world.
Quality is not cheap, and it is not easy. It is also not really the path to riches, so it is often actively discouraged by managers.
Presumably you would still need tooling to enforce teams not breaking their API, and also to prevent people just modifying services to expose private internals that should not be part of the public API?
1) engineering needs a reason to embrace abstraction. Because the only thing that stops a new cowboy engineer is their peers explaining to them in what ways this isn't the Wild Wesst. One can assume if rules would benefit the team, they'd be doing them already, so why don't they? Maybe they perceive feature output velocity to be too high to risk changing method. Maybe decisionmaking power rests in the hands of one system architect who is holding the whole machine in their head so things that look complex to others seem simple to them (in that case, the team needs to spread around peer review signoff responsibilities on purpose, so one engineer can't be the decisionmaker and the architecture itself must self-describe). Maybe (this is how I see it usually happen) they were a three-person startup and complexity crept up on them like a boiling frog. Whatever the reason, if you're gonna convince them otherwise, someone's gonna have to generate hard data on how changing the abstraction could make their jobs easier.
2) If management has no idea what "prioritize quality" means (meaning no metrics by which to measure it and no real grasp of the art of software engineering), the engineers will interpret buzzwords as noise and route around them. Management needs to give actionable goals other than "release feature X by Y date" if they want to change engineering culture. That can take many forms (I've seen rewarded fixit weeks and externally-reported issue burndown charts as two good examples).
3) Engineering leadership needs time and bandwidth to do training so they can see outside their prairie-dog hole over to how other teams solve problems. Otherwise, they get locked into the solutions they know, and the only way new approaches ever enter the team is by hires.
And the key thing is: microservices may not be the tool for the job when all is said and done. This is an approach to give your engineering team ways to discover the right patterns, not to trick them into doing microservices. Your engineering team, at the end of the day, is still the best-equipped people to know the machinery of the product and how to shape it.
Really, you can't? Then I struggle to see how'll get anything else right. I've done it by using separate build scripts. That way only the interfaces and domain objects are exposed in the libraries. Now you lock down at the repository level access to each sub-project to the team working on it. There you go: modularity, boundaries, all without network hops.
With an HTTP API boundary, you don't get to reach into my code - it is literally impossible.
I mean take your pick, anything open source be it desktop or web has a huge and deep dependency tree all the way down to libc.
Just wondering.
EDIT: Typos and desktop.
... followed by howls of anguish from the rest of the business when it turns out they were relying on reports generated from a data warehouse which incorporated a copy of that MySQL database and was being populated by an undocumented, not-in-version-control cron script running on a PC under a long-departed team member's desk.
(I'm not saying this is good, but it's not an unlikely scenario.)
The team that maintains the service is also responsible for how that service is represented in the data warehouse.
The data warehouse tables - effectively denormalized copies of the data that the service stores - are treated as another API contract - they are clearly documented and tested as such.
If the team refactors, they also update the scripts that populate the data warehouse.
If that results in specific columns etc becoming invalid they document that in their release notes, and ideally notify other affected teams.
Once you get to this point, there's no path forward. Either you have to making some breaking changes or your product is calcified at that point.
If this is a real concern then you should be asking what you can do to keep from getting into that state, and the answer is encapsulating services in defined interfaces/boundaries that are small enough that the team understands everything going on in the critical database layer.
This definitely happens but at some point someone with authority needs to show technical leadership and say "you cannot do this no matter how desperately you need those reports." If you don't have anyone in your org who can do that, you're screwed regardless.
When I hear success stories like that, I have to ask "Is there some inherent benefit to the abstraction or did you get lucky in picking your cleave-points?"
- Performance optimizations which can't be realized without changing the interface model. For example, FOO_TABLE should actually be BAR and BAZ with different update patterns to allow for efficient caching and querying.
- Domain model updates, adding/updating/removing new properties or entities.
This kind of update will still require the 32 consumers to upgrade. The API-based approach has benefits in terms of the migration process and backwards-compatibility though (a HTTP API is much easier to version than a DB schema, although you can also do this on the DB level by only allowing queries on views).
I honestly expected people used passive-aggressive corpo-slang only for work / ironically. But this reads intentionally obfuscated.
But, to answer to the substance: you for some reason assumed that whoever designed first solution was an idiot and whoever designed the second was, at least clairvoyant.
The problem you describe isn't a result of inevitable design decisions. You just described a situation where someone screwed up doing something and didn't screw up doing something else. And that led you to believe that whatever that something else is, it's easier to design.
The reality of the situation is, unfortunately, the reverse. Upgrading microservices is much harder than replacing components in monolithic systems because it's easier to discover all users of the feature. There are, in general, fewer components in monolithic systems, so less things will need to change. Deployment of a monolithic system will be much more likely to discover problems created by incorrectly implemented upgrade.
In my experience of dealing with both worlds, microservices tend to create a maze-like system where nobody can be sure if any change will not adversely affect some other part of the system due to the distributed and highly fragmented nature of such systems. So, your ideas about upgrades are uncorroborated by practice. If you want to be able to update with more ease, you should choose a smaller, more cohesive system.
There are definitely reasons why a microservice architecture could make sense for an organization, but "we don't have good application engineers/architects" should not be one of them. Going to a distributed microservice model because your teams aren't good enough to manage a modular monolith sounds like a recipe for disaster.
Instead of microservices, you could add a static analysis build step that checks if code in packages is calling private or protected interfaces in different packages. That would also help enforce service boundaries without introducing the network as the boundary.
With microservices, you deploy the updates to public facing services first, write any learnings down, pass it along to the next most vulnerable tier.
Heck on multiple occasions I've been part of projects where just updating the build system for a monolithic codebase was a year+ long effort involving dozens of engineers trying to work around commits from everyone else.
Compare this to a microservice model where you just declare all new services get the new build tools. If the new tools come with a large enough carrot (e.g. new version of typescript) teams may very well update their own build tooling to the latest stuff without anyone even asking them to!
Dry is literally "Don't Repeat Yourself" and is definitely pushed for cleaning up redundant code, so it's not unreasonable for people to think that's what about. It's only recently that people have pointed out that there's a difference between Duplicated code and Repeated code.
https://ruby-doc.org/core-3.1.0/NEWS_md.html#label-IRB+Autoc...
Isolation is not anything to scoff at, it's one of the most powerful features you can encode into your software. Isolation can improve performance, it can create fault boundaries, it can provide security boundaries, etc.
This is the same foundational concept behind the actor model - instead of two components being able to share and mutate one another's memory, you have two isolated systems (actors, microservices) that can only communicate over a defined protocol.
That is not too true at all. I've seen "microservice" setups where one microservice depends on the state within another microservice. And even cases where service A calls into service B which calls back into service A, relying on the state from the initial call being present.
Isolation is good, but microservices are neither necessary nor sufficient to enforce it.
Let me be a bit clearer on my point because I was wrong to say that you have to treat a service as being totally isolated, what I should have said that they are isolated, whether you treat them that way or not. There is a physical boundary between two computers. You can try to ignore that boundary, you can implement distributed transactions, etc, but the boundary is there - if you do the extra work to try to pretend it isn't, that's a lot of extra work to do the wrong thing.
Concretely, you can write:
rpc_call(&mut my_state)
But shared mutable state is trivial without the process boundary. It's just... every function call. Any module can take a mutable pointer and modify it. And that's great for lots of things, of course, you give up isolation sometimes when you need to.
If your web service is generally low-state and these problems are manageable for the complexity scale you're solving for, microservices aren't really something to even consider - I mean, you basically have a microservice already, it's solving a single problem within a bounded context, give or take. It's just... one service and one context.
On top of this, you need to consider that most of the software you are going to write will be based on existing components. Many of these have no desire to communicate over network, and your micro- or w/e size services will have to cave in to their demands. Simple example: want to use Docker? -- say hello to UNIX sockets. Other components may require communication through shared memory, filesystem, and so on.
Finally, isolation is not a feature of microservices, especially if the emphasis is on micro. You have to be able to control the size and where you want to draw the boundary. If you committed upfront to having your units be as small as possible -- well, you might have function-level isolation, but you won't have class- or module- or program-level isolation, to put it in more understandable terms. This is where your comparison between the actors model and microservices breaks: first doesn't prescribe the size.
http://domino.research.ibm.com/library/cyberdig.nsf/papers/0...
In no way is this throwing out performance. It's sort of like saying that Kafka is in Java so you're throwing away performance when you use it, when there are massive performance benefits if you leverage partition isolation.
> Many of these have no desire to communicate over network, and your micro- or w/e size services will have to cave in to their demands. Simple example: want to use Docker? -- say hello to UNIX sockets. Other components may require communication through shared memory, filesystem, and so on.
I'm not sure what you're referring to. Why would that matter at all? I mean, ignoring the fact that you can easily talk to Docker over a network.
> Finally, isolation is not a feature of microservices,
Isolation is a feature of any process based architecture, whether it's SoA, actors, or microservices.
> well, you might have function-level isolation, but you won't have class- or module- or program-level isolation,
You get isolation at the service layer. I don't see why that would be contentious, it's obvious. If you're saying you want more isolation, ok, you can write your code to do that if you'd like.
> first doesn't prescribe the size.
Yep, the actor model is very low level. Microservice architecture is far more prescriptive. It's one of the reasons why I think Microservice architecture has been far more successful than actor based systems.
So, here are some concerns that affect performance, they mostly come from the need of various translations done by either iptables, eBPF analogues, arpatables, DNS server(s). If you read about benchmarks of Calico and Cilius, you'll see that they concentrate on performance of eBPF code necessary to do all these translations. My claim about performance tax is based on the idea that w/o Kubernetes you wouldn't need such translations (but, of course, you could build your own version of Kubernetes networking with a lot of software-defined virtualization, in which case you'd be in the same boat).
> Kafka
Is as horrid as it sounds. It's awful performance-wise on all counts. I'm not sure what point are you trying to make. Can you choose a better example? I mean, Kafka performs very poorly in all configurations, so, to think that it can be a good example of software that wants to achieve good resource utilization is just bound to give you bad results.
> You get isolation at the service layer. I don't see why that would be contentious, it's obvious. If you're saying you want more isolation, ok, you can write your code to do that if you'd like.
You either genuinely didn't understand what this is about, or pretend to not understand something really simple. "Micro" in microservices means that your services are small. There aren't any meaningful isolation tools or approaches when it comes to microservices, because isolation at the level of service doesn't matter / is trivial to achieve by many other means / is not a problem in real-world programs.
> Yep, the actor model is very low level.
This is simply a nonsense statement. Low on what scale? Your answer reads as if it was generated by a chatbot. I.e. words from the same general domain strung together, but make no sense.
> Microservice architecture has been far more successful than actor based systems
How did you count? How do you even tell if something is a microservice-based? This is just as absurd of a claim as saying "78% of enterprises use Kubernetes" (I believe I saw this unfettered inanity on cloud-native foundation's Web site). How do you tell if it's successful? What if actor model is a more generic description which beside other things, also captures microservices?
I mean, in more simple language, this is talking out of your rear. It's not a real argument anyone should pay attention to.
You're the one who brought it up?
> My claim about performance tax is based on the idea that w/o Kubernetes you wouldn't need such translations (but, of course, you could build your own version of Kubernetes networking with a lot of software-defined virtualization, in which case you'd be in the same boat).
You don't need any of those and I don't know why you think otherwise. You can just use the host network and do whatever you want, as with any container.
> I'm not sure what point are you trying to make.
That Kafka's architecture allows you to put data into partitions and route it based on that data, which allows for "shared nothing" architectures. But whatever, you're clearly not going to get this point from this example. To be clearer, your point of "you pay a cost here so you're losing performance" ignores that you can get performance elsewhere.
> There aren't any meaningful isolation tools or approaches when it comes to microservices, because isolation at the level of service doesn't matter / is trivial to achieve by many other means / is not a problem in real-world programs.
Not true at all. Services that own a domain of work are a great place to perform isolation and security boundaries.
> Low on what scale?
As in an actor is a foundational primitive for asynchronous computation. You have to build up protocols on top of actors, hence all of OTP.
> I.e. words from the same general domain strung together, but make no sense.
I think that's because I actually know what I'm talking about and you're finding it hard to keep up?
> How did you count?
Because it's obvious? Like it's not even close. Actor based systems are exceedingly rare, microservices are not.
> I mean, in more simple language, this is talking out of your rear. It's not a real argument anyone should pay attention to.
One of us actually knows what they're talking about and I doubt we'll agree who ti is.
a) Intermittent failures force you to treat systems as if they can fail - and since every system can fail, that's a good thing. This is why chaos engineering is great.
b) Failures across a network are the best kind - they're totally isolated. As I explain elsewhere, it's impossible to share state across a network, you can only send copies of state via message passing.
These things matter more or less depending on what you're doing.