Hacker News Reader

> They force you to not share memory on a single system, they encourage you to focus on how you communicate, they give you scale and failure isolation, for force you to handle the fact that a called subroutine might fail because it's across a network.

The problem: distributed systems are hard to get right. Better stay away from them unless you really need them, AND you have the time/resources to implement them correctly. The benefits of microservices are a bad excuse, most of the time.

Exactly, distributed systems are hard. Very hard. It's an entire field of study.

Don't opt for something very, very hard unless you have to.

Heh, I have a masters degree focused on distributed systems. My thesis was about tracing and debugging in microservice-style systems. I generally write monoliths, on purpose. The massive overheads and debugging nightmares are not worth it most of the time.

Global state still gets pushed out into backend services (redis, postgres) and I can still scale horizontally all day, but there’s no crazy chain of backend interservice HTTP requests to cause no end of chaos :)

Do you mind if you can link me to your thesis? I am actually looking into tracing systems right now.

Honestly, you're probably better off looking at https://opentracing.io/. For a bit of context, I started working on my thesis in 2009 when there weren't many commercial tools available for doing this kind of tracing; by time I was done, there were a ton of open source and commercial tools available.

Thesis is here: https://harvest.usask.ca/handle/10388/ETD-2015-06-2185

> If you read up on the fundamental papers regarding software reliability this is something that's brought up ad nauseum.

I don't believe there are any papers that show that adding network hops to an application makes it more reliable. I would be extremely interested in any references you could provide.

Not the network bit, the separation of state and the process abstraction.

https://www.hpl.hp.com/techreports/tandem/TR-85.7.pdf

Here's one paper that discusses building reliable systems from two abstractions - the process and transaction.

> for force you to handle the fact that a called subroutine might fail because it's across a network.

That just adds one failure mode to the list of failure modes people ignore due to the happy-path development that languages with "unchecked exceptions as default error handling" encourage.

> Functions allow shared state, they don't isolate errors. Processes over networks do. That's a massive difference.

Except not, because "just dump that on a database/kv-store" is an all-too-common workaround chosen as an easy way out. This problem is instead tackled by things such as functionally pure languages such as Haskell and Rust's borrow checker, and only up to a certain degree at which point it's still back into the hands of the programmer's experience; though they do help a ton.

> That just adds one failure mode to the list of failure modes people ignore due to the happy-path development that languages with "unchecked exceptions as default error handling" encourage.

There are only two meaningful failure modes - persistent and transient. So adding another transient failure (network partition) is not extra work to handle.

> Except not, because "just dump that on a database/kv-store" is an all-too-common workaround chosen as an easy way out.

Just to be clear, microservices are not just separate binaries on a network. If you're not following the actual patterns of microservice architecture... you're just complaining about something else.

>Just to be clear, microservices are not just separate binaries on a network. If you're not following the actual patterns of microservice architecture... you're just complaining about something else

So what you're saying is that the way to avoid this problem in a microservice architecture, is to be disciplined and follow the right patterns. Then couldn't I just follow the same patterns in a modular monolith (eg: avoid shared state, make sure errors are handled properly, etc) and get the bulk of the benefits, without having to introduce network related problems into the mix?

Because engineering discipline is actually hard. Not necessarily in the "here is how you do it" sense, just in the sense of getting the buy-in from engineers and engineering leadership that will make it happen.

This is like the one thing that microservices might actually be sort of good at: drawing a few very hard boundaries that do actually sort of push people in the general direction of sanity, e.g. it's easier to have basic encapsulation when the process might be on another computer...

I cannot figure out how you can see that. RPC just adds a "Remote" on top of the "Procedure Call" part, we add a failure mode but the thought process is the same.

As witnessed by many teams, spaghetti happens just as poorly in a distributed monolith as it does in a proper monolith, it just adds latency and makes it harder to debug.

The boundaries you're imagining are not drawn by the technology nor by the separate codebases, they're drawn by the programmers making the calls. And I guarantee you that the average developer with their usual OOP exposure can understand much more easily where to draw decent boundaries following some pattern like Clean/Hexagonal/Onion/Whatever Architecture as opposed to microservices, where it's far more arbitrary to determine the concerns of a microservice, specially when a usecase cuts through previously drawn boundaries.

> Then couldn't I just follow the same patterns in a modular monolith (eg: avoid shared state, make sure errors are handled properly, etc) and get the bulk of the benefits, without having to introduce network related problems into the mix?

Sure. Microservice architecture is a set of design patterns and a discipline for knowing how to structure your applications.

Many, including myself, would argue that by leveraging patterns such as separate processes as a focal point for the architecture leads to patterns that are harder to break out of and abuse, but of course anyone can do anything.

Error handling is the easiest one. With any 'service oriented' approach, where processes are separated, you can't share mutable state without setting up another service entirely (ex: a database). Microservices encourage message passing and RPC-like communication instead, and it's much easier to fall into the pit of success.

Could you do this with functions? Sure - you can just have your monolith move things to other processes on the same box. Not sure how you'd get there without a process abstraction, ultimately, but you could push things quite far with immutability, purity, and perhaps isolated heaps.

> Except not, because "just dump that on a database/kv-store" is an all-too-common workaround chosen as an easy way out.

Then maybe we should be criticizing that instead? Like, that'd still happen with Haskell or Rust (or, in my experience, Erlang, with that KV store specifically being ETS). Seems like that's the thing that needs addressed.

> Functions allow shared state, they don't isolate errors. Processes over networks do. That's a massive difference. > > If you read up on the fundamental papers regarding software reliability this is something that's brought up ad nauseum.

I think you missed the point - just using separate processes does not guarantee you separate errors and state between services, there's lots of ways to get 'around' that. What if the two services talk to the same database/service? What if there's weird co-dependencies and odd workarounds for shared state? What if one service failing means the entire thing grinds to a halt or data is screwed up?

Now that said, yes, if you use good development practices and have a good architecture microservices can work quite well, but if you were capable of that you probably wouldn't have created a non-microservice ball of mud. And if you're currently unable to fix your existing ball of mud, attempting to make it distributed is likely going to result in you adding more distributed mud instead. In other words, the problem here isn't really a technical one, it's a process one. And using their current failing processes to make microservices is just going to make worse mud, because they haven't yet figured out how to deal with their existing mud.

> just using separate processes

Microservice architecture is not just using separate processes.

If you're talking about 'over a network', sorry I left that out, but I'd argue it's largely implied. That said, if you add 'over a network' there nothing that I said changes.

My point is that microservice architecture is not just the singular pattern of "code talking over the internet", but a collection of patterns and techniques, focusing on when and where to split up code, and to focusing on the communication strategy you use.

You can 'get around' microservice architecture by not doing it. The point is that if you're familiar with it it's a lot easier to 'accidentally' be successful - or at least, that's the proposition.

> My point is that microservice architecture is not just the singular pattern of "code talking over the internet", but a collection of patterns and techniques, focusing on when and where to split up code, and to focusing on the communication strategy you use.

> You can 'get around' microservice architecture by not doing it. The point is that if you're familiar with it it's a lot easier to 'accidentally' be successful - or at least, that's the proposition.

I don't disagree with your definition (Though I think it's a hard sell to say that's the "correct" definition, when it really depends on who you ask), but the things you described aren't exactly novel programming concepts unique to or invented by microservices. I could say the same things about OOP - modularity and API design aren't new things.

With that, the idea that things like shared state are removed due a network being between the services just isn't true, it still requires design effort to achieve that goal - I would argue the same design effort as before. And if your previous design efforts and development practices (for whatever reason) did not lead to good designs, and you're not actually making an attempt to fix the existing issues along with the reasons why you were making such decisions, then you're likely just going to repeat the same mistakes but now with a network in-between.

And yes, to an extent I agree it's not "really" microservices at that point, you're just emulating something that "looks" like microservices when it's really a ball of mud. But can't you just as easily argue they weren't creating a proper "monolith" to begin with?

> the things you described aren't exactly novel programming concepts unique to or invented by microservices.

I don't think anyone has ever claimed novelty in microservice architecture. It's very clearly derivative.

Shared state is moved. You get some level of isolation simply from the fact that there are two distinct pieces of hardware operating on state. You can then move state to another piece of hardware, and share that state between services (somewhat discouraged), but this is a much more explicit and heavy lift than just calling a function with an implicitly mutable variable, or sharing a memory space, file system, etc.

> But can't you just as easily argue they weren't creating a proper "monolith" to begin with?

Maybe. You can say this about anything - there's no science to any of this. I could say functional programming leads to better code than oop, but I couldn't really prove it, and certainly you can still do crazy bad garbage coding things with FP languages. But I would argue that the patterns that make up microservice architecture can help you make bad things look bad and good things look good. It's not magic fairy dust that you can just use to make your project "better" by some metric, but no one informed would ever claim so.

> Functions allow shared state, they don't isolate errors.

So why not use Haskell (or other pure language)? It's pure, so functions don't share state. And you don't have to replace function call with network call.

You'll never achieve the level of operational stability of an air gap that you would without a strictly in process system.

Shared state is part of it. Isolation of failure is another. Your Haskell code can still blow up.

Immutability makes that way easier to recover from, but it's just one part.

Of course, microservices are much more than processes over a network, they're a discipline. And I think one can go much further than microservices as well - microservice architecture doesn't tell you how processes should communicate, it's more focused on how they're split up, and leaves the protocol to you. Good protocol design is critical.

> They force you to not share memory on a single system,

So instead people use a single SQL database between 20 microservices.

> give you scale and failure isolation,

Only if you configure and test them properly, and they actually tend to increase failure and make it harder to isolate its origin (hello distributed tracing)

> force you to handle the fact that a called subroutine might fail because it's across a network

They don't force that at all. It's good when people do handle that, but often they don't.

> I would argue that if you're manually scaling things then you're doing it wrong

And I would argue that if people are given a default choice of doing the wrong thing, they will do that wrong thing, until someone forces them not to.

Microservices allow people to make hundreds of bad decisions because nobody else can tell if they're making bad decisions unless they audit all of the code. Usually the only people auditing code are the people writing it, and usually they have no special incentive to make the best decisions, they just need to ship a feature.

If your teams are defaulting to doing the wrong thing in every case, you are having a crisis of leadership.

The problem might lie in your technical leadership (e.g. making bad design decisions and failing to learn from experience) or from product leadership (e.g. demanding impossibly short turnaround on work, leading to reliance on solutions that are purely short term focused).

Well, I'd answer that with two things 1) good leadership is probably rare, and 2) doing the wrong thing isn't always wrong.

In my experience, the best leadership comes from people who don't hyper-focus on one thing, but are constantly thinking about many different things, mostly centered around the customer experience (because that's the whole point of this thing: to make/sell/run a product, not to have perfect software craftsmanship). I find those people rare and usually not working at lower levels of a product.

In a sense, doing the wrong thing is perfectly fine if you don't need to be doing the right thing to provide a great customer experience. Of course maintainability, availability, recoverability, extensibility, etc are all necessary considerations, but even within these you can often do the wrong thing and still be fine. I have seen some truly ugly stuff in production yet the customers seemed happy. (though there were probably some chronically sleepless engineers keeping it alive, which sucks and shouldn't happen)

I don't like microservices because of how poorly they're usually implemented by default. But at the same time, even poorly implemented microservices give some great benefits (more rapid/agile work, reusability, separation of concerns, BU independence), even if you're struggling against some harmful aspects.

> Processes and networks are amazing abstractions. They force you to not share memory on a single system, they encourage you to focus on how you communicate, they give you scale and failure isolation, for force you to handle the fact that a called subroutine might fail because it's across a network.

Stop on! Its so much easier to enforce separation of concern if there is an actual physical barrier. Its just to easy to just slip bad decision through peer review.

So I totally agree

Stop off! Drop a bunch "microservices" into the same network without any access control and you don't have a physical barrier at all! In fact, it becomes even harder to have a clue as to what's interfacing with what unless you can observe your inter process traffic.

This is a completely unrelated issue. Microservices have nothing to do with access control. If anything, they should lend themselves to more fine grained control, but again, microservice architecture says nothing about it.

This is exactly the point of the original article by Monzo. Microservices don't provide any tool to deal with access control, thus you have to implement it yourself. I rather avoid that when I can.

Unrelated? I' beg to differ. More to the point, all systems have an access control model. It could be explicit or implicit depending on how the system is built. Regardless, it defines the barriers and boundaries between everything in the system.

The comment suggests that just by separating things in the popular sense of microservices results in a barrier to enforce separation of concerns. That's how I read it, at least, and I find that to be is misleading.

This is the crux I see around popular discourse of microservices. It's often presented without a broader context.

> without any access control

The point, I reckon, is that you should be implementing access control.

> Stop on!

Not sure if typo or joke about out-of-order message processing in distributed systems.

> just to easy to just slip

And maybe also about the impossibility of exactly-once message delivery?

Plenty of languages allow to write modular code, all the way back to early 80's.

The developers that aren't able to write modular code, are just going to write spaghetti network code, with the added complexity of distributed computing.

I'm reminded of Jeff Bezos and his push for services over web protocols.

this is the closest reference I can find:

http://homepages.dcc.ufmg.br/~mtov/pmcc/modularization.pdf

The point of his edict was that he wanted to reduce silos, not to "adopt web prootcols."