Comment by zzzeek | Hacker News Reader

Comment by zzzeek | Hacker News Reader

Parent Full thread

> Take for example Celery. With Celery all you need to do to implement a queuable background task system that's trivially scalable is to write the background tasks,

that's a little bit of a straw man because that's not the "microservice" architecture this post is talking about. I personally wouldn't call that a "microservice" architecture, I'd call it, "a background queue", although strictly speaking it can be described as such.

what this post is talking about are multiple synchronous pieces of a business case being broken up over the http / process line for no other reason than "we're afraid of overarchitecting our model". This means, your app has some features like auth, billing, personalization, reporting. You start from day one writing all of these as separate HTTPD services rather than just a single application with a variety of endpoints. Even though these areas of functionality may be highly interrelated, you're terrified of breaking out the GOF, using inheritance, or ORMs, because you had some bad experience with that stuff. So instead you spend all your time writing services, spinning up containers, defining complex endpoints that you wouldn't otherwise need...all becuase you really want to live in a "flat" world. I'm not allowed to leak the details of auth into billing because there's a whole process boundary! whew, I'm safe.

Never mind that you can have an architecture that is all separate process with HTTP requests in between, and convert it directly to a monolithic one with the same strict separation of concerns, you just get to lose the network latency and complex marshalling between the components.

Programmers are, by and large, quite bad at not breaking encapsulation when they're dealing with a monolith. Not their fault, really, but when management comes to you and says, hey, can't you get it done in just a day, we really need this, and you know you could if you just hacked through that particular isolation barrier just this once, and yeah it will create bad bugs if things change in a particular way in the future, but you'll leave a comment so it will be fine and this way you can go home and have an ipa and finish re-runs of that show you like so you break the rules just this once. And that only happens a few more times, and then slowly bit by bit things become intertwined and rigid, and then five years down the road when half the people have left and the team has grown and other teams use the project and are trying to commit code to it you have to go to management, hat in hand, sorry, we need to rewrite it from scratch to keep adding features, new and exciting and unexpected things happen when we change things in the current project.

Independent services that create a coherent whole enforce isolation barriers. I don't believe in microservices. These things don't need to be micro. They can be just normal-sized services. I don't even particularly care internal to those barriers how bad things get. There are programmers that write shit I think is hot garbage and will cause all kinds of bugs as time goes on. But if they are confined to their space space, and they're solving their problem and are happy and making management happy then w/e. That piece of the whole will collapse and die eventually, but it won't take everything with it. It's just a piece, and it provided value for a while, so probably worth it from a business sense.

But when you have a monolith? And Developer Dave and Programmer Pete tell Big Boss Bob that they could sure get that feature out quick if only it weren't for those pesky rules preventing them from putting in a few mutexes in that one module so they can just read some data directly from it, and boy wouldn't it that be swell? Well Big Boss Bob says we need to get this feature SHIPPED BOYES so buckle up and put in those mutexes, and now the fucking thread goes into a deadlock state every so often but it's really intermittent so you spend hours debugging the damn things and late nights because yeah features gotta ship but shit gotta work, and you trace it back to Developer Dave and Programmer Pete and their change but what do you do? Big Boss Bob said do it that way and what? You gonna whine to Director Dan about it? Is that gonna get you back your late night spend figuring out what was going on? Nah.

Make systems that are just small that when people break them completely it doesn't mean the end of the world to throw it out and start over.

> Programmers are, by and large, quite bad at not breaking encapsulation when they're dealing with a monolith.

What is there about microservices that makes them better at this? Most of the stuff I have seen has been highly coupled.

Physical isolation. I don't actually like the arguments but I can't quite say it's wrong, because the Java landscape just put a huge amount of effort into modularity and encapsulation because developers kept using reflection to bypass module boundaries. Usually for performance reasons. With a microservices architecture that is impossible anymore because there is physically no way to read across address space boundaries without sending a message and introducing an RPC, which would require the support of the target module.

Now that said, with the new jigsaw module system in the jvm, and the multi-language support that is constantly getting better, a disciplined enough senior management team could enforce module boundaries within the process. It means any change to jvm command line flags would require the approval of the most senior tech lead, because that's how module boundary enforcement can be disabled, but if you have that and it works you would get significant performance and simplicity benefits.

Hard process boundaries, largely. I mean, unless you use mmio in which case you're probably going out of your way to break things. Also design constraints should be taken into account- if you need to build highly performant systems, yeah, probably shouldn't have a bunch of services talking to each other over http/json. But you could probably also isolate the part of the system that actually needs to be performant.

Also, like I said, not a fan of the "micro" terminology, it just confuses the issue.

And keep in mind that coupling is not the opposite of encapsulation. Encapsulation just means hiding internal state. So unless you're going completely out of your way to break things, you'll have services that communicate through some message passing protocol, which means services are inherently encapsulated. It's not that you can't break that, it's that generally it's hard.

Here's an example of breaking it, though:

You have an API that talks to the database. You need to get users, groups, and preferences. Instead of writing different endpoints to access all of these things individually, clever you decides to simply make an endpoint that is "postQuery" and you give it a sql string and it will return the results. Great. You have now made the API effectively pointless.

Another example: you have an API that needs to perform standard accounting analysis on an internal dataset. Instead of adding endpoints for each calculation (or an endpoint with different inputs), you create an endpoint "eval" that will internal eval a string of the language of your choice. Congrats, you can now use that API to execute arbitrary code! No need to define any more pesky endpoints.

So yeah, absolutely people can make shit decisions and build garbage. It's entirely possible. But hey, at least it's pretty obvious this way and if you see your team do this you can look for another job.

You'll never ever be able to stop bad developers from making poor choices and ruining things.

So you have encapsulated services... boss comes and says "we need feature X right away". What if feature X spans all of your microservices? The bad programmer will hack together a monstrosity between multiple services. It's the micro-lith problem: instead of a monolith, you have a monolith disguised as micro services. Now their poor choices are spread across a lot of services and distributed, it's not really confined to just one service.

> What if feature X spans all of your microservices? The bad programmer will hack together a monstrosity between multiple services.

Wait, what is a good programmer supposed to do in this scenario?

I'm really tired of immensely awkward and problematic design patterns that are intended to act as guard rails for good design. It is a myth that this actually works. A rushed project is a rushed project; a team that is inclined to overbuild will overbuild no matter what artificial constraints you give them up front. A microservice design that has concurrency problems can be extremely difficult to debug, in my personal experience this is easily much more difficult than debugging a monolith. Having to spin up 30 services in order to reproduce an issue that could otherwise be done in a single in-memory unit test is a real thing.

Not every tool is right for every job. But the idea that services provide no guardrails is, in my experience, categorically false. Like I said, not a fan of "micro", but there is a ton of inherent value in defining implicit isolation barriers. It's absolutely true that there is NO design impervious to idiots. This has always been true and will remain true until the heat death of the universe. But man, "hard" isolation barriers are nice to have, either through services or packages. I worked for a company and I was on a team that built a set of packages which were distributed to a bunch of other teams, and due to organization changes another team took ownership of one of those packages. In almost no time flat it became absolutely convoluted. Not my problem though, right? Here's the thing: it became buggy and hard for them to maintain, but it had no effect except for when using that particular package. I had worked on a project previously where there was a legit monolith, no packages or library delineations, one code base, and boy with all the teams sticking their fingers in that particular pie things got out of hand in a BAD way. It was so bad that certain teams would get into change wars where one group would make a change in a module that would break another group who would change it back and break the first group, and back and forth.

After my team had taken that project and split it into a bunch of smaller packages, it's not like everyone magically became better programmers. The same people who were introducing fucking spaghetti code that did who-knows-what in overly complex ways were still around, and they were given effectively their own sandboxes. In fact quite a few more teams within the company began using those packages, which just became modules they shipped with. We no longer had to deal with people screwing with core packages because they no longer had ownership, they could no longer make merges into that area of the code base, so we could keep things stable.

So I'm a hard sell on monoliths. Like, I'm not actually pro micro-services, per se. I'm mostly just anti-monolith. Giant code bases with multiple teams simultaneously contributing are doomed to fucking disaster.

One last comment on this, if you have like 5 guys just hacking away on some project and you feel that splitting it up into 30 micro-services is the only way to make things work then you've got problems. I'm not talking about small teams building straightforward systems. I'm talking about giant organizations building giant complicated systems, trying to design and manage that.

> that's a little bit of a straw man because that's not the "microservice" architecture this post is talking about. I personally wouldn't call that a "microservice" architecture, I'd call it, "a background queue", although strictly speaking it can be described as such.

It is not a strawman; it's a concrete example of the technical, practical, operational, and economical advantages of microservices architecture, more specifically service reuse, specially managed services provided by third parties.

While you're groking how a multihreadig library is expected to be used, I already fired a message broker that distributes tasks across a pool of worker instances. Why? Because I've opted not to go with the monolith and went with the microservices/distributed architecture approach.

You can have a worker pool with a monolith too. Just run multiple copies of your app.

> While you're groking how a multihreadig library is expected to be used, I already fired a message broker that distributes tasks across a pool of worker instances. Why? Because I've opted not to go with the monolith and went with the microservices/distributed architecture approach.

I'm so confused by that statement... because I can't for the life of me figure out how you got there.

You absolutely can have a monolith which is multi-threaded or asynchronous and "resource/task pools." The JVM for instance has threads, I use BEAM (Elixir) personally, and it's even pre-emptively scheduling my tasks in parallel and asynchronously... but, I still don't get what multi-threading has to do with microservices.

Microservices and monoliths are boundaries for your application they aren't implementation details (i.e. all microservices must be asynchronous is strictly not true) in and of themselves, they're design details. That design can influence the implementation but they are separate.

Ex. there are plenty of people who use Sidekiq and redis like you're using Celery but don't call it a microservice. It's just a piece of their monolith since it's largely the same depdencies.

beam is god damn magical. It would be hard to replicate that kind of decentralized monolith without considerably more work with any other technology.

I mean consider a gen server mounted in your supervisor tree. its 5 min of work; tops. Doing the same with kubernetes would require coordinating a message broker, picking a client library, creatig a restart strategy and networking. all of which would add considerably to your development time

Reply on news.ycombinator.com