I've been merging microservices back into the monolith | Hacker News Reader

I've been merging microservices back into the monolith

bennadel.com

282 pointsrmason196 comments

It's not about microservices or not, it's about domain boundaries, application architecture, coupling. Whether it runs in a service or in a monolith isn't the deciding factor of wether its good or bad.

Architecture and design is. A 100 services app with 5000 lines of code each could be really badly designed and entangled, and a monolith with 500.0000 lines of code could be very well designed...

It's about code quality, right abstractions, decoupling, cohesion. Correct domain modelling Etc.

Microservices are just a tool, but you have to have a good design first.

Yeah. One thing I've noticed is that people are really bad at working towards high-level, nebulous goals like "code quality, right abstractions, decoupling, cohesion". Keeping these goals in mind requires constant thought, analysis and decision making as each case is different and there's no universal solution.

So people jump at simple, single-faceted goals like "normalise the database", "spin out microservices", "don't use HTML tables", "rewrite using Rust" etc. These involve no further thought, no continuous evaluation of nebulous targets like quality, no keeping up with moving goals.

It explains quite a lot of problems with politics too. Choose from red or blue. No further thinking required. If even that's too difficult, just pick the one your parents/friends picked. Easy!

Yea.

Once you have good domain boundaries you can use microservices to make the split more visible for less thoughtful devs and use the associated tooling (contract testing, ... ) to reduce operational coupling around deployments and tests.

But aside from that, anything you can do with microservices you can also do in monoliths.

Some people think that if you start creating micro services you will magically just "get" those boundaries.

and magically get the problem of distributed transactions. :)

I am yet to see a single monolith that doesn't deal with exactly the same issue itself.

Aren't monoliths by definition not distributed?

monolithic source code can be deployed across multiple servers.

A true monolith is like a true function - it's absolutely useless because everything it does is pure and therefore it can't interact with reality at all.

In less metaphorical terms: you still need to sync external APIs, sync filesystem with the database, deploy to S3 as you don't want to saturate your web workers, or have conflicts in your parallel red-blue deployments.

If your monolith doesn't do anything I had just described then you probably aren't doing anything exciting anyway so it's quite unlikely that you've got a lot to say about software architecture.

Oh, you don't know how true this is.

> But aside from that, anything you can do with microservices you can also do in monoliths.

Microservices can give you some things that a modular monolith can't provide: independent deployments, different tech stacks, independent scaling and better resiliency.

I'd be interested to hear about the sorts of systems that you've worked on and how anything you can do with microservices you can do with a monolith. Some problems that I've had to solve in recent years:

- Urgently deploying a fix in one part of the system without having to deploy the _whole_ system. - Updating the framework or the language (my favourite: updating from Java 8 to Java 11, which broke a lot of things and which we would have hated to have to do all in one go, if we'd had a monolith). - Scaling different parts of the system independently (both in terms of running different services on different numbers of nodes and in terms of using different hardware for application nodes and DBs). - Load testing different parts of the system independently. - Trialling a new language or framework in a small part of the system without affecting the rest of the system.

I personally wasn't aware of how much of a big deal some of these problems would have been until our system grew to the size that it is now, with a dozen or so development teams working on at least as many services. I personally have no idea how you could solve these problems on a system this size without splitting it into independently deployable services, without incurring a massive cost.

Indeed, if the team is not able to write modular code, it is not putting a network in the middle that it will sort it out, the outcome is just spaghetti network calls.

I always found the "microservices make things simpler" argument to be suspicious. At the end of the day it is just a monolith with unreliable network connections between components. Each microservice may be less complex, but you have just pushed complexity from the application layer to the devops layer. In a well written monlith the components shouldn't need to be any more complex than a microservice component.

thats because the main advantages of microservices have nothing to do with writing code.

as an example, the PathofExile.com website goes down every time the game servers restart because they share infrastructure. This should be avoided.

But you can avoid this quite easily without resorting to microservices.

Someone forgot to use a load balancer with red/blue deployments.

From a technical point of view, we're on the same page. But then there is the reality of modern software development.

A few years ago, the new CTO of a Union Square Ventures' investment tapped me as technical advisor to review this startups efforts to expand their software to new verticals. He came in as part of a reshuffle. He had brought his Director of Engineering from his previous (well known) company, and there was even a Star CEO (which is the reason I signed up!) with a monumentally impressive CV. As compensation, I was billing lawyer hourly rates.

After two months of reviewing their existing architecture, and picking the brains of the dev, support, and sales team the new verticals, I came with an architectural solution to their immediate and long-term needs. I gave them exactly what they said they wanted.

What then happened is that, for one, the Star CEO turned out to be an echo chamber of "growth growth growth" and entirely disappointing as a leader of a software company. And this was a SV veteran from both HW and SW side of SV. Huge disappointment.

The CTO, who was entirely clueless, apparently gave in to back channel efforts of his DoE to disregard my work product. A few days later I walk into an all hands meeting where the "next generation" was announced to be "a ball of mud" of "microservices". This same CTO later took me aside and basically offered to give me hush money ("will you be my personal advisor?") to keep quiet, while getting the same (insanely high) hourly rate for the remainder of my contract.

So this is the reality of modern software development. Architecture is simply not valued at extremities of tech orgnizations: the bottom ranks are young developers who simply don't know enough to appreciate thoughtful design. And the management is facing incentives that simply do NOT align with thoughtful design and development.

And yet not a single name or identifying piece of information that people can take action on, so these borderline frauds will continue drawing huge paychecks and hampering development for years to come.

The CEO was not a fraud and given the genuine fully-established and world class credentials of his technical reports in previous engagements, entirely unlikely that he could have faked it with that crowd. No. It seems he had adjusted per prevailing and current norms of SV.

Beyond that clarification, I think it irresponsible to name names. This wasn't a case of "fraud" - Peter Principle ruled the day. Both CTO and DoE had in fact lead a successful very large scale internet product company, which is the reason they were tapped. It just happens that this domain was far more nuanced and complex than the domain of their previous victory tour.

Also, as expected, they (including the super enthusiastic junior engineers that kept pushing for "microservices") have all moved on.

This is just how the industry works.

I read an article recently basically decrying how junior devs (and some not so) are holding tech leaders to ransom unless they adopt the architecture of their choice - microservices being the current hot paradigm, and so they are capitulating in order to keep them, despite the fact that probably 99% of these devs are demanding something they will have zero experience with.

I definitely saw the same thing playing out at my previous company, and not being in the mood for a conflict over that have moved on.

In short: DDD - Domain Driven Design

That's also the most useful thing i learned with doing microservices at work.

Note many good articles/videos about practical domain boundaries/designing AggregateRoot though. Does anyone have any tips? I feel that i'm lacking somewhat in designing them.

This is something that is totally underappreciated in software. I guess it's difficult to do and takes experience to get right. And probably more difficult to explain how to do to others.

Having worked on enough crap code in the past (and right now) my primary thoughts are to try and make things understandable for the next dev that needs to pick it up after me.

>it's about domain boundaries, application architecture, coupling.

Thank you! As a service consumer, if I need to call the same 3 other services for EVERY OPERATION, then we've achieved microservice hell.

>monolith with 500.0000 lines of code could be very well designed... As someone who was part of a "de-monolith" effort, the first step of our process was to create internal service facades that grouped things according to the logical domains that aligned well with our required functionality, data storage, and availability requirements.

Once those were functional, decoupled, and comprehensible, we actually had an easy to maintain, clean monolithic code base that exposed a small handful of services that were de/loosely coupled and conformed to workflows we need to support via web services.

A real issue with microservices is that you can no longer use the database system for transactional integrity. Microservices are fine for things you never have to back out. But if something can go wrong that requires a clean cancel of an entire transaction, now you need all the logic for that in the microservice calls. Which is very hard to get right for the cases where either side fails.

(Today's interaction with phone support: Convincing Smart and Final that when their third-party delivery company had their systems go down, they didn't actually deliver the order, even though they generated a receipt for it.)

Yeah, and while things like the Saga pattern[1] is interesting, it hasn't impressed me with its elegance or cleverness :[ (also, there seems to many real life cases where it would not work)

1: https://www.youtube.com/watch?v=xDuwrtwYHu8

I personally don't like saga pattern. For example when some operation fails, it should be reverted ob every service it went through. But what to do, when that revert fails? It's just matter of time when the system ends up in inconsistent state.

Is there a better option? And what is it? (genuinely curious, I've never had to deal with scale beyond where a single RDBMS could handle atomicity for me, and I'm curious about the best practice for handling this in bigger systems)

Two phase commit is the traditional answer. Only extreme "enterprise" environments support it like JTA transaction managers in Java land.

But 2PC is slow.

The modern answer is to move scalability to your DB layer and have X instances of your monolith. Some NewSQL databases like CockroachDB and TiDB are reasonable choices as long as you avoid complex SQL

The modern answer? Really?

I'm sorry but this is exactly what tons of "old" businesses have done for ages. But they are being told over and over that that isn't "web scale".

In my old life we just scaled via the DB and were able to deploy as many copies of our monolithic services as we needed to scale. We had a few monolithic services (no, not microservices, really nothing micro about them at all) that even shared some core code through a library.

In my new life that is no longer wanted and we need microservices and multiple databases. Except that it's all just logically split and in reality ends up in the same DB instance. But you know, we could scale it to a new DB instance if we ever needed to. Not that we will if you ask me because we run maaaany customers on the same database instance too. So it's all fake and we could've just used the same old boring working fine approach as any old business. Go figure.

Yeah you're right to an extent. But in the past you could hit a point where one DB could not handle the traffic. There's tons of businesses out there doing one DB instance per customer because of this.

With Cockroach and TiDB you can host unlimited traffic on the same db as long as you're careful about queries. I guess you could with Mongo or whatever too if you're okay with data corruption

In my experience being careful with the queries is harder and/or more expensive. Developer time is either very expensive or you just don't have the right developers.

It also depends a lot what kind of money you can throw around. Are you a big fat insurance company? Music business? Bank? You are just gonna throw money at Oracle and your hardware vendor and its gonna scale (thinking 10-ish years back into my past here as an example). I think we came from like 4GB of RAM on the DB and a couple of CPUs with single path i/o.

Could have spent ages optimizing all the different workloads. And this was already using read only replicas heavily for around the globe acceleration. This internal app was used 24/7 from different places in the world tho heaviest usage was Europe/US timezone. Instead they got 128GB of RAM, 16 cores IIRC and 4-path i/o (still spinning disk w/ SCSI at the time). Sure it cost a lot I'm sure. But I doubt it cost more than a year's salary of _one_ of us developers. And optimization would've required the application developers, the backend batch job developers and the (shared resource) DB admins to work on a multitude of workloads and use cases to analyze and optimize.

FF to my current job with lots and lots of but mainly smaller customers. One DB instance per per customer would be a lot of overhead. We do one schema per customer on PostgreSQL right now with sharding.

>> A real issue with microservices is that you can no longer use the database system for transactional integrity.

> Is there a better option? And what is it?

If you truly have a distributed system, then you have to pick from a number of not-ideal options.

But, there are certainly many systems where the developers and architects have a choice as to whether to create a microservices or a monolith. If you're facing that sort of choice, the "better option" (in terms of being a solution to distributed transactions) is to use a monolith, and not have to deal with distributed transactions at all. You get the facility to just rollback the entire transaction.

Disclosure: It's only my opinion, that I developed through trial and error during building ecommerce applications in a small team. Problems that I solved may be different than problems you will encounter.

I aim for the system that will fix itself. So for example, when new product comes to product service, product service stores the product in a strict consistent manner without relying on other services. Other services, which want to know about that new product, polls the product service in regular interval and gets new products. Also you need to make endpoints idempotent.

The good thing is, that when something bad happens, the system data will eventually converge. Sometimes there will be bugs, so you will fix the bug and then wait, until system fix its data. The bad thing is, that the system is only eventually consistent. You will need to track the delay and keep it short.

As adrianmsmith mentioned, its usually better to create monolith when possible.

I hope it was helpful.

Jimmy Bogard's talk "Six Little Lines of Fail"[0] is an interesting case study of all the ways things get so much more complicated when you can't just roll back a transaction if something goes wrong.

[0]: https://youtu.be/VvUdvte1V3s

Monolithic software maintained in a single computer/repository is so much better than software spread across many computers and repositories. There are some first-order arguments against a monolithic codebase, but the higher-order points will win out every time. I.e. the reason you want a separate codebase for each "concern" is probably because no one figured out how to work together as a team toward a common objective. This lack of teamwork will ultimately kill your project if you do not resolve it, so whether or not you do microservices in the expected way (i.e. HTTP RPC) will make no difference either way.

If you still want "micro" services because it actually makes technological sense, you can build these at a lower level of abstraction under some /Services folder. Many languages have namespaces, which allow for boundless organizational schemes. Only for a lack of imagination would you fail to organize even the most complex of applications in this manner.

Out of curiosity, what approaches do you use to manage zero-downtime deployments during system operation with a monolith?

Not parent, but how about a load ballancer and VM's? =) Or, if you have 1 machine I believe you can do things in Linux / BSD to start a new binary without killing ongoing connections to the old process.

Edit: Oh, or you know, just don't? Maybe no-one will notice 5 secs of downtime :P depends ..

You can use SO_REUSEADDR and SO_REUSEPORT to bind multiple instances to the same port. This means you will be using the Linux network stack as load balancer though.

Even if you have only 1 machine, you could deploy a reverse proxy in front of it (think nginx, haproxy and co). Then you can start your application on two different endpoints, and instruct your load balancer to go from one to the other. Nginx supports live reloading of configuration, I think that should work.

The same you would use for a micro services setup except their is just one of them? Eg, AWS ECS is very straight forward to get this working for a monolithic setup. In the end you need the same patterns, health checks, load balancer, draining, etc regardless of monolith vs individual micro service.

It would probably help if you explained why you think having a monolith has any bearing on possibility of zero-downtime deployments in a first place.

It might not necessarily, and I was genuinely wondering what patterns people use, but there are a few things about our monolith in particular that have seemed to make it harder for us:

- The monolith has to do all the work of the whole system when it starts, so start-up time is a lot longer, which makes rolling deployments much slower

- The monolith is much heavier in resource requirements, so it's much more expensive to spin up multiple of them

- The monolith has a much larger surface area, so post-deployment but pre-release verification is much more complex

- The migration path for a smaller service to support zero-downtime is simpler than for a larger service, so on average it's probably easier to get it working for an existing 'microservice' (although may be easier for the monolith than an equivalent full set of microservices)

Not a real issue at all in my experience. Businesses all over have done this for decades.

If you have many many microservices what is the combined resource usage vs. the monolith? Probably about the same and microservices may use more resources actually if we are talking anything with a runtime like Java.

Migrations in my experience mean migration of data so its about the volume of the data you need to change and not about the code operating against that data. Whether you have your monolith or a bunch of microservices waiting for the conversion job does not matter. It also does not matter whether your monolith or the microservices have to be able to read the data at any given time. With a quick starting microservice you may be able to say "eff it a transaction might fail but the new service will be up quick and the user can redo it" but that's not what you really want in big mission critical workloads. You really want to wait until all transactions on the old nodes finish before shutting them down. So your load balancer will route all traffic to the old nodes until new nodes are up, then shift traffic over and you kill your old nodes when all transactions have finished. Then you deploy those nodes and bring them back in. You have reduced redundancy and less capacity to handle load if we are talking old real iron businesses but in current cloud environments you just start up new instances and literally just kill old nodes after you've let the transactions finish. So you could even deploy during high load times.

I'm not talking hours of deployment here. Monolith doesn't mean you have to have all the cruft and startup times that say a jboss server with EJBs. You can have a monolith that starts up in a reasonable amount of time if you make the right choices for how you build your monolith.

Post deployment verification is slower why? If you do this manually why does it matter whether your changed customer flows are spread across 6 microservices that you just deployed new versions of or your monolith?

Also monolith doesn't necessarily mean you only have exactly one service. At my last place we had multiple services for various parts of the overall system but they were all monoliths themselves and shared some code through a library as well. But the services definitely weren't micro and did lots of things that weren't that related in the end. Could've split it up into many microservices.

There are many options.

For us, the approach is to use multiple instances of the monolith deployed simultaneously (listening on different ports) and to move traffic using a purpose-built software load balancer that is part of the same solution stack.

The load balancer software is an extremely simple concoction based upon the exact same primitives as our main application (AspNetCore). The only intelligence is in looking at request trace ids to determine old vs new routing.

A deployment can be resolved within a few seconds using this approach, even under heavy load.

We use a single VM per customer environment and are able to effectively realize zero downtime during business hours. We are granted maintenance windows after every business day and over weekends, so it is a little bit easier than if we were managing a credit card transaction processing system or similar.

To be fair, our approach only works because of how integrated we have our entire vertical. We went all-in on writing our own way to build, deploy and manage our own software. The persistence mechanism was developed with all of this in mind.

We are moving beyond all of this deployment magic though. There is a bold new configuration-driven realm that we are entering into which quickly begins to obviate the need for frequent & disruptive software deployments in the first place.

The way heroku approaches this is by deploying the new version while the old version is running and then within a few minutes switch routing from the old version to the new version, and then shutting down the old version.

https://devcenter.heroku.com/articles/preboot

Blue/green deployment is often used. There are always two instances of the app running, with one of them handling requests.

You deploy to the other one, do a health check, warm it up and start routing to it.

No downtime.

"Monolith or Microservices" and "Monorepo vs Polyrepo" are best treated as two entirely separate engineering questions. You can have a Monolith built from code in many repos, or many microservices maintained in one big repo.

There are various pros+cons for whichever combination you choose, there's no "best" answer that applies to every single situation.

Seconded.

In a job we were mostly Serverless/Microservices, but tested and deployed as a Monolith. We got many benefits from the Monolith (simplified management, testing, code wrangling), and some benefits of Microservices (theoretically independent functions, scaling/costing down to zero).

Microservices tend to provide good isolation barriers in systems and those tend to increase stability of highly complex systems.

Though Erlang is a counterexample.

Having a compiler able to check your whole system at once, tends to increase stability of highly complex systems.

But microservices breaks that, as well as introduces many other consistency problems, plus huge complexity overhead, all of which are forces toward instability in highly complex systems.

I suggest that the important factors for stability are not really about microservices, and that depending on the situation microservices is usually a net negative for stability, for the reasons mentioned above.

This isn’t true. You can run all your micro services together and check them on a single computer. Micro services doesn’t break that.

So, no net negative.

As other people have pointed out. It’s a tool... you can have bad code in a monolith just as much as micro services.

Microservices does break compile-time type checking. Runtime correctness != compile-time correctness. This is all assuming you managed to convince everyone to use the same language and throw everything under 1 solution too.

I absolutely love that our application takes just a little bit longer to build than a smaller isolated microservice. This is when my computer is doing all of the hard troubleshooting work for me ahead of time. 32+ cores diligently seeking out my wasted time and banishing bugs before they can even make it into QA.

Determining that your types don't line up during integration testing seems like a foolish approach if you value your time.

No it doesn’t.

Typescript/GraphQL both have strong type checking. I think you need to be more familiar with the available tech before spreading FUD. You can even compose the micro services into a monolith if you want everything to just run. It’s a tool not a religion.

Yes I’m assuming conventions for services, just as in a monolith I’d assume standards for design and composition (usually an overriding architecture like MVC or some such).

I am very familiar with the tech. You cannot move the goal posts around regarding the nature of your application architecture and then claim that I am spreading FUD as a result. Strong type checking across the scope of the entire problem domain is the guarantee that I am ultimately looking for. Only if you compile your whole application into a single binary output do you get these guarantees.

I do not care that each microservice independently passes type checks. This is hardly a compelling argument if we are talking about delivering working products. The most critical points of verification in a system with multiple services are where all of these pieces connect back together.

The interaction between the services (connection points) would also pass type checks. (Because they are typed as well)

... so that’s the whole problem domain.

No move of goal post. And no, like I said with things like typescript (And flow or Sorbet) you don’t need to compile any binary at all let a lone a single one.

Also: a monolith does not refer to a single binary it refers to a single application code base. (You can have monoliths in Ruby/Python/JavaScript/php/etc where no binary is produced)

I think you are misunderstanding what I’m saying, because I’m basically saying the same thing again:

- service A consumes service B - they are both type checked (as you readily admit) - the call site of service B in service A is type checked as well. - there are 0 points in the code or interactions that are not type checked.

I have worked at several companies with SOA. In fact, I was lead at one of the companies where we were breaking monolith to smaller services. We were having lots of issues with scalability with monolith. First we tried breaking to scale it horizontally by creating shards and routing users to different shards. That helped but with the growth we were seeing, we were back to scaling issue in 14 months. We broke it further into 4 shards and started working on SOA. After year and half we had dozens of smaller services and scaling was really smaller problem as it boiled down to scaling specific service. Over all, few points to add for without regards to SOA that I didn’t see here in threads -

- smaller blast radius: every change is small and specific to service so easy to rollback and understand the impact

- load tests: capacity management was relatively easy; small services with small dependencies

- easier to update dependencies: java version updates was not huge project with every feature development on hold

- autonomy: team had more autonomy as it didn’t require committee approvals

- prolific design patterns: services could use different architectural patterns

This obviously came with lot of other issues - latencies, cross service atomocity, logs correlation. But at the end I believe pros outweigh the cons and I would continue to use SOA pattern wherever I could.

Industry has been trending towards microservices/lambdas which in my opinion take it too far. Finding that balance between Monolith and micro service is what works in my opinion.

In your opinion what is the difference between an SOA and microservice architecture? When does a microservice become just an SOA service or vice versa?

I think the biggest difference is as defined by Martin Fowler:

" When building communication structures between different processes, we've seen many products and approaches that stress putting significant smarts into the communication mechanism itself. A good example of this is the Enterprise Service Bus (ESB), where ESB products often include sophisticated facilities for message routing, choreography, transformation, and applying business rules.

The microservice community favours an alternative approach: smart endpoints and dumb pipes. Applications built from microservices aim to be as decoupled and as cohesive as possible - they own their own domain logic and act more as filters in the classical Unix sense - receiving a request, applying logic as appropriate and producing a response. These are choreographed using simple RESTish protocols rather than complex protocols such as WS-Choreography or BPEL or orchestration by a central tool."

https://martinfowler.com/articles/microservices.html#SmartEn...

Look this is a great answer, and I love Martin Fowler. That's one rational explanation. It may however be a rationalised explanation that misses out on the difference that makes a difference to understanding what's really going on.

So here's another perspective, that of a software anthropologist .. .

Common patterns of networked systems were developed over decades, and as it developed over decades it built various advanced features on top of the core network call/response features - service busses - routing - useful abstractions like decoupling services from host processes - propogating security contexts through the chains of service calls that can develop when these systems evolve - strong typing, which enabled layers of tooling to replace manually doing jobs with these systems and various advanced features.

That was called SOA, because it looks at software architecture as a set of services.

Then along came a bunch of programmers that weren't very studious, started from scratch just doing basic message passing without all the advanced features that had been built up by the decades of previous hard won lessons. That was called microservices. Note the removal of the 'architecture' from microservices!

slightly /s

I work at a company that uses a hybrid monolith/service approach to serve millions and millions and millions of users. Unless you work for one of the top-top-top social networks we probably serve many more users that your systems do.

And even our existing services are not "micro": they are relatively large, and were mostly extracted to handle tasks that needed a different language.

This massive code base is a very good example of how most people can do just fine with a monolith. I even learned to appreciate this uniform approach to things: monorepo, unified ci/cd process, shared responsibility.

I don't think the number of users has as much to do with partitioning a system as the variety of required tasks does.

It would appear to make sense to separate two things

- a service as isolated business logic with clean requirements process, ownership, SLAs, interfaces, testing, and build infra for maintenance

- a service as an endpoint you can access from most any environment (other services in whatever language, web apps)

The trick is to keep these two things apart and assign services to physical/virtual nodes/pods/whatever as late as possible rather than making deployment decisions through choosing implementation techniques. Eg it's not reasonable to expect scalability by deploying individual services to a large number of nodes with excessive granularity of services; having the option to deploy a called service on the same host as the calling service to have essentially no network overhead might make more sense. It's also not reasonable to attempt to scale out services to a large number of nodes when your bottleneck is an RDBMS or other storage.

This was already very clear with 2nd gen SOA architectures like SCA (service component architecture) around 2007 or so, with options for binding implementations to remote protocols (SOAP) or locally via procedure calls, or both at the same time. This separation is notably absent from microservice architectures which always want to produce a pod or vm image as result artifact.

Now SOAP (and SCA and other SOA frameworks) also allowed transactions and auth context propagation; something that isn't even on the radar of microservice-like approaches. The (many) ones I saw at customers at least only naively implemented the happy path, not allowing for two-phase commit or at least compensation services to be called on abort by an aggregating service.

You are so right here. Clearly you worked with enterprise distributed systems and learned the industry body of knowledge for that - things like configuration driven binding 'services' to 'host processes', distributed transactions or compensation, auth propogation, and advanced tooling for distributed development enabled by strong typing.

Alas most developers weren't aware of all these things that SOAP delivered.

Programming's Eternal September of the twenty first century has thrown the baby out with the bathwater - those unwashed masses didn't understand why SOAP was a wee bit complex, the very good reasons!

They RESTed instead of scrubbing up on their studying. They twittered and twattered instead of studying. In their multitudes they cast a POX of JSON on our world!

They weren't the strong silent type to refactor, to compensate and rollback their mistakes .. and that is why they renamed SOA to micro services .. to make the huge mess sound smaller than it really is ..

okay okay I will get off my soapbox now

Don't forget POJSON services lol!

> - prolific design patterns: services could use different architectural patterns

Very true.

But there is still a lot of architecture that crosses microservice boundaries: How the domain is split, the dependency hierarchy (or lack thereof) between services, data lifecycles, ...

It's important not to forget that.

This is a great article - One thing I always try and vouch for is that you don't need to go "all-in" on microservices. There is nothing wrong with having a main monolithic application with a few parts separated out into microservices that are necessary for performance or team management.

The author hits the nail on the head at the end:

  If I could go back and redo our early microservice attempts, I would 100% start by focusing on all the "CPU bound" functionality first: image processing and resizing, thumbnail generation, PDF exporting, PDF importing, file versioning with rdiff, ZIP archive generation. I would have broken teams out along those boundaries, and have them create "pure" services that dealt with nothing but Inputs and Outputs (ie, no "integration databases", no "shared file systems") such that every other service could consume them while maintaining loose-coupling.

> One thing I always try and vouch for is that you do need to go "all-in" on microservices

Did you mean "don't need"? I agree with the rest of your post, but this first sentence confuses me

You are right! Sorry typo fixed

yeah, but is that microservices? This is what we did the first place I worked, 15 years ago.

It was called modular design. I think the main difference with micro services is the accessibility on the network. I would like to be corrected if I am wrong.

Microservices are indepently deployable modules. Thus the same principles of modular design in a monolith apply. So, yes, you're correct with the network part.

My question was rhetorical, to be honest :) point being that the only new thing, if any, with microservices is that they should be micro and you should have a lot of them.

People have been running >1 services for a long time. Sometimes calling it SOA and sometimes calling it nothing, just doing it because it made sense.

Totally agree, I find a lot of tech is like fashion - leaving and returning again later with a new cooler name.

  just doing it because it made sense.

This ^^. Call it what you like - if it works, it works

I've done similar batch processing with Spring Batch before (Java).

In my experience, this will result in a ton of unneeded extra complexity. Microservices are very heavy and should be cut along domain boundaries and not technical boundaries so that they are as independent as possible and you reduce the need for synchronous communication. In the example above, I would probably use cloud functions or a distributed actor system.

This seems quite subjective - I have worked with services like these and would probably call that SOA where you need to split code concerns along domain boundaries.

In the example above, I would probably use cloud functions or a distributed actor system.

Perfect, a microservice (perhaps just service?) can be whatever you want it to be, and I don't see any reason why you shouldn't split code along technical boundaries if it somehow improves your software

"Future thinking is future trashing".

Though I've had mixed feelings about this quote, in the case of the author this is probably valid. Yes, microservices are not a bad idea. They are in fact a great idea and they are needed SOMETIMES. But there is a time and place for everything. They are unnecessary and completely counterproductive and cumbersome unless you have a very large and complex system which needs to be distributed. If that is the case, of course, god speed. But if you can work without them you should not go for a microservice architecture: let's face it, they are hard to design, develop and in many cases a nightmare to debug when something goes wrong. Annoyingly the term "microservice" is yet another PR campaign gone horribly wrong. As a consequence it has become a buzzword like blockchain, AI, agile, etc. Just because FAANG is doing it, does not mean that your online shop for selling socks needs microservices in any shape or form.

A short while ago a friend dragged me in as a side consultant for one of his clients. He owned a site which is basically craigslist for musical instruments and he was in the process of hiring a company to rebuild and subsequently modernize his site. Realistically the site has around 5k users/day tops and no more than a GB of traffc, and a mysql database which over the course of 15 years is less than 50GB. Basically a tiny site. The company he was about to hire had designed an architecture, which involved a large EKS cluster, 15 microservices, mysql, postgeres, elastic, memcache and redis, complicated grpc communication and their claims were that this architecture would make it the best in the business. OK, let's give them the benefit of a doubt, I advised him to ask them how much more performance would he gain out of that and if there are any other advantage over the typical vps with a webserver. Their response was "Infinitely more performant, you'll be able to scale to 100 million users per day and the system wouldn't feel a thing. That is what Google, Facebook and all other large companies are doing". Yeah... And does he expect to have 100 million users pour in at any point in time? Take a wild guess... Mind you, they asked for the same amount of money for a simple "buy now" website.

The bit about Conway's Law

>Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure.

is mind blowing. How did I not notice this the whole time?

Awesome article. <3

A lot of the job of effective large organizational software/management consultants is understanding where there are mismatches between org structure and software architecture and helping to drive changes to get those in alignment. A big part of this is finding better places to put the interfaces (whether they are human or API), because they're usually holdovers from the past and no longer related to the business structure or architecture.

It's colloquially described as "shipping the org chart". Which I always liked as a way to describe it to a wider audience.

You can use this to change the architecture of your software: change the structure of your teams, and your software architecture will follow.

Here is a post by Martin Fowler James Lewis in 2014 pointing out the same thing [0]. A lot of the thoughts expressed there I think are still relevant but I guess we had to wait for this hype cycle to play out?

[0] https://martinfowler.com/articles/microservices.html#Organiz...

note the law doesn't specify if this is by necessity/on purpose - but designing a system which doesn't do that is usually setting yourself up for failure.

It takes one to "experience" this phenomenon from the inside of an org to appreciate it. Analogous to how one would have to work on largish codebase/architectures to appreciate design patterns.

I have a corollary to Conway's Law.

The number of network hops an end customer's request goes through before it is served is directly proportional to the number of teams in that org.

A native English speaker could help me simplify that sentence :-)

Oh the amount of time our industry wastes behind technology for the sake of technology! First, let's write tons of articles extolling the virtues of microservices. Then let's counter that with tons extolling the virtues of using microservices the right way. Hype followed by wisdom and then go back to hype again! When are we going to mature! Boring is good and predictable.

Evolution through natural selection depends on mutation. If we stopped all mutation, we would stop evolution too. We would be stuck in local optima, never trying to find a better global optimum.

Not every new practice will work out. The bad ones will be weeded out by natural selection. But if we try to stop evolution by discouraging mutation, we will stagnate.

This century has been programming's Eternal September.

So many programmers are too new to understand the huge amount of context for our industry, and it will probably not improve while the number of programmers continues in hypergrowth mode.

Another way to think of it is that half the industry is still in the stage of not even knowing what they don't know.

Huge amount of context of our industry!? All 60 years is it! How do we manage!

(Meanwhile over in the 1000s of years old industries...)

It's insane to me that for a few years so many organizations were somehow convinced that MONSTROSITIES like trying to do distributed transactions using things like 2 phase commit just to make the almighty microservices god happy was an acceptable way of doing things.

The first law of Microservice architecture should have been- do the microservice boundaries you have defined result in the need for cross service transactions? If so, slam on those breaks hard.

The author correctly identifies the problem as part technical, part people. My shot-in-the-dark estimation is probably 70% of teams doing microservices do it to solve the people problem before (if ever) they need it to solve the technical problem.

Technical solutions are _usually_ bad solutions for people problems, and architectural patterns are probably even worse solutions at solving the problem of human collaboration. It doesn't help that microservices are mostly a better-sounding name for "SOA, but smaller", that has grown in prominence mostly to sell you hosting for your very many microservices. Microservices takes one of the hardest and most important parts of of SOA (service boundaries) and replaces it with...smaller.

Glad to see someone at a larger company publishing about this.

We used services to great effect at the company I previously worked at. There was not a single engineer that ever voiced a concern over how things worked. A couple important notes:

* We used a mono repo and services only had to talk to other services from the same commit (or branch). We didn't version service APIs, I think that would have been a huge waste of time for us.

* Our tests did unit tests and integration tests.

* We could use different languages as the problems required. We started one data processing service in Python and migrated it to Go when it became more CPU bound than we were comfortable with. We have an old but extremely reliable C++ service that had been running for almost a decade.

* Each service ran in its own container. When a container/vm would run out of memory, we all didn't have to scramble to figure out who broke something. The person who owned the code for that service dealt with it. If a service started kicking out errors nobody but the owner had to be distracted.

* We had a good team of architects that understood how to build things. We didn't split services across atomic boundaries. We knew how to build interfaces that handled errors robustly. If you don't know how to do these things, you service APIs can quickly become a mess.

* The world is services. Just think about AWS and how it works. There is no such thing as a monolith, eventually you interface to services. Be it a weather API, a CI server, deployment system, a central logger, etc. Just because you didn't write all those services doesn't mean you don't have a service architecture.

* We had really disjoint services that would have made zero sense to put together. For example, the web server that handled the front-end vs a back-end data system that refreshed a product database and built PDF assets based on the data in overnight batch runs. We actually had many of the later for different customers. We had services for central logging, for monitoring, for metric storage, etc. No way would I ever want to push all that into a single service and debug a GC issue or segfault.

* If we had to hotfix a service, we could deploy just that service with zero concern the other services would have an issue.

For us, services were a key part of building a very reliable system that we could update rapidly with minimal risk. We did this for almost a decade. We had a very good group of engineers and never did I hear one of them say services were holding us back. After this experience, I would say everyone one of them is an advocate for the appropriate use services.

> * We used a mono repo and services only had to talk to other services from the same commit (or branch)

Huh, that was a new one, can you expand? How did that work in practice? Did all the services update to the latest commit automatically?

We had a weekly deploy (plus any hotfixes) that built the images for all the new VMs (we used KVM) and then when done, stopped all the services and restarted them with their new images. Total down time was about 15 sec which for our application was acceptable. The build time was a lot longer than that (3-4 minutes), but that happened before the existing services were stopped so it didn't affect downtime. If any of the services failed to build, the deploy was aborted and the existing services were left running uninterrupted.

The deploy tool was home grown and was part of the repo. Having everything in a mono repo was really important because it dramatically reduced the number of permutations of interfaces that had to be tested down to one.

These trade-offs might not be right for everyone, but for us they were fine and allowed us to create a highly efficient development environment.

Thanks. So by deplyoing them all at once, as one unit, you avoid having to deal with the huge pain of backwards/forwards compatibility.

But what would happen if 1 of them failed to start? Or start, but immediately fail its tasks? Then you would have to revert all of them, at the same time, after some has already run for a short while, potentially writing data on the new format (for example). So I guess you can't avoid the problem with backwards compatibility completely..

Or, I guess, more realistically, you wouldl try to fix the failing service fast, without rolling back anything :)

Depends. I ran into this myself a couple times over the years and dealt with it depending on what changed. What was really nice was the weekly deploy. All commits were fresh in my head so there were not any changes from long ago that I forgot about. That was actually really important. On some holidays we would not do a deploy and even two weeks of changes was a lot more to remember than one week.

So, if I knew the changes well, I could roll back if nothing in the API changed. We didn't like that, but sometimes you just need to be pragmatic. Other times an update was made that forced you to get it working. In this case services helped because rather than the whole system being down, only part of it was. So many of our services were background type work that in most cases, customers would never even know if they were down for even 1-2hrs.

We had good staging environments with a lot of reporting, monitoring and metrics. We never deployed to production code that had not run reliably in staging for a least a couple of days.

> * Each service ran in its own container. When a container/vm would run out of memory, we all didn't have to scramble to figure out who broke something. The person who owned the code for that service dealt with it. If a service started kicking out errors nobody but the owner had to be distracted.

Separate containers for multiple services owned by the same team sounds like a lot of overhead. If I want to work on something do I have to spin up a bunch of containers, with all the overhead that brings? Can I easily use a debugger across all of the code that implements a given feature?

You're talking like there was a single "owner" for each service? What did you do when they went on holiday / left? How did people share knowledge about the system?

> * The world is services. Just think about AWS and how it works. There is no such thing as a monolith, eventually you interface to services. Be it a weather API, a CI server, deployment system, a central logger, etc. Just because you didn't write all those services doesn't mean you don't have a service architecture.

Dealing with an outside service brings a lot of overhead. It's harder to debug, harder to test. When things are maintained by a separate team it makes sense to treat them as an outside service, but within the same team why would you make all that extra work for yourself?

> We had services for central logging, for monitoring, for metric storage, etc. No way would I ever want to push all that into a single service and debug a GC issue or segfault.

Sounds like you need a better language. I agree that anything that might segfault belongs in a separate service, but most of the time the solution is to not use stuff that might segfault.

> * If we had to hotfix a service, we could deploy just that service with zero concern the other services would have an issue.

How does that fit with "services only had to talk to other services from the same commit"?

> Separate containers for multiple services owned by the same team sounds like a lot of overhead. If I want to work on something do I have to spin up a bunch of containers, with all the overhead that brings? Can I easily use a debugger across all of the code that implements a given feature?

Most of the services just ran normally on a developers system, or we had a docker script that could spin things up locally, or you used one of the development sandboxes running on the system.

BTW, we had a system that allowed very low cost creation and management of containers (VMs in our case). I think that is an important detail.

Could you use a debugger for a feature? Most likely not, because a lot of our features already pulled data from external sources which you had no visibility into.

> You're talking like there was a single "owner" for each service? What did you do when they went on holiday / left? How did people share knowledge about the system?

Same problem every development team has. Who setup up the CI server? Who setup the front-end proxy? Who configured the AWS setup? Everything needs redundancy. We had a list of all tools and services and had multiple names assigned to each one. Even without services a team should have this.

> Dealing with an outside service brings a lot of overhead. It's harder to debug, harder to test. When things are maintained by a separate team it makes sense to treat them as an outside service, but within the same team why would you make all that extra work for yourself?

Because it worked really well and we didn't see that extra work. I disagree that external services are hard to interface to. They are hard to work with when they are buggy, but if they hit the specs, they generally are not an issue. I have no problem interfacing to S3. Because of the loose coupling, things were very extensible and easy to modify. A few times we would replace a service with a new one by adding the new one and slowly migrating everything over to use it. In some cases, that migration took over a year, but that was fine.

> Sounds like you need a better language. I agree that anything that might segfault belongs in a separate service, but most of the time the solution is to not use stuff that might segfault.

LOL, it was Python. Bugs happen, things break. Everything can eventually segfault for all kinds of reasons. Run a complex system for a decade and you will see all types of issues. I remember running OpenVZ before switching to KVM and occasionally we would even see kernel panics.

> How does that fit with "services only had to talk to other services from the same commit"?

You conveniently left out "or branch". Listen, I am not trying to sell anything, I don't care what you think. I am just sharing my teams experience. It was overwhelmingly positive using services to build a complex system with real revenue from Fortune 100 customers for almost a decade.

> Same problem every development team has. Who setup up the CI server? Who setup the front-end proxy? Who configured the AWS setup? Everything needs redundancy.

Sure, so all those things should be owned by a team rather than an individual. But that makes the whole "When a container/vm would run out of memory, we all didn't have to scramble to figure out who broke something. The person who owned the code for that service dealt with it. If a service started kicking out errors nobody but the owner had to be distracted." kind of pointless; the whole team should know about the service and be aware of any recent changes to it, and everyone should be able to debug it.

> I disagree that external services are hard to interface to. They are hard to work with when they are buggy, but if they hit the specs, they generally are not an issue.

Well of course programming is easy as long as there are no bugs! But the most important and hardest part of programming is debugging, and every time you hit a service boundary you have to figure out which side of it the bug lies on; that ends up being a significant amount of work. Practically speaking you need to implement a stub version of the service for testing with, monitoring that can confirm whether that service is up and running, maybe some kind of tracing/replay that lets you track specific problematic requests... and that's even more work for internal services, for something like S3 at least some of the bits and pieces for doing that have been built already.

> Because of the loose coupling, things were very extensible and easy to modify.

Absolutely, but you don't need a network boundary to achieve that. (I mean, maybe you do in Python because it has no visibility rules/enforcement and incredibly poor dependency management, but that's not a general problem).

> You conveniently left out "or branch".

Different commits on the same branch can be just as different as different branches. You can't have it both ways, either you keep all your services in lockstep which means you have to restart everything to fix a bug, or you have to worry about different services being on different versions. (Actually, since you normally can't restart a group of microservices atomically, you have to worry about version compatibility even if you try to always deploy new versions of everything together, IME).

> Listen, I am not trying to sell anything, I don't care what you think. I am just sharing my teams experience. It was overwhelmingly positive using services to build a complex system with real revenue from Fortune 100 customers for almost a decade.

Your post sounds like a salesperson/cultist with the "not a single engineer that ever voiced a concern over how things worked" emphasis. I've worked in multiple microservice-oriented environments including those that claimed success for microservices, and they've all either had the same problems I'm talking about, or actually not been talking about microservices but rather reasonably sized services (i.e. having teams - groups of ~10 people who worked together and had standups together - maintain one or two services each).

Then we had different experiences. I am not claiming you are wrong and I don't appreciate you trying to discount our experiences.

If you shared it as one person's experience, or even an overall team consensus, I'd say fair enough. "not a single engineer that ever voiced a concern over how things worked" comes across like you're saying no reasonable engineer could disagree with you.

Seems like microservices has moved past the "peak of inflated expectations" and into the "Trough of Disillusionment" phase of the hype lifecycle.

Thank god. I'm tired of semi-technical product managers throwing around the term.

I agree this is a good thing. Now that knowledge just needs to be accelerated up the corporate ladder to CTO's/CEO's.

> Seems like microservices has moved past the "peak of inflated expectations" and into the "Trough of Disillusionment" phase of the hype lifecycle.

The only change I've been witnessing with regards to microservices is where their critics place their personal goalpost.

Microservices is a buzzword that is used as synonym for distributed systems and the evolution of service-oriented architectures after removing the constraints of rigid interface-related XML-based technologies like UDDI and WISDL in favour of ad-hoc interfaces. Some responsibilities are moved to dedicated services when the operations side justifies it, and services are developed based on key design principles to reflect lessons that have been learned throughout the past couple of decades.

But even if the hype associated with a buzzword comes and goes, the concepts and its uses are still the same.

Unfortunately containers are still into the "peak of inflated expectations".

Glad to see a reality check post around the hype of Microservices.

This article absolutely nails it. Finally, recognition that microservices are a technical tool for solving a people and organizational problem. We need to understand that a lot of "new" technology paradigms (especially those coming down from FAANG or other large organizations) are often designed to solve the organizational problems of operating at scale, not to provide some kind of technical panacea we should all aspire to built towards.

People are godamn hopeless.

I just saw a guy trying to argue that writing functional-style Java is the "new" way to write Java, and that not using functional interfaces means you're writing "old" style code.

I don’t know about Java, but we’re seeing a shift to functional in the mobile space, where non functional code is definitely “old” code. In Swift with Combine and in Kotlin with their thing.

"Old" in what sense though? In the style/fashion sense?

In the “writing non-functional code in 2 years will be the same as writing Objective-C today” sense.

> "Old" in what sense though? In the style/fashion sense?

If you take the time to look into it, you'll discover that the functional style is only a change in the "fashion sense" to those who are totally oblivious to their advantages.

I get how the pedantic take on monads turns people away from the functional side of programming, but you'd be hard pressed to find anything to criticize how returning either/result monads from promises is not a huge improvement over the boilerplate-rich/pure OO approach to Java.

I feel guilty when I write a FOR statement in Rust.

Me too. Mostly because it’s a smell to me that I’m mutating some internal state. I try to avoid that as much as possible because it quickly becomes soup.

I feel guilty when my C# method isn’t immediately returning a LINQ expression.

That seems like an unnecessarily confrontational way to put it.

What I would say is that, from a 10km perspective, functional interfaces are just where the Interface Segregation Principle takes you when you really lean into it, and that dependency injection can be thought of as just a convenient way to do partial application in bulk, and that CQRS encourages you to create an ever-wider separation between your commands and your queries, and that, in general, once you get far along where clean object-oriented design wants to lead you, the whole functional vs OO debate often starts to feel like it's quibbling about syntax as much as anything else.

People are still using Java?

Yes Java is uqititous and it's not going away for 30 years at least. See Cobol. Java also has had significant improvements in the language so that there is less incentive to use other JVM languages like kotlin and it's fully interoperable with heartland code bases as far as I know

Forgive my ignorance: what's a "heartland" code base?

>People are still using Java?

C'mon, you've been around here long enough to know better than to be making such comments

https://insights.stackoverflow.com/survey/2020#most-popular-...

https://octoverse.github.com/

Whenever possible, yes.

Sane language. Fantastic tooling. Huge ecosystem. Backwards and forwards compatible like few other things.

The number of flaws that make Java inferior to other JVM languages is surprisingly small. By far the worst parts are excessively verbose "Beans", nullpointer by default, the JEE ecosystem (there are good alternatives) and the high RAM consumption (a JVM flaw). Honestly 80% of the value I derive from using a different JVM language come from avoiding NPEs, getters and setters. The remaining 20% can be very useful but they are certainly not essential nor can they be considered mandatory.

> People are still using Java?

Yes, pretty much the whole world is using Java. Some FAANGs use it as basically their default software stack, and at most have introduced some Kotlin along the way.

The world doesn't revolve around flavor of the month tech stacks.

It revolves around JavaScript I think. ;p

Javascript in the browser, Java everywhere else.

Sure, if node didn’t exist, you’d be close to right.

If you check GitHub JavaScript and typescript are outstripping Java.

Username checks out ..

Have you ever heard of search bubbles?

The contortions I see used day-to-day to write code in the latest/greatest pardigm du-jour is criminal... I routinely have to revisit "functional masterpieces" of "easy to grok" code, and completely re-write to minimize CPU and/or memory consumption.

> I just saw a guy trying to argue that writing functional-style Java is the "new" way to write Java, and that not using functional interfaces means you're writing "old" style code.

Sounds right to me. (Of course not having first-class functions was just as bad 20 years ago, but fewer people had realised it back then).

I would say 99% of all articles on this site and r/programming on reddit have someone say exactly what you have - that microservices are an organizational tool. At this point your view is not an unpopular one.

However, microservices can also serve actual technical purposes too. But the nuances of accepting two viewpoints often placed in juxtaposition to one another is not internet friendly. Microservices, service oriented architecture, whatever you call it depends on your level of cynicism i suppose (or optimism?).

I agree, I don't think my view is unusual or controversial, which is why it surprises me that the writers of the blog posts that get submitted to here or on r/programming rarely acknowledge both the technical and organizational aspects. Just yesterday there was the article "Back to the '70s with Serverless" which compared smalltalk and its fast feedback loops to the complexity of cloud and serverless computing and argued it was about good marketing.

So I will keep commenting that it isn't just about technical issues and it isn't just about organizational ones, and you must look at both to really understand what is going on.

But I probably did get a little too excited that a technical post acknowledged and made it an important point in their article.

> I would say 99% of all articles on this site and r/programming on reddit have someone say exactly what you have - that microservices are an organizational tool. At this point your view is not an unpopular one.

Microservices have been from the start an organizational tool, with reliability and the ability to scale horizontally to avoid constraints of vertical scaling trailing at second place.

The rationale is that the first bottleneck that is hit by a growing organization is developer's throughput (i.e., the ability to deploy bugfixes and features without hitting blockers due to multiple teams being affected by a changeset). After that point, a service has to grow a tad more until reaching a point where the requests bounded to a subset of features justifies peeling them out to independent services.

> ...microservices are a technical tool for solving a people and organizational problem

They're also available to solve technical problems. For instance, it's hard to run some python3.5 and some python3.9 in a python monolith. A monolith must choose one python version and cut over en masse at some point. Microservices allow more gradual adoption (in production, not just in test suites!) of low level technology changes.

Right, but you can also eg run both versions of python on different machines and switch between them at the load balancer layer. That also lets you eg quickly roll back the upgrade from a single area.

You can do all these things either way. The question is which way suits your tools and organisational structure.

> Right, but you can also eg run both versions of python on different machines and switch between them at the load balancer layer.

That sounds to me like you are complementing the downsides of running a monolith with the downsides of operating a distributed system, without any noticeable upside.

I should clarify; that technique doesn’t work if there are calls between systems on different versions unless they are tested on both.

I wouldn’t run a system where the two servers call one another.

This is a technique for reducing risk via toggles at the load balancer. The reduction is achieved by making it possible to deploy smaller changes to users, and by reducing the latency of rollback.

These are only useful if you have the ability to introspect the system state in production and confirm whether a new version is working correctly.

I still fail to see the point of your approach.

It sounds like you're referring to blue/green deployments without using it's name, for some reason, and while failing to present any case of why a monolith provides an advantage. You're just showing downsides without any upside as a tradeoff.

I mean, with microservices you can also do blue/green deployments, and they are not all-in like monoliths. Each deployment only covers a subset of features that can gradually be rolled in and out with the only risk of causing partial failures.

But if you have a lot of migration to do, you're keeping compatibility across the whole stack that entire time. If you have more atomic processes, the trivial ones get migrated trivially and aren't forced to carry technical debt for extended periods of time.

And there are worse scenarios than python upgrades. What happens if your processor or your OS reaches end of life? Do you keep running the whole stack on Windows XP without security patches until the last bit is ready?

If some parts are running xp exposed and unpatched, your organisation needs to fix that before tackling application architecture. The question doesn’t make sense; I might as well ask which galaxy I should colonise.

No technique or tool will fix your organisation.

Hopefully you fix compatibility with newer versions of windows years before XP goes end of life. A lot of people seem to have created these problems for themselves by buying into proprietary platforms that then get yanked from underneath them. You can generally choose stacks that don't have this problem (or not to nearly the same degree).

It is possible, and easy, to have two virtual environments with a different python version in the same project. I am doing this right now with a long-lived project, where a deploy script uses python 2 and everything else python 3.

Only in different processes.

That's not a big problem in a small org/app

They're the cure and the disease.

Without a way to keep the system heterogeneous, people have to accept that they are on the same upgrade treadmill as the rest of the people in the organization, which means they have to constantly invest time and energy into keeping up with things.

Containers or microservices mean that the upgrade happens fifteen times, not once, and each group has their own little passion play about why they should do it later instead of now. And in the time it takes to argue about 15 upgrades, 2 more versions have come out.

This is the same issue as with static linking. This debate is quite old and was once "decided" already in favor of dynamic linking...

Using dynamically linked libraries enforces to upgrade all dependencies in lock-step. But you need to fix bugs or security issues only once for your whole software distribution.

Solving the problem of needing to run two different technology stacks by running two different technology stacks is decades older than the concept of micro services.

I'm a huge monolith proponent. But that doesn't mean I think every application should be one service. I just believe you shouldn't create a second service until you have a good reason to like "we need to run on two different tech stacks", or "our team is getting hard to manage let's split it into two teams that work on 2 services".

Problems are typically older than solutions, yes.

And it's simple to concede that monoliths need to be split sometimes. It's much harder in practice to do so. Most monoliths do not have standards, let alone enforcement mechanisms, to ensure that the design of the monoliths leave open splitting later. The things that can make that process difficult (or impossible) are many and subtle, such as global state, in process locks, interpreter protected consistency guarantees, disregard for ABI implications, etc. Having different processes (in production!) does guarantee that a snarl of those problems will not prevent splitting things up later.

> such as global state, in process locks, interpreter protected consistency guarantees,

If you need to share state or you need consistency/locking/transactions across bounded contexts on one service it's trivial to implement. But this can be incredibly difficult to do correctly across multiple servers. No one has implemented a multi-phase commit over multiple services and said "dang that was easy" afterwards.

I think you missed part of this, from the article (emphasis mine):

> Microservices Solve >Both< Technical and People Problems

I'd love some additional examples of technical problems microservices solve that are unrelated to scaling the organization.

I've mainly come across the occasional one-off where a different language is significantly better suited to the problem at hand, so that has been broken out as its own service, and there's another one here in the comments about migrating a python application between versions (I could argue the semantics of microservice here, but I'll roll with it), but I have not come across many where a microservice was the "right" solution to a purely technical problem.

Examples:

1) Long-living connections. One part of your application offers large file downloads, so that your users sometimes take significant amount of time to download, and the error rate shall be low. Think people doing `curl -L https://github.com/.../some-commit-hash.zip` in CI. I'd guess this is not served by github's ruby monolith. Or you use websockets. Redeployments typically stop all running tcp connections. Splitting this part off allows you to redeploy the main application frequently without disrupting websockets or downloads.

2) Reduce startup time. For some languages/ecosystems, startup time is significant and scales with the size of the codebase. For example, a typical java enterpise app like keycloak has half a million lines of code and takes about 1 minute only for startup. This is even more relevant when running things in AWS Lambda or google cloud run, and can also be relevant for integration tests. Having several deployables each handling a subset of the functionality may give you better cold start times than having one that contains all code.

3) Resiliency toward resource exhaustion. If two functionalities are served in the same process or VM, and one of them

* has a memory leak

* uses the connection pool to the database inefficiently and thereby clogs it up

* has an endpoint that is overloaded by a misbehaving client

then the other functionality is also affected, which can be avoided by putting them into separate processes/containers/VMs. Splitting services allows to reduce the effort spent on resource hygiene and rate limiting.

4) Binary size. It is often convenient to compile static info or asset-like things into the application, think geoinformation on timezones. This bloats the binary, splitting that part off into an independent deployable can give you a much smaller binary for the main application.

1) Sure, that may be an example of a good microservice, but I'm not sure that is a requirement to follow a broad microservices architecture. This can also be handled via proxies/load balancers and rolling deployments of monoliths.

2) Agreed, for individual services. Maybe/maybe not for the entire system.

3) I think this is generally true, but if you have a service that others are dependent on, you can still have these problems. Auth services for example (that was a fun day at work).

4) Sure, but now you have two binaries.

Microservices are easier to understand and debug.

This should speak for itself. If you ship a service which is well defined and only handles a reasonable amount of functionality, reasoning about its runtime behavior, memory, CPU usage, IO, etc, becomes much simpler.

Microservices are easier to test, either through unit testing, load-testing, integration testing etc. If you push a new version of a microservice and you see it's using more memory than the previous iteration, it's usually trivial to work out which line of code caused the change.

Microservices are mentally easier to grok for engineers working on them. I've worked on monoliths that required months for engineers to get up to speed on working with them because they did SO much stuff and if we had a memory issue with our monolith it could take weeks to diagnose the issue and we had to have our best engineers look at those problems because the service had grown so big and complex over many years.

> Microservices are easier to understand and debug.

I think an individual microservice is easier to grok and debug, but I also think a system made of microservices is harder to understand and debug. Depending on scale this is a bit of a wash to me, YMMV.

I broke out all our code for displaying public content into a few different microservices so my sites could easily grab them, which allowed me to play with different frameworks and ways to display our content (blog posts and directory/references for biology). The new content microservices endpoints are also a public (though undocumented) API as an extra bonus.

Basically it gave me the flexibility to work faster, and generating the code (e.g. the REST endpoint or the Svelte/Sapper front-end) is also a lot faster, and I get to host them on Vercel for free

Oh and it also helps me to separate business logic code and other not-so-public data to somewhere else (Heroku instead of Vercel) and it just runs completely separately. That way I don't have to worry about accidentally sharing data

As a solo designer/engineer this Vercel + microservices / tiny APIs + tiny frontends makes my life much easier

My take on this (which I've not really thought through).

With O-O programming - the code itself is fine. You're breaking it up into smaller objects, each well defined, each with self-contained state. Nice and easy.

The problem is the behaviour of the system isn't defined in that code. Instead it's defined in the pattern of messages that are sent over time across different bits of code. And because it's temporal, that pattern isn't written into the actual code itself - but to understand what the system is doing, you need to understand that pattern.

The same goes for micro-services.

Each individual service can manage its individual state, it can be architected and designed in the best way for that particular requirement. But again, the behaviour of the system as a whole depends on how the different services interact (and just as importantly, fail - which is where transactions and other methods of coordinating across your code become important).

So (and I write this as a dyed in the wool O-O programmer), in both cases, the surface problem is solved but all we do is move it into a harder to observe place.

My last job had 43 microservices, one per SQL table. Best part, it was across 8 repos, and shared a core library. Even better, it was python, so imagine adding an additional functions argument to the core library, and updating the caller in 7 other repos, pushing 7 PRs, and it still breaking 3 weeks later in production because you missed a function argument in one of the repos

Microservices were/are an attempt of standardizing software development practices at really large organizations.

For example: your typical big bank has hundreds or thousands of teams (both internal and through professional services companies) developing all kinds of applications under different technologies/hardware.

In this case due to domain/organizational scale it is not feasible to have a mono repo (or 2, or 3) with a giant codebase.

And as a result the typical situation is an expensive hell full of code duplication, ad-hoc integration methods, nightmarish deployments and such.

Microservices are helpful in situations like this (even if they are not perfect).

But for a company with a couple of development teams and a domain that can be understood almost entirely by a couple of business analysts, it´s overkill.

I don't really see what this article nails. So, if I understand the argument correctly. Microservices were used to increase development speed. While old pieces were left behind (legacy) new systems have come up to modernize (these are still microservices)? His team, now responsible for many, legacy unloved old microservices, are being merged back into a monolith. The real question is, is remerging all the code back into 1 application the right solution for their problems they had with stability.

I think mental models are important, and having a huge blob of unrelated roles, makes sense to the current development team. But won't, just like the old situation, to the new developers.

Perhaps it's just the clickbait article, but a better title would have been. "Homogenizing our wild-west legacy microservices".

For me personally microservices was a god send, working on getting stuff done, instead of dealing with ancient code that doesn't reflect the current business anymore.

I still buy in the thought of, if you can't develop a great monolith, you sure won't be great at building microservices. modular-monolith is the cool thing currently. Create a Monolith, without the shortcuts that create problems in the long run. Public interfaces are your most valuable pieces in the system. Worship them, code-review them, fight about them. Currently I could care less about the implementation itself. Does it solve our problems, is it fast enough, is it tested great, ship it. What language you used, architecture. database, I don't care. Just make sure it's a joy to use from the outside.

If more developers would spend longer on thinking about the problems and less with throwing large amount of code that makes them feel smart. Making a microservice doesn't fix that problem.

Think that what is missing is the stability that microservice are able to give in its most optimal form. Each service being the main source, the second it leaves the system it is stale reference data. Is stability important? use the old reference data, how fresh does your data really needs to be.

This is about a legacy system, and scaling back to a monolith, because of that need.

Not holding my breath in anticipation this won't be misapplied and cargo culted.

I have been working in couple organizations where I have thoroughly explained why we should undo the "microservices" approach and go back to monolythic application.

Basically, the problem is that in most cases going to microservices is being pushed by management to complete exclusion of understanding whether the organization is ready to implement it.

Microservices require that you have mature approach to many topics. You need to really have nailed down automated deployments and you really need to have nailed down automatically creating development infrastructure.

In one company I had a tool where I could log in, give a new project name, click couple parameters like namespace and a completely new microservice would be generated with BitBucket, Jira, automated deployments, confluence space, etc.

If you need to spend time to configure anything individually for a project, to do deployments, etc., you are not yet ready to do microservices.

So in all those cases where we have scaled back on microservices the developers switched from developing the application to being full time engaged with managing the application and its numerous configurations.

In one of the applications we had 140 services rolled back to one. Before, preparing the deployment would take 2 weeks as the developers would be sending and compiling emails with information what needed to be done where. Then the deployment would take one day as an engineer would be executing all those runbook instructions. Each of the 140 services had its own version which made ensuring correct versions a separate problem.

After the change where we have rolled it all into a single service under a single version, the entire thing took 2h. Still manually, but way cheaper and more reliably.

It would be nice to have some of those basic ideas that led to your thorough explaination of the pit falls of microservice development. What you write in this post is too general, sure if you have to release your microservices as one mega monolith, then it feels like an easy choice. When your application is not suited for microservices there are still things that can be easier: updating dependencies because of security is usually easier if the whole code base needs it. So it all depends on the application, what kind of development work does it need in a ten year frame, etc.

I really do not recommend big monoliths, but the most important this is that it should be easy to start development, that is usually easier with microservices. I feel your pain the automation really only starts to matter when you are above 40 services in my view.

I am not talking about pitfalls of microservices, I am talking about pitfalls of people trying to do microservices.

Microservices is not a single idea, but a point in a progression of maturity of various aspects of the application and the team developing it.

Think about this, you don't get to be F1 driver just because you decide to. You need to spend a considerable amount of time preparing to be an F1 driver. If your manager (ie. yourself) decides to sit in F1 driver seat but disregards advice to first get years of experience before doing so, this is only going to end in a disaster. And it will not be "a pitfall" of F1 car.

Obviously the problem are managers who misplace their focus. "Microservices" is a shiny object everyone wants to brag about but nobody wants to do the ground work necessary.

So I see teams that don't even have repeatable build/deployment procedure want to do microservices. Why not have repeatable builds and deployments first? This is already going to improve the situation a lot and be basis for further development which may culminate, at some point, with microservices-style environment.

Microservices taught me how to correctly apply DDD in a monolith.

That's probably the most useful thing I picked up on the last year's.

From what I understand (never actually worked with MS, so no hands-on experience) a "proper" MS architecture will also include separate data persistence layers for each service.

I.e. if your service manages persisted data, it will access (and own) a dedicated storage instance which no other service has access to.

Assuming it is really true, folding a microservice back into a monolith should also mean that the data are ported back into the main DB? The article does not seem to spend much time discussing this, though. So maybe it is one of those ideas that everyone conveniently decides to ignore when they actually start dealing with a real world problem?

Some services don't own any of their own data, so for them the question wouldn't arise. You could split them out without migrating data, and you could fold them in without it as well.

Otherwise, yes, moving a service out of a monolith without moving its data leaves you in a pretty tricky and problematic place, though the reverse is much less true: a single service having multiple data layers for different things isn't really a big operational or organizational problem.

> a single service having multiple data layers for different things isn't really a big operational or organizational problem.

Your statement is true, and I’d like to consider the product/feature side as well. Enterprise SAAS customers expect their Single Source of Truth to be, well, a single source. They want reports across all of the data they have put into a product. A feature that should be a couple of JOINs can quickly become pretty dicey when data is sliced across multiple data stores.

This is exactly what always made me very skeptical about microservices (as I said, no actual experience so I might just change my mind if I start using them) - for any non-trivial problem reasoning about your data in a integrated and coherent way seems pretty daunting.

For that reason people employ "data warehouse" systems and architectures. Microservices aren't complete without that complexity.

My problem with this approach is that my system basically handles a complex inventory and allows complex orders out if it, orders you can modify/update/cancel over a very long (months) period.

So we do have data warehouse for strategic analysis and planning, but we need to also manage complex data which has to be consistent and available 24/7.

You push the data from all the microservices to a single data warehouse and do the reporting from there - so all the aspects that modify the data are microservices, but the reporting is based on a data storage monolith.

One strategy is to use the strangle pattern where your new monolith just proxies the old service as you slowly or lazily migrate the data or re-implement the service.

You don't have to, in the short term. Just pull data from different databases.

You don't get the advantage of relational DBs though (ie, the relations, no JOIN for you) so you probably want to migrate at some point. It doesn't seem impossible though, write to both your micro-database and new tables of your main database for a while, then switch to the main database only when you're confident.

(Almost) Everything is "possible" - but looks pretty inefficient to do this. So are MS relevant/appropriate when you are trying to push stuff through the door at a very fast pace, but have to go back to something more "monolithic" sooner or later? Just like the article is suggesting?

Inefficient as in slow? Yes it is (slower than a JOIN, of course), this is explicitely a migration strategy to move from a MS architecture to a monolithic architecture. The alternative is a big-bang migration, which is a much bigger risk.

> So are MS relevant/appropriate when you are trying to push stuff through the door [...]

That really wasn't my argument, I was highlighting that it's possible to do a progressive migration from MS to monolith, I was not making an argument that one is "better" than the other.

To answer this question though: no. A MS architecture will probably give slower responses than the equivalent monolithic architecture (multiple DB lookups, inter-service comms), but that's just one element in the balance (and rarely the most important one, it's rarely important whether you get a response in 100ms or 200ms). I'm not going to go into "why microservices", there's plenty of litterature about it on the internet by people that know much more than me.

A decoupled monolith is _harder_ to achieve than decoupled services. Services are decoupled because they have to. It's perfectly normal to go from "ball of mud monolith" -> "decoupled services" -> "decoupled monolith". These transitions do not _need_ to happen either, they're just perfectly okay

I think most microservice implementations get it wrong. You're not supposed to run one service per machine or even process (that leads to orchestration hell and fragmentation cost at the loss of redundancy): The way you get all benefits of a microservice architecture is if you run all services on all machines: by VM sandboxing them all on many distributed machines f.ex. like this does: http://host.rupy.se

This requires your entire solution to be async.-to-async. end-to-end though. But performance, scalability and strength is unmatched: it becomes anti-fragile!

I think microservices can work for large organizations with many teams. However, for smaller orgs and smaller teams it can for sure be a real killer.

It's hard to keep track on different applications, with different dependencies, different deployments etc etc.

Of course one could standardize things but then some of the benefits of using microservices to begin with disappears. I am a big believer in the big monolith but that is because I am almost always working in either a very small team or alone.

"Not only were the various teams all competing for the same deployment resources, it meant that every time an "Incident" was declared, several teams' code had to get rolled-back; and, no team could deploy while an incident was being managed."

So, how exactly did switching back to monolith solved the above problems? Seems to me like your actual problem is in QA and managing releases than application architecture whether it's towards or away from microservices. What I'm hearing is that you decided to fallback on relying on people to not push broken code out to production. Which is fine and all but what are you going to do if your team grows again? Rearchitect the system into microservices? A little heavy handed don't you think? In your particular case, I think, be it monolith or microservice architecture you need to work more on improving QA and streamlining releases. These, once automated, don't depend much on the number of developers. I've experienced this with a team of 3 as well as a team of about 15 (not 30 though). I don't think that the application architecture has anything to do with your problems one way or the other.

Having said that, if the late night calls stopped after this exercise, then that's fine because no one should have to go through that!

> So, how exactly did switching back to monolith solved the above problems?

It seems that you didn’t read the whole article, but instead rushed to express your superiority.

Switching to a monolith helped because they are a single team maintaining multiple services. Combining the services reduces their overhead. The deployment problems haven’t recurred because their team is the only one deploying the monolith (or at least, that’s what’s implied.)

I did read the article. Saying your now small team to be "careful" is not a real solution which I mentioned in my comment. What do you mean by expressing my superiority? Expressing my opinion backed by my own professional experience which happened to disagree with your own?

Incorrect.

Eloquent. So... we should change application architectures as the team changes size?

That appears to be what the article is suggesting.

Team size and structure definitely is relevant, as many of the main arguments for or against specific architectures are about how it makes certain developer processes faster or safer or easier or more predictable; so yes, it seems reasonable that the most appropriate architecture would change as the team changes size - though it does not mean that we should change the architecture, as the benefit may not be worth the cost of change.

Changing a system architecture twice like this, specially to go back to what it was before is a large cost to a company. In this line of thinking, they may have to move back to a microservice if they had a bigger team for the exact same application! This is fine by me, but rather than what other benefits they enjoyed by migrating the second time, a clear analysis of motivations behind moving to microservice and then back to monolith without compromising the benefits attained by doing the first migration would be a lot more enlightening. Because I don't imagine the first migration to microservices happened just because they had too big a team. I suppose this thread will naturally attract people who hate microservices in principle.

Logic:

Because of microservices, we had bugs (okay, what kind of bugs?). Therefore we moved back to a monolith again (why did they use a microservice architecture in the first place). Then the bugs disappeared (how?) and we got a whole bunch of other benefits as a bonus (any trade offs?).

Cool, so the initial migration to microservices must have been done by complete idiots then. It seems that even to question that is too much here.

I think you're starting to get the point:

There are idiots out there. And they're doing microservices. For no reason besides hype.

The article says exactly why they moved to microservices: too many teams were deploying the monolith and it slowed them down.

The article also says why they are moving back to a monolith: the services in question are legacy and being replaced, and only one team is maintaining them, so the reason they moved to microservices no longer applies. In addition, combining the services makes their lives easier enough that it’s worth it from their perspective.

Your aggressive tone and assumption that they didn’t have good reasons for what they’re doing is why I said you were “rushing to express superiority.”

There is an alternate view of what microservices should be. Check out the sample chapter for Monolith to microservice by Sam Newman. As described in this article, microservices seem like just a way to horizontally scale development and deployed implementations. Nothing like independently deployable network functions corresponding to business needs.

This article isn't about microservices. It's about a monolith that microservices were built on top of, then that code pulled back into the monolith.

If you built serviced with its own data, built translation layers to handle deltas, you could move away from the monolith over time and have discreet services for your domain and sub-domains.

Because it's the closest you can come to something that feels like meaningful agency in that dreary an environment

I really like the energy you bring to working on maintenance / legacy code. I also have some bananas that need straightening. Maybe next year you can show us one more time how to break up a monolith???

People should be more worried about writing non blocking IO code.

Blocking a thread in the cloud is paying the provider to do nothing.

Using multiple threads also solves that problem. If one thread blocks on IO, others can have the CPU.

microservices are a technical tool for solving business problem , business problem => people and organizational problem

This article should be No. 1 on HN.

For a week.

Reply on news.ycombinator.com