Hacker News Reader

Comment by nickcw | Hacker News Reader

I think that the problem here was that they were fighting against Conway's Law: https://en.wikipedia.org/wiki/Conway%27s_law

> Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure.

I think microservices work well in organizations that are big enough to have a team per microservice. However if you've just split your monolith up and have the same team managing lots of microservices you've made a lot more work for the team without the organisational decoupling which are the real win of microservices.

In my experience it is really difficult to fight Conway's law, you have to work with it and arrange your business accordingly.

View on HN

As with a lot of things, it comes down to communication. Between teams, and between the services they write. Which is just another expression of Conway's Law.

IIRC Fred Brooks pointed out that the # of bugs in a system correlates closely with the # of lines of communication within and between the teams. Joshua Bloch recommends in "Effective Java" that, if possible, 3 potential clients should participate in the design of an API, for the same reason. So a well-designed interface or OpenAPI spec is worth its weight in gold.

Ofc, "microservices" here means separate running instances available on a network. But monoliths can be "service"-oriented as well. OSGi was good for this in Java, but any system able to load shared objects or plugins dynamically can follow the same pattern. And the benefit is that, if your app hits the jackpot and needs to scale outwards, the service interfaces, ie the lines of communication, are already well-defined.

So, service-oriented monolith first, then microservices if needed.

> So a well-designed interface or OpenAPI spec is worth its weight in gold.

When I worked on a SOA team, I tried to begin any new effort (whether a new API or modification to existing API) solely discussing the API contract. It was (ideally) high-level enough that business analysts and project managers would understand it, and it helped to guide us away from getting mired in implementation discussions too early.

At that organization, we rarely had the opportunity to involve multiple customers at the same time during design discussions (we were typically engaged to help a specific consumer implement a specific feature), but the institutional memory in the SOA team helped us to keep in mind existing/potential other users of each particular webservice.

The last place I worked that split the devs into UI and backend teams was in a sort of slapstick comedy situation. Nothing ever shipped on time because the front end and backend could never quite talk to each other or needed elaborate conversations to do the simplest of things. This was our new flagship project, I got consolidated in from another team and ended up as a lead not long after.

We had been doing some UML modeling, sequence diagrams during planning, and still having this problem, so rather than repeating the same action and expecting a different outcome I started trying to flip the script. What ended up working was not code diagrams but data flow diagrams and sequences. To get X you need Y, and to derive Z you need A, B, and X. To publish you need all five.

After that, the APIs mostly wrote themselves, we reordered a few different forms, but most importantly variance dropped like a rock.

Can you share examples of your data flow diagrams? Do any open source projects share these documents?

Mostly these were white boarded, but essentially I/we would draw a collaboration diagram (although I could have sworn these used to be called something else). They showed what data was needed to make certain decisions (eg, a conditional drop down that is populated based on another piece of data, or complex validation steps) and where to get data that already existed.

activity diagram?

Yeah it looks like the activity diagram was substantially altered in UML 2.0. What we called an activity diagram then looks more like a collaboration diagram now.

Aside, take a look at tools like PlantUML as a way to create your diagrams. It's higher-level than, say, rolling everything with Graphviz, while easier to share and edit than a bunch of PowerPoint/Visio/etc. files.

The great thing about generated-diagrams is that you can easily store and version the original text representation along with the code it describes or applies to.

https://plantuml.com/

> As with a lot of things, it comes down to communication. Between teams, and between the services they write. Which is just another expression of Conway's Law.

This is so accurate. I've heard engineers give state not needing to communicate, chillingly, as a positive for microservices, like "we won't need to talk to each other if all of us are working on different services". My other favorite is using microservices as an excuse for why the product isn't working "oh, my service is working fine, but his service is doing this when it shouldn't", when we're on a small engineering team.

I think there's an element of truth to the engineers' claims. Working on different code bases means there are a lot of things you would otherwise need to talk about that now you don't. It's very much the case that you still need your interfaces to be clear (in fact, clearer!) but those discussions can be somewhat isolated, so more work can proceed asynchronously. Just how isolated depends on how exact (and correct) the specifications are, which is a question of trading up-front work against interruption.

> not needing to communicate

sighhhhhhhhh

API documentation is a medium of communication as much as any user interface.

If you don't keep this in mind, then using your service's application programming interface will be a bad experience.

Yeah, the problem with microservices is because the organisation structure is wrong. I’ve literally heard every excuse about microservices at this point. My architecture is better but it doesn’t have a snappy name; it’s called the smallest possible number of services that can be reasoned about and network partitions are NOT necessary to create bounded contexts in a codebase, often just a directory is FINE.

I agree. I hate the term microservice for the same reason I hate superlative infected clickbait titles. There's no need for half of the word to exist. Service. What's wrong with service?

I agree with you, but I find some value in people using that term - it signals to me that I should consider the the architecture was prematurely split-up and could suffer from the various pitfalls associated with microservices.

Microservice implies systems that are decoupled for deployment purposes. For example, Microservice A could restart to a new version while Microservice B keeps running. This is a more complicated interaction contract than services where their deployment is coordinated in concert.

But this was true in the middleware type of products too, and you can't get more monolithic than that.

I don’t think this is accurate. I’ve worked at companies that did “service-oriented architecture” long before the rise of the term “microservice” and it was clearly recognized that different “services” shouldn’t be so coupled together you can’t redeploy them separately.

Yes, you're right. A system that requires services to be deployed together is just a distributed monolith.

This thread considered an issue: whether services and microservices are equivalent concepts. They are not. There is a quality that is held by Microservices, yet which is not universally held by Services.

You have observed that other services also have that quality. Indeed. Nowhere did I say, "all services with decoupled deployment are microservices".

Revisit. I can where brown9 is coming from. I could have avoided leaving that interpretation open by writing, "here is an example of a quality that is held by all X, yet not by all Y".

There was a period of time when Service had a different meaning than microservice. A service traditionally may exist across bounded contexts and be almost a mini-monolith whereas a strict microservice should touch very few data models and exist strictly in a bounded context.

Of course real life is messy and plenty of people realized writing small single purpose services was valuable, and plenty of people build giant "microservices" that have nothing to do with the original term and are just badly constructed monoliths.

It is probably too late to change the name. But you have a good point, the "micro" prefix is highly misleading. Furthermore, there is little guidance in the literature on how big the microservices are.

Plenty. A subroutine is a service. A library is a service. A Windows daemon is a service. The vendor I just inked a contract with provides a service. A web service is a service.

I really hate that word when used without further definition.

> A subroutine is a service.

Which makes the term "microservice" even weirder, given that any microservice is going to be bigger than a single subroutine.

> There's no need for half of the word to exist.

Yes there is. A service is a very generic concept to the point it's only relevant as a high-level concept.

The concept of a microservice makes all the sense in the world if you look back to where we came from: web services. When compared with all the work and requirements and complications of using SOAP and WSDL and UDDI and everything around, just sending small JSON payloads around, and the ability to peel off smaller services leveraging that architecture approach, was a far lighter and uncomplicated way of doing business.

I mean, the name microservices becomes obvious once you look back and all that you see is macroservices.

That's not what the person you are replying to said though, "the organizational structure is wrong". More like: It is a mistake to use microservices UNLESS you have a certain organizational structure/capacity already.

I think they were saying something more aligned with your opinion than you read it as.

We are dealing with poorly defined terms. However, services mapping ~1:1 teams was generally called service oriented architecture not micro services. Micro services involved breaking things into even smaller chunks, so backing off of that idea really just means SOA as originally defined is a bad idea.

That’s not quite right, SOA as originally defined had no mapping to team structure or deployment runtime, it was mostly about defining discrete service interfaces and ensuring your clients used that contract rather than back channels to communicate. Most often you had a dozen services running in a single app server cluster. Conway’s law was rarely discussed (with some exceptions).

Microservices tended towards a single runtime per service, ensuring the deployment lifecycle was tied to the build lifecycle and thus allowing for independent evolution.

I am not saying that’s how SOA was defined, just that it was used to refer to such team organization around architecture. EX: Amazon famously uses a Service-oriented architecture where a service often maps 1:1 with a team of 3 to 10 engineers. https://en.wikipedia.org/wiki/Microservices

At the beginning Microservice was generally viewed as more granular than SOA, though that’s been backed off of.

The general view of microservices was largely invented out of thin air ;) when you look at Martin Fowler’s wiki or Adrian Cockcroft’s presentations which were the originating popularizers of the term , it was all a reasonable refinement of SOA.

But then you’d get some that would make bizarre claims like a microservices must be under 100 lines of code. :shrug:

People where not pulling that from thin air.

Cockroft’s Rule of Thumb

Can complete a service in two weeks or less Completed = coded, tested, and in production • Fits in “one or two developers’ heads”

At that rate you quickly hit hundreds of services.

Or an NPM package works nicely (or .NET assembly, Ruby Gem, Java whatever, etc.)

Library code for sensibly defined pieces 100%... but if you aren't sure of the abstraction, copying code can be more forgiving than making a mess.

Noo! Building teams around software components cements your architecture and prevents most cross-cutting improvements.

I'll claim that splitting a well-structured monolith into microservices will always make it less maintanable, but it might be worth it if you need to for some reason like elasticity or failure tolerance.

But for the love of god, keep the design open. Don't tie the existence of internal software components to peoples livelihoods.

> Don't tie the existence of internal software components to peoples livelihoods.

The claim is that such ties, at the macro-structure level, are inevitable and exist regardless.

The point is then to determine the best way either to restructure the organisation, or, the code base, to cope.

I think the ties arise because people are actively seeking areas of responsibility. Software components are an obvious grab if your eyes are on the software specifically. But there are other ways of dividing your teams; based on for instance customers, use-cases, aspects of the code (performance, security).

The problem is that the software usually keeps expanding until programmers find it hard to cope. If you split teams up so that some people are only concerned with a certain part of the codebase, chances are you are going to grow the size of the codebase by a quite large factor.

I think there should be an incentive in place to keep the codebase small and understandable by most.

It's pretty hard to keep the design open once the whole architecture is bigger than what a single programmer can keep track of. Say, the Linux kernel. The overall architecture is fixed, there's no way around it. At that point, splitting into components that are maintained separately does no harm. AFAIK the Linux kernel is maintained like that already in practice, even if it's a single repo.

I agree with you in such cases, but I'm willing to bet that most codebases don't need to be as big as they are, and that it's better to create an incentive to collaborate and keep the codebase maintanable and small.

The opposite of "has a team around it" is "abandoned". Or at least low down on somebody's priority list.

That's generally true, and it's a big problem with microservices, because they need so much upkeep.

But if your code is living as a few hundred or thousand readable lines in the common codebase, that isn't really a problem. The code is there, readable and working, and if anyone needs to change it they can. If it falls out of fashion, it can be deleted.

What is your alternative? Tying "the existence of internal software components to people's livelihoods" across the expanse of the entire codebase is the only remotely effective approach I've seen to scaling the SDLC at scale.

"What is your alternative?"

Aggressively small teams, with no hands-off middle-management layer.

You can build massive capability around a small number of well-managed message-backbones and a single codebase. By keeping the number of hands small and the structure flat, you force high standards. (Skilled staff won't tolerate distractions caused by bad engineering or inadequate automation.)

Heuristic for analysing firms: who has strategic power in decision-making? Conventional answer: a group of hands-off middle-managers who run on meeting tempo, and who are valued by how many people and systems report into them. Under AST: an engineering effort running on maker tempo in cooperation with a hands-on sales effort.

Microservices tend to have multilateral contracts with other systems in the organisation. This steers all planning towards meetings. This creates middle-management bloat.

Is there any example where this works (articles, presentations, etc)? In particular, anywhere with more than a couple dozen developers?

Amazon has a famous love for what they call “two-pizza teams” and you can find writeups about the philosophy by searching the term. The joke is that a team should be small enough that you only need to order two pizzas to feed them all. The philosophy is about the number of participants in the decision-making process. Keep teams small and give them total ownership of decision making so that decisions can be made by a small group of people who work with each other every day. That way no meetings (and certainly no cross-team meeting) need to happen for most decisions to be made.

> The joke is that a team should be small enough that you only need to order two pizzas to feed them all.

That's a tricky way to measure, given that I can eat a large pizza myself in a single sitting ;)

Amazon is very well known for having A LOT of middle managers too, so I'm not sure it's a good example?

Seems sorta reasonable that if you need a manager for every 6-8 engineers, you would end up with a lot of managers.

OP’s post was “ Aggressively small teams, with no hands-off middle-management layer”. 6-8 swe teams + hands off people manager reporting to middle manager, who reports to director, is how Amazon organizes teams, therefore it isn’t an example of what their suggestion was...

Think of all the open source libs. Generally speaking, anyone can contribute to any part of the project.

That's not to say that some people are better than others at certain parts of the codebase, but you don't want people fighting to keep old cruft in because it's on their job title (figuratively speaking).

You can organize around customers, use-cases, platforms, concerns or other things. Some might naturally map 1-1 to software components, but the software component should not be the "reason d'aitre" for a team, rather the customer experience or something else which can transcend multiple interations of the software.

I see, you meant things in a more literal sense. I generally agree with you in that case, that the customer experience should be the thing which the team owns, which incidentally involves owning software components. But on the other hand, it's also certainly the case that at a company of a given size or in a given sector, certain kinds of software components and infrastructure are not directly customer facing and yet must be owned in house, and logistically serve as one of (if not the only) competitive advantage over competitors.

Is it wasteful to have whole teams at GOOG, FB et al owning and improving the state of the art of infrastructure? It depends. At a certain point, there are enough internal customers for teams to reach contribution margin positive on engineering initiatives that have no direct but only second order effects on customer experience.

I've seen this pendulum swing both ways, often within an organization. Cross functional teams owning code bases allows divergence to specialize and ownership of a release, teams with a single functional focus allows efficiency of work and cross cutting gains.

Both have their boatloads of suck, neither is inherently better. Interestingly, trying to mix them to get the benefits of each doesn't seem to invalidate any of their downsides; often it exacerbates them.

Mmmmyes and no. Depending on the size of your project, that may not be the case. I've had to work with two titans of monoliths, maintained by relatively small teams(anywhere between 2 and 6-7 people for several million lines of code). At some point managing a codebase this big within a single project becomes a huge burden, for both developers and even more so for those who develop and do code-reviews(first hand experience right here). At times I've spent 3 weeks straight doing code reviews with 2 notebooks filled with notes and diagrams of the different components inside the code. And at that point, the easiest and most sensible thing to do is chunk out large parts of the project and put them aside as a microservice with the adequate amounts of tests. For small projects, microservices make little absolutely no sense. But in the case of something the size of AdWords(which my two such experiences can be compared to), you are playing with a raging lion if you decide to go monolith.

My argument here is that it's not so much the size of your team but rather the size and scale of your project that needs to be taken into consideration.

Good monoliths are highly modularized. But it's a whole different thing to package up a module as a separately deployable unit for external "public" use (external to your app, that is, not your company).

I'm just curious to know, when you said "the easiest and most sensible thing to do is chunk out large parts of the project and put them aside as a microservice" ... were these chunks separately deployable units for external "public" use.

You're right, and I think you've highlighted what makes a good monolith so hard to build and maintain.

You need to be disciplined to keep a monolith highly modularized. For microservices, in contrast, their architecture encourages modularization.

I don’t know that you need to be much more disciplined to write a large application in a module way vs writing any application in a modular way. A monolith could definitely get messy though if you write them how I see people write microservices.

I think this is actually one of the reasons that microservices became a thing to begin with: teams wouldn't actually apply engineering best practices.

Microservices actually make you encapsulate your code, at least within the microservices, because you can't call out to it directly. They don't necessarily force you to implement the single responsibility principle, but they do a good job of pushing you. Microservices implement a service-locator pattern through DNS or web routing, one form of the dependency inversion principle. Microservices make you pass data around as entities, instead of Active Record instances.

The price for this sort of thing is very steep, though; distributed systems are inherently icky, harder to trace, and more prone to failure, and besides this, you've added network overhead to each service call.

I wish more engineering teams would consider spending half the effort of microservices on simply disciplining their monoliths. They might get somewhere...

> They don't necessarily force you to implement the single responsibility principle, but they do a good job of pushing you.

In my experience, if your services are developed by the same people, and not separated by teams, engineers will often tightly couple the services with fragile and opaque dependent changes regardless.

While in monolith this is painful, at least you have a complete stack trace and the ability to run things through a step debugger you orient yourself. In a distributed system tribal knowledge tends to be your only savior.

When we design systems, we need to spend more time thinking about what is most likely to happen as opposed to what we feel should happen.

>I wish more engineering teams would consider spending half the effort of microservices on simply disciplining their monoliths

100%. This is an uphill battle, though. I've encountered so many engineers who equate "real engineering" with "building giant machines." You just can't convince them otherwise.

I've watched people build giant, real-time stream processing pipelines compromising tons of moving pieces (lambda, sqs, s3, sns, stepFunctions, etc..) to build... a reporting table, and all for... 1.3gb of data. Literally.

Ultimately, despite the "sell," I don't think microservices as a forcing function for good practices works in practice. If the team lacks the skills to build a disciplined monolith, then they 100% lack the skills to build a distributed one.

Oh, all of those were heavily modularized to begin with. But that wasn't enough to keep them manageable. So at the end what we did is figure out which are the core components between the different modules, isolate what they did and put them aside in a smaller microservices, which were easier to track, maintain and monitor. What was once the monolith is now arguably just an interface/API for all the heavy lifting which is done by microservices. Again, my point is that all this must be done depending on the scale and complexity of your application. If you are going to make an authentication microservice for an application that has 50,000 users which simply fetches a username and compares a hash in a database, obviously you are doing it wrong. I am talking about applications which in the simplest of times operated on 24 different databases located in completely different geographical locations(the case of my first such monolith). Some of those databases used different engines. And due to the nature of the infrastructure and the requirements we couldn't simply ditch everything and start over from scratch. So splitting everything into microservices was the only option. And this is something I was working on back in 2012 iirc so back when microservices were considered witchcraft by most people. And yes, I'm talking about several million lines of code and 2 developers - my inexperienced out of uni ass, and an utterly conservative dev twice my age. Took us around 6 months but the project was extremely successful.

There is this trend in technology - every few years everyone changes their minds about everything:

* 2012 - sql is the best.

* 2016 - sql sucks, nosql is the future

* 2020 - nosql suck, sql is the best.

* 2024 - {fill in the blank}.

The same thing is happening with microservices. But in addition docker, kubernetes and recently unikernels have joined the party. The concept is the same though.

What I am trying to say is that either of those can be good or bad in different scenarios. It's a question of picking the most appropriate one for the situation.

The fun is that we have seen this so many times.

Sun RPC, CORBA, DCE, DCOM, XML-RPC, SOAP, REST, gRPC,....

If you've got 7 people maintaining millions of lines of code, you're going to have a heavy burden no matter what you do. Extracting a service does not a priori simplify anything. It can encapsulate and enforce a more strict boundary, and optimize compile time or test suite throughput and operations for the extracted logic, but it always comes with overhead, and if the interface between the services is not well-defined and stable it can easily be a net-negative in terms of productivity as you are now giving up your in-language tools for distributed systems tools. Now if you have large swathes of code stable functionality, then it's easier, but at that point why not just isolate modules within the same codebase?

I cannot agree more: I worked at a company where we went from a monolith deployed on IaaS with a couple handful of engineers to Docker containers deployed on ECS with over 200 engineers. The main reason we did it was because Docker+ECS was cheaper than a bunch of EC2 instances and you can't effectively use 200+ engineers with a single monolith.

After 2 years we had over 450 microservices while keeping our AWS bill flat or slightly decreasing.

On the other hand, over 200 engineers on payroll is way more expensive than a couple handful!

Presumably you're getting significant value out of the additional engineering work in which case the architecture shift probably makes sense (to stay aligned with the expanded organizational structure), but there are also cases where a small and flexible team maintaining a simple monolith would be much more nimble and cost-effective.

In all honesty, I think the monolith/microservice distinction misses the point a little bit.

It's inevitable that the longer the codebase exists, the more difficult it is to maintain. It's a battle that you can't necessarily win and it's turtles all the way down as your dependencies, and their dependencies, tackle the same issues.

All it takes is one or two roughly defined APIs and you've already created the nucleation point for ever-more tech debt, and while you'll be able to tame some of it you won't manage all of it due to business requirements, or other teams depending on private APIs to save time, or whatever else you can imagine. Switch the architecture and you'll either have all your problems bunched in one codebase, or you'll have distributed your problems all over the place.

I'd go as far as saying that a perfect monolith and a perfect distributed architecture are theoretical ideals that require perfect communication to build them.

Maybe its conways law, or maybe it's just that designing a distributed service is difficult, and when you break a monolith down, you're having to deal with distributing that monolith N times, and solving those CAP issues N times, which usually is not trivial. Not to mention tuning the network.

I don't agree with your premise that development structure == deployment structure. There are plenty of good ways of splitting up development of a monolith without the huge devops headache of deploying microservices.

A team per microservice? That sounds really wasteful. How many microservices need constant evolution?

But does conway’s law require microservices? It doesn’t say anything about microservices.

> Melvin Conway, who introduced the idea in 1967.

Yes, I don’t think that you need microservices to be able to tackle Conway's law. At least it doesn’t have anything to do with each other.

You could still do microservices and still fail to deal with Conway’s law.

> You could still do microservices and still fail to deal with Conway’s law.

That's what the poster suggests happened. Nowhere do they suggest that microservices are overall required.

You don't tackle Conway's law. You can't. You use it on your favor by creating organizational structures that reflect the design that you want in your software.

There is nothing wrong with most developers working on and communicating about the entire code base. Having teams work in silos is not a benefit. You're touting as a benefit what is one of microservices' gravest issues - teams stop communicating beyond the surface level of their respective APIs.

Have you tried coordinating entire teams to work on a shared codebase?

Honestly, I have never been in an organization so large that this became a necessity (if you solve tens of different problems, that would require almost thousands of developers). But coordinating single developers without an API is hard enough already, I can only assume for teams its nearly impossible.

Define “codebase”? You can have multiple services, user facing apps or modules inside a single repository, but if there are no boundaries coordination will be difficult of course.

The definition implied by the GGP is: shared codebase = everybody will change the same lines; separated codebase = people will work on different sides of an API.

At least, that's what I understand from his comment.

> I think microservices work well in organizations that are big enough to have a team per microservice.

Presumably by definition we’re talking about a few hundred lines of code, or a couple of weeks development time here at most. What does this team do all day otherwise?

So you are saying something in line of: let's increase our development staff X-fold and then we can finally do the same thing that way fewer people doing just fine right now?

With microservices there's no way around it, as there's additional overhead when splitting a for loop between multiple services. Won't stop people from jumping on the bandwagon though.

They're clearly not saying that. If your team is too large to effectively work on a monolith, splitting it up can make sense, but you also need to split the team into smaller groups responsible for different parts. And if you don't end up with teams responsible for individual services, you likely split to small. And quite possibly, your staff isn't large enough to warrant it.

Just because monoliths may have diminishing returns at certain team/project scale doesn't mean the scale itself is the problem...

The problem is with people trying to do "cool" things when completely unwarranted

Reply on news.ycombinator.com