Hacker News Reader

Comment by necovek | Hacker News Reader

I don't think you read the post carefully enough: they were not running a distributed monolith, and every service was using different dependencies (versions of them).

This meant that it was costly to maintain and caused a lot of confusion, especially with internal dependencies (shared libraries): this is the trade-off they did not like and wanted to move away from.

They moved away from this in multiple steps, first one of those being making it a "distributed monolith" (as per your implied definition) by putting services in a monorepo and then making them use the same dependency versions (before finally making them a single service too).

View on HN

I think the blog post is confusing in this regard. For example, it explicitly states:

> We no longer had to deploy 140+ services for a change to one of the shared libraries.

Taken in isolation, that is a strong indicator that they were indeed running a distributed monolith.

However, the blog post earlier on said that different microservices were using different versions of the library. If that was actually true, then they would never have to deploy all 140+ of their services in response to a single change in their shared library.

Shared telemetry library, you realize that you are missing an important metric to operationalize your services. You now need to deploy all 140 to get the benefit.

Your runtime version is out of date / end of life. You now need to update and deploy all 140 (or at least all the ones that use the same tech stack).

No matter how you slice it, there are always dependencies across all services because there are standards in the environment in which they operate, and there are always going to be situations where you have to redeploy everything or large swaths of things.

Microservices aren’t a panacea. They just let you delay the inevitable but there is gonna be a point where you’re forced to comply with a standard somewhere that changes in a way that services must be updated. A lot of teams use shared libraries for this functionality.

These are great examples. I'll add one more. Object names and metadata definitions. Figuring out what the official name for something is across systems, where to define the source of truth, and who maintains it.

Why do all services need to understand all these objects though? A service should as far as possible care about its own things and treat other services' objects as opaque.

... otherwise you'd have to do something silly like update every service every time that library changed.

Because of organizational standards.

I don't understand your answer I'm afraid

As you mention, it said early on that they were using different versions for each service:

  > Eventually, all of them were using different versions of these shared libraries.

I believe the need to deploy 140+ services came out of wanting to fix this by using the latest version of the deps everywhere, and to then stay on top of it so it does not deteriorate in the same way (and possibly when they had things like a security fix).

Except if it's a security fix?

If a change requires cascading changes in almost every other service then yes, you're running a distributed monolith and have achieved zero separation of services. Doesn't matter if each "service" has a different stack if they are so tightly coupled that a change in one necessitates a change in all. This is literally the entire point of micro-services. To reduce the amount of communication and coordination needed among teams. When your team releases "micro-services" which break everything else, it's a failure and hint of a distributed monolith pretending to be micro-services.

As I said, they mention having a problem where each service depended on different versions of internal shared libraries. That indicates they did not need to update all at once:

  > When pressed for time, engineers would only include the updated versions of these libraries on a single destination’s codebase.
  > Over time, the versions of these shared libraries began to diverge across the different destination codebases.
  > ...
  > Eventually, all of them were using different versions of these shared libraries.

The blog post says that they had a microservice architecture, then introduced some common libraries which broke the assumptions of compatibility across versions, forcing mass updates if a common dependency was updated. This is when they realized that they were no longer running a microservice architecture, and fused everything into a proper monolith. I see no contradiction.

See my response to a sibling comment: they did not have "forced" updates and they really ended up with:

  > Eventually, all of them were using different versions of these shared libraries.

Which is sort of fine, in my book. Update to the latest version of dependencies opportunistically, when you introduce other changes and roll your nodes anyway. Because you have well-defined, robust interfaces between the microservices, such they don't break when a dependency far down the stack changes, right?

Reply on news.ycombinator.com