Hacker News Reader

> I'm at this very moment gearing up to begin a new large project and am wondering what design makes the most sense

IMO a monolith is always the starting point until your team gets big enough to justify breaking it up.

> Seems like this could be dealt with by applying more discipline and just making sure to design things to be more modular and.. separate

This really depends on the team size. As you get above, say, 30 engineers contributing to your codebase daily communicating those logical boundaries becomes nigh impossible without significant operational overhead. You either accept the chaos and let your codebase become a Big Ball Of Mud (most common) or you introduce so much process overhead that your team paradoxically gets slower the more people you add. I'm sure there are exceptions but it's rare.

> Do we need an entirely different architecture, with all the pitfalls that microservices entail, because we can't seem to keep things clean and separate?

Microservices are probably not the tool to reach for no. Several "fat" services separated along some kind of bounded context within your domain? I've seen that work great as a next step in a monolithic architecture evolution. Teams can have the autonomy they need to make decisions quickly, separation of concerns is enforced by the network. The extra overhead incurred isn't untenable since you only have a few services instead of a hundred.

> A single regression within a monolithic code base can bring the whole system (in this case, all of Uber) down.

Serious question: why though? I can't think of a backend system where a single regression could bring the whole thing down. If something blows it up, it rarely takes the whole application down. What's this scenario exactly? Maybe I'm only thinking of certain types of applications, but a single failure somewhere needn't bring the whole thing down.

A crash in a stateless distributed tier (e.g. web servers) will not take the whole system down.

But a crash while processing any request in a stateful non-distributed multithreaded system, can bring it down. It's much harder to make stateful systems resilient and fault tolerant. Usually it's solved by moving the state somewhere else (e.g. databases, message queues, or Kafka-style AOLs).

Exception: monolithic stateful systems using BEAM/OTP/Erlang/Elixir. Here the blast radius is per Erlang process, so in the worst case only a single request/user/session should be affected. There are some classes of errors (e.g. out-of-memory, overloaded process message queues, or crash in native code library) which can bring down an entire Erlang node. This can be mitigated by running a cluster of several Erlang nodes (which is supported out-of-the box by BEAM VM and OTP).

Erlang/elixir is a very nice middle ground situation where you can build "monolithically" but also inherently be building services that are isolated, and you can scale to 50-150 nodes before getting any more complicated.

150 nodes is a lot. I love the beam.

That's for the standard full mesh clustering, Erlang distribution can be further tuned with hidden nodes, etc. I think WhatsApp scaled clusters to 10_000 nodes[1].

I personally only run max 10 nodes Erlang cluster (on physical servers) in production, and ~ 20 on AWS for dev/testing workloads.

But I think it's better to have a federation of smaller clusters, or separate cluster per datacenter/AZ.

Erlang distribution isn't a silver bullet, though. But for 80% of cases it's enough.

---

[1] Scaling Erlang cluster to 10,000 nodes

https://codesync.global/media/maxim-fedorov-scaling-erlang-c...

Yeah then also there's partisan. I didn't figure anyone that needed to go beyond 150 nodes would fail to learn this stuff, but I appreciate you mentioning it so I don't accidentally drive-by convince someone there's an upper limit rather than engineering tradeoffs :)

I was experience a case : whole system (a stateless distributed tier) down caused by a user input (the user try again and again until all instances down).

Crashed instances should restart. On EC2 one can put all instances running stateless code into autosacling ground behind load balancer, or ReplicaSet for deployment/service in k8s.

BTW: I expirienced the entier us-east region going down just in time when I was demoing my product to a VC. Turns out it was a bug in an AWS Management Console for that region.

I'm also generally micro-service skeptical, but I'll do my best to give what I believe an answer is. The industry had a huge micro service phase for like the last decade, and I think in the end most of the same arguments given for micro services could be answered with just modules. Someone else can correct me where I'm wrong, but I think maybe the one argument for a separate service vs module is security, where you want literal process isolation for data.

I guess another benefit is micro services make it easier for teams to work in whatever language/tool they're comfortable with.

I think that's it really. Better defined ownership could be done with modules. Testability could be done with modules. Most of the usual arguments could just be answered basically with well defined code, and when the interfaces are all async then it even means even those modules could have micro services as their underlying implementation.

As a side note, I think even a lot of the data/process isolation stuff could be better mitigated by capabilities-based technology. It's kind of crazy to me that stuff isn't used more.

There are two aspects here I think.

Modularity in terms of code, and modularity in terms of deployment lifecycle.

As you say most dependency problems can be handled at the level of the code, depending on how you structure the builds etc. ( personally I think automatically pulling in dependencies is a mistake - it should be slightly painful to add a new dependency ).

For me, the reason for microservice like deployment is having a separately lifecycle - being able to start,stop, update separately from other services.

Another aspect which I don't think is talked about enough, is that every deployment has a copy of the code ( in whatever executable form ) - this means I can deploy version 1 of code A in service X and version 2 in service Y at the same time if I need to - ie I can incrementally update the system.

So one way to think about what should be deployed together ( or not ) is thinking not about the code per se - but also the natural deployment cycle.

It's not just about code -- microservices can offer a degree of modularity on an infrastructure level. For example, a microservice can own its own database.

Being able to experiment with languages and tools (as you mention) is also a pretty big advantage. HTTP offers a universal communications interface beyond language-level APIs and ABIs. When we think about an entire department with many teams with a diversity of expertise, at least we can guarantee everyone understands HTTP.

Having a separate process also allows for decoupling pieces of application logic so that they can be deployed independently of each other. You can build this into a monolith but it's typically hard and error-prone.

A pipeline I work on right now ingests large amounts of data from an FTP site and dumps it all onto a Kafka topic which is handled by another process on the consuming end. If this were a monolith it would run faster, but any errors or crashes would either require reprocessing the entire file (not really possible given its size) or keeping state ... which is what we are using Kafka for. We can restart pieces of the pipeline at any time and they just resume where they left off. Having a monolith has no overwhelming advantage to us in this use case.

That is a sensible split into services but when companies try to split services based on teams then there rarely are any reliability improvements since team difference may not match actual reliability concerns.

Wouldn’t writing/reading to/from Kafka sort of be orthogonal to micro services?

I say this not to argue one way of another for services, just to say that I think your example is just Kafka, or more generally event driven architecture. One thing that defines services is a request/response messaging pattern. In other words a service makes a request and then they wait for a response, like http. What you’re talking about resembles a service because the producer/consumer are separate processes, but they don’t really have much else in common with services, due to a different messaging pattern.

No? Why are micro-services only defined by synchronous/socket communication? Even monolithic threaded apps rely on asynchronous workflows. The fact that a 3rd party is used to relay the request doesn't really matter.

My point was that I can build a monolith or micro service around Kafka and they end up being equivalent but the multi process approach works better for my needs. The original question was asking what advantages a separate process would have. I gave one.

> Why are micro-services only defined by synchronous/socket communication?

In the name is the word "service", which means a process that waits and responds to a client's request. This act of request followed by response is fundamentally synchronous.

That very much depends on your definition of service.

OASIS defines a service as "a mechanism to enable access to one or more capabilities, where the access is provided using a prescribed interface and is exercised consistent with constraints and policies as specified by the service description". [0]

I see nothing that requires synchronous communication only. Virtually every large micro-service architecture I have ever seen communicates through queues asynchronously.

[0] https://en.wikipedia.org/wiki/Service_(systems_architecture)

In response to all the people who are talking about the lifecycle of services, or independent deployments, wouldn’t something like small talk essentially do this?

It’s super old tech but that doesn’t mean that what’s new is automatically better. And in this case something like small talk sort of seems like it’s from the future when compared to the hodgepodge of tools that are sort of all duck taped together that form the basis of modern infrastructure.

I think in a way our ideas as modern developers have been held back by the present day status quo. In a large way that’s in part because it’s many of our jobs to build these systems.

Sorry, I know it’s a bit of a rant, and I don’t mean any offense at all. I’ve worked on platform before, so maybe I’m projecting, but I really do think it’s good or even necessary to question the present day wisdom.

>I kind of get this...The idea is that if the other area of code simply isn't accessible to you, you're forced to keep the concerns separate?

IME about half of the microservice architectures Ive worked on have managed to make sure that the concerns and network boundaries cut across completely different places.

At least this mistake in a "monolith" (hate that word) is easier to undo.

Yeah, in all micro service architectures I have worked in people have taken shortcuts which ended up in concerns spread all over different services.

This misstanke can be done there just as well as in monoliths.

An async deadlock in a buggy library will under load exhaust the process threads and take down everything. One bug can cause large objects to be allocated, freaking out the garbage collector, causing increasingly large lag spikes under load, and timeouts in seemingly unrelated code paths. A configuration change affecting only one part of the system can prevent the entire process from starting up. Modularization is one valid way to increase resilience by limiting blast radius.

You can fork(2) in monolithic applications if you want address space separation.

Throwing in the Self-Contained Systems approach for your consideration.

Not necessarily the One-Architecture-To-Rule-Them-All (none is), but as you've mentioned a "new LARGE project", it might be a good choice.

See https://scs-architecture.org and https://manuel-kiessling-net.translate.goog/2016/01/02/die-a... for more information.

This is great, thanks very much for sharing

> The idea is that if the other area of code simply isn't accessible to you, you're forced to keep the concerns separate? Seems like this could be dealt with by applying more discipline and just making sure to design things to be more modular and.. separate. This is the hallmark of any good code base. Do we need an entirely different architecture, with all the pitfalls that microservices entail, because we can't seem to keep things clean and separate?

Yes, because code is written by humans, and humans make mistakes. "Just have discipline and write good code" doesn't scale, especially in, as Uber's case, quickly growing organizations where large numbers of people new to the codebase are contributing.

What is it about network calls that makes it easier to scale separation over function calls. To me the RPC you choose should be independent from the separation part of it.

I'm not sure why it's suddenly so difficult to take down a monolithic application. It's totally doable and isn't that difficult. It happens all the time too.

For me, the funny thing is I've found that it can be just as easy for a single broken microservice to bring down everything else, and it can be a huge pain figuring out where such a problem originated from because the wrong services are erroring out.

> Serious question: why though? I can't think of a backend system where a single regression could bring the whole thing down. If something blows it up, it rarely takes the whole application down. What's this scenario exactly? Maybe I'm only thinking of certain types of applications, but a single failure somewhere needn't bring the whole thing down.

Suppose you have a web based CRUD that processes HTTP requests. To do this, it connects to a DB. There is one thread pool for HTTP requests, there's another thread pool for DB connections. Suppose there's an issue that causes a DB connection to hang (either a really slow query, or an issue with the DB thread pooling solution itself). All of the sudden, the DB thread pool is slowly getting exhausted, until no more queries can run because everything is hanging. Soon after, the HTTP request thread pool fills up, because requests are waiting for processing but none is being done. Users either see their requests frozen, or, if you're packaging both the API and the front end resources in the same deployment unit, soon nobody can even load the static resources of the page. I've seen this happen with a Java web app that used an old DB pooling solution that caused a prod outage years ago. Had to rip it out and replace it with something else.

That said, the same could happen with stuff like scheduled processes conflicting, memory exhaustion, CPU utilization, disk usage and so on. Now, there shouldn't be anything preventing you from running multiple instances of monolithic codebases in parallel, though sometimes systems aren't written with that in mind (e.g. no DB queues/RabbitMQ for messaging/scheduled processes, no assumption of multiple instances accessing the same data, in-memory caches instead of Redis and so on) and you end up with a singleton application instead, which is worse than just a monolith. Either way, planning for components/systems failing and being able to recover from that is almost always a good idea (health checks, automated restarts, some horizontal scaling, uptime notifications and alerting, some instrumentation like APM), unless you're really short on time.

None of that means that you should start with microservices though, unless your whole team is really comfortable with that. Monoliths are okay, as are modular monoliths (same codebase, deploy different features for app instances with feature flags, like FEATURE_ENABLE_API, FEATURE_ENABLE_SCHEDULED_PROCESSES etc.).