Hacker News Reader

As is often stated, microservices is a solution for scaling an engineering org to 100s of developers, not for scaling a product to millions of users.

You can get the separation benefits of microservices in a compiled language with modules that only communicate over well-defined interfaces, constraining each team within their own module without having to introduce a network call between each operation.

Python dev is cheaper and faster though. People arent gonna kill velocity by making their backend in c++ so the devs can have seperation of concerns, something that can, and should, be self enforced with discipline

Java or C# is a nice middle ground. But even in python you can enforce said separation - one module can only import from itself, libraries or any other module’s “services” object, and must export its functions in its own “services” object.

Microservices is bad for teams without discipline to implement "separation of concerns". They hope that physical network boundaries will force the discipline they couldn't maintain in a single codebase.

While microservices force physical separation, they don't stop "Spaghetti Architecture." Instead of messy code, you end up with "Distributed Spaghetti," where the dependencies are hidden in network calls and shared databases.

Microservices require more discipline in areas like:

Observability: Tracking a single request across 10 services. Consistency: Dealing with distributed transactions and eventual consistency. DevOps: Managing N deployment pipelines instead of one.

For most teams Modular monolith is often the better "first step." It enforces strict boundaries within a single deployment unit using language-level visibility (like private packages or modules). It gives you the "Separation of Concerns" without the "Distributed Spaghetti" network tax.

> Observability: Tracking a single request across 10 services

I'm not sure if this is a discipline issue in the way that domain driven design, say, is a discipline issue. If you instrument requests with a global ID and point at tool at it then you're basically done from the individual team perspective.

Uh, that's not my experience at all.

Sure you can say e.g "this property wasnt set in this request while being processed by this service managed by this team", but why it wasn't set will inevitably need multiple teams, each doing in-depth analysis how such as state could've been caused because they always inevitably become distributed monoliths - the former is being provided by the instrumentation, but the latter isn't (and even the former is not perfect, as not all frameworks/languages have equal support)

> and shared databases.

According to my understanding this is one of the reasons why microservices were invented, to prevent shared databases?

Unfortunately that message was way way behind the bombast of "microservices everywhere now" that preceded it for years, to the detriment of many small orgs.

I've seen engineering orgs of 10-50 launch headlong into microservices to poor results. No exaggeration to say many places ended up with more repos & services than developers to manage them.

The worst I ever saw was an engineer misunderstood microservices, and made a service per endpoint.

He started complaining to management that 50 CI/CD setups was his limit he could support.

He was absolutely amazed when I showed him he could combine endpoints into a larger logical service. 50 services became three, and it’s still three a few years later now.

Nanoservices

I once worked with a startup that had 3 total engineers, and their architecture diagram called for 8 micro services. We tore that architecture diagram up pronto

Out of interest how do you get the authority to make those decisions and have the existing developers continue working productively after this?

To me it seems like microservices (or cloud, or whatever) is often overused for career/buzzword reasons. The engineers pushing for it aren't asking for your advice, they want to build an engineering playground - denying them the opportunity is unlikely to suddenly make them productive at driving the business forward with a simple stack when their original idea was to play with shiny tech instead.

The only way I see out of this is to have management buy-in to get the microservices and their developers out the door, replaced by more competent people.

> their original idea was to play with shiny tech instead

This is a behavior I would say is very hard to manage out of people and should be screened for aggressively in interviews.

This behavior is self-inflicted by a decade of low pay and lack of significant raises to reward seniority.

The most effective way to increase income for a developer is to join a place, rack up as many buzzwords as possible and leave after 2-3 years, using those buzzwords to secure a higher-paying role somewhere else. Rinse and repeat until you get a management position where you can use politics to increase your income instead.

If you want guys that use boring tech to drive the business forward you have to pay them upfront the money they’d otherwise make playing the above game. It still makes sense (an engineering playground is anything but cheap) but good luck getting an employer to pay anything above “market rate”.

The time up to and including Covid saw massive developer salary increases. They've dropped (and lots have been laid off) post-Covid, but the last ten years cannot be described as stagnation.

The salaries may be high in absolute terms, but they’re still low in relative terms - compared to what jumping ship would give you when you have a resume full of buzzwords.

I wouldnt describe the 2010s as low pay for devs

Indeed Youth is wasted on the youth

> Out of interest how do you... have the existing developers continue working productively after this?

I don't know if you can in general - by happy coincidence the engineer who had made that plan found his way to the door shortly thereafter

do they all manage to have 4 different ways to do something like "notify user about x", all In use because they could never be bothered to complete the "upgrade"?

That's often the case yes. In a monolith a developer disgruntled about the situation can clean up the mess in a weekend, test it and push it through. No chance of that happening in microservices - you'd run out of weekend just opening PRs in the dozens of repos and dealing with all the nitpicking and turf wars.

I've had the unfortunate experence to run into a situation where two dev's who hated eachother ended up building two systems with one passing the customer's API calls' http content directly to the front end... It was supposed to be back end and front end.

Exactly the problem yes. Once you have more services than developers, you are probably running into infrequent releases and orphaned projects.

So whenever an inevitable common utility improvement is made, the effort of pushing out 100 repo releases for services no one has touch since Jim left 3 years ago is terrifying.

When there is a breaking change is going to be made and you HAVE to do the 100 releases, it's terrifying. Everyone says it never happens, but work on a project/team for 5 years and it does eventually, even once is enough to swear me off this "architecture".

The best descripcion of microservices comes from "The Grug Brained Developer" (https://grugbrain.dev/):

"grug wonder why big brain take hardest problem, factoring system correctly, and introduce network call too

seem very confusing to grug"

Grug actually covers this in his essay:

> note, this good engineering advice but bad career advice: "yes" is magic word for more shiney rock and put in charge of large tribe of developer

Microservices definitely contribute to having a "large tribe of developer" to manage.

Obligatory Krazam:

https://youtu.be/y8OnoxKotPQ

And to add to this: virtually every programming language allows you to define multiple entry points. So you can have your workers in the exact same codebase as your api and even multiple api services. They can share code and data structures or whatever you need. So, if you do need this kind of complexity with multiple services, you don’t need separate repos and elaborate build systems and dependency hell.

> Autoscaling is a solution to the self-inflicted problem of insanely-high cloud prices, which cloud providers love because implementing it requires more reliance on proprietary vendor-specific APIs. The actual solution is a handful of modern bare-metal servers at strategic locations which allow you to cover your worst-case expected load while being cheaper than the lowest expected load on a cloud

And how do you predict with certainty your "highest expected load"? And if you're in a space like ecommerce, where you have 1 week out of the year with x10 or x50 the load, I doubt it would actually be cheaper than using autoscaling. Especially today, with the costs of memory and storage. Not to mention that whenever you hit your load maximum, you have a few months of lead time to get extra capacity.

And FYI, "proprietary vendor-specific APIs" sounds very scary, but if you think about it for a few seconds, those APIs end at configuring an autoscaling group which is mostly about your min/max, and scaling rules. Yeah, it's proprietary, but it's 3-4 parameters to configure based on what you need, and from then little if any adjustment is needed. And you can take the same logic and port it to any other cloud provider within ~10 mins at most.

> And how do you predict with certainty your "highest expected load"? And if you're in a space like ecommerce, where you have 1 week out of the year with x10 or x50 the load, I doubt it would actually be cheaper than using autoscaling.

If you know when that week is, where's the problem in spinning up extra capacity just for that period, a week in advance?

Retailers, whether online or not, know with a very high confidence what their expected load is going to be a week from now.

But you have to buy those servers, set them up, use them for that week, and then they're left idle for the whole of the year.

You don’t need to buy them, you can get them from AWS for that period only. Doesn’t mean your whole stack needs to be on AWS.

Doesn't mean, it significantly complicates state. Because the place that stores it (say, your bare metal machine with your database) is now the bottleneck.

None of the autoscaling solutions presented in the post address the database issue either.

I agree that in general the database is always going to be your bottleneck and autoscaling appservers is generally bullshit.

But let’s assume your DB is fine and you do actually need to scale appservers beyond what your provisioned capacity can allow, you can trivially spin up VMs at AWS or your provider of choice, Wireguard to your “VPC” at your bare-metal provider (or just SSH tunnel the ports) and run your appserver binary/container/etc on it.

Hell in a pinch you can do the above with your dev laptop and serve traffic from it. It will work and your customers won’t even be able to tell.

> But you have to buy those servers, set them up, use them for that week, and then they're left idle for the whole of the year.

I didn't say you have to buy them. I said you can spin up extra capacity.

I was reading it and got seriously confused by separate database and server for a measly 1000 users. With the two separate you can scale vertically to handle a million users if all you’re doing is basic web/rest type stuff, probably more.

I feel a bit of sadness for people who had never used a bare metal server and seen how insanely capable hardware is today.