Hacker News Reader

Comment by bedobi | Hacker News Reader

I've never understood all this drama. In my experience, monoliths are universally bad in every imaginable way. A huge part of my career has been splitting monoliths into cohesive smaller codebases which could be called microservices.

How to split them has never really been a problem, you tend to develop an intuition and the monolith kind of splits itself.

Eg, currently we have user profiles (user id > the users names, email etc) in one microservice, relationships between users (user id > other user ids) in another.

Each has its own datastore, deployment, scaling etc. It works great and while there is a small overhead cost to pay for splitting things it's vastly preferable to the monolith we used to have, which contained lots of things that had very little to do with each other, took ages to test and deploy, if it went down it was game over etc etc etc.

View on HN

> A huge part of my career has been splitting monoliths into cohesive smaller codebases

Did that ... create business value for your company?

> Eg, currently we have user profiles (user id > the users names, email etc) in one microservice, relationships between users (user id > other user ids) in another.

This sounds like a parody of microservices.

Yes, it did create business value.

* our enterprise db was bursting at the seams containing Literally Everything. Now, every part of the split up monolith has it's own self contained data store tailored to what is appropriate for that particular thing. (some use MariaDB, others Redis etc etc)

* developing, building, testing and deploying took ages. Eg if I only needed to capture some new detail about a business partner user (eg their mfa preference app vs sms) I would still have to do deal with the unwieldy monolith. Now, I can do it in the dedicated business partner user service, which is much easier and faster.

* the whole monolith, including business partner facing operations, could go down because of issues to do with completely unrelated, non critical things like eg internal staff vacation hours.

I could go on.

As for the different services I described, if most of the callers did need both pieces of data it would have made sense to combine them into a single service. But, overwhelmingly, callers are interested in one piece of data or the other, and the load profile, tolerance for staleness in caching etc etc for each of the two services are vastly, vastly different. This is why we chose to split the two into different services.

The few callers that do need to obtain both pieces of data just make concurrent calls to both and them zip them into a single result.

> * our enterprise db was bursting at the seams containing Literally Everything. Now, every part of the split up monolith has it's own self contained data store tailored to what is appropriate for that particular thing. (some use MariaDB, others Redis etc etc)

Why do you consider an enterprise DB "bursting at the seams" to be a bad thing? Isn't that what enterprise DBs are built for? Seems like you traded having everything in one large database to having everything scattered in different databases. You probably sacrificed some referential integrity in the process.

> * developing, building, testing and deploying took ages. Eg if I only needed to capture some new detail about a business partner user (eg their mfa preference app vs sms) I would still have to do deal with the unwieldy monolith. Now, I can do it in the dedicated business partner user service, which is much easier and faster.

You traded a clean codebase with a solid toolchain for probably a template repository that you hope your users use or everyone is reinventing some kind of linting/testing/deployment toolchain for every microservice

> * the whole monolith, including business partner facing operations, could go down because of issues to do with completely unrelated, non critical things like eg internal staff vacation hours.

This could apply to any software. Sure, a monolith can have a large blast radius, but I can guarantee one of your microservices is critical path and would cause the same outage if it goes offline.

> The few callers that do need to obtain both pieces of data just make concurrent calls to both and them zip them into a single result.

Almost like a database join?

Who says the monolith we had was a clean codebase with a solid toolchain? It was a giant piece of software that had existed since the 90s, with hundreds of developers working on it at any given moment, thousands of people coming through it over the years, full of bizarre things that would make anyone scream and tear their hair out. (each of which may have been the right thing to do at the time, but combined over the years wasn't optimal to say the least)

It's obvious nothing will convince you, but I maintain that eg internal vacation hours should not live in the same codebase as partner facing business critical things.

You can very well have a bad monolith and still think micro services is a bad idea, and his concerns especially around referential integrity are perfectly valid and valuable. Maybe all you needed is to fix the existing monolith incrementally?

Of course microservices should be sliced with referential integrity in mind. But a monolith by definition has everything, including utterly unrelated things that have absolutely nothing to do with each other, bundled up into a single, giant, well, monolith. What referential integrity could there possibly be between eg business partner user profiles and internal staff vacation hours?

There is nothing inherently bad about storing unrelated things in the same db, in general.

There might be security issues but good DBs offer security grants that are granular enough to deal with this.

Data sovereignty might be an issue forcing you to split, that depends on the domain and application requirements.

But in general two things being unrelated isn’t a reason to split the DB and splitting the DB isn’t a reason to split into services.

I’m not for or against microservices. I’m definitely for good reasoning though!

> Who says the monolith we had was a clean codebase with a solid toolchain?

You didn't. They're just trying to force the issue for some reason by straw manning your decision.

No offense, but someone coming in and splitting out a database into multiple bespoke data stores because the primary database is "bursting at the seams" is basically my nightmare. If your description of your reasoning is accurate, it's a completely broken mental model.

Well, that's not what happened. The whole org from devs to the board were highly aware of the problems and knew and agreed what had to be done. I'm just one of the many people who architected and implemented the change.

Could you be specific about the size of your company and the size of the data you're processing? And could you also be more specific about the business value it created? Are you an investor in the business? Is it your own money on the line?

Publicly listed, ~13 billion USD revenue, 22k employees, millions of b2b partners and users.

When the monolith goes down, it's hundreds of thousands, sometimes millions of dollars of revenue lost both during and as a result of an outage.

You really, really don't want a problem with eg internal staff vacation hours to do that.

Ok, interesting. That kind of scale is not exactly the norm though, so I don't agree with your implication that monoliths are always a bad idea.

I really don't know why anyone who wasn't so resource strapped as to have no alternative would want their $$$ services sharing a build-test-deploy-rollback lifecycle or database with the internal staff vacation hours tracker.

It seems like the real solution here is to outsource vacation hour tracking to one of the HR apps for the purpose (see Workday) instead of having the domain expert engineers split the company’s application apart into micro services so they can have a custom vacation tracker.

It was just one of the worst examples. A more obvious case would be the different lines of business. They have very little to do with each other and shouldn't share code or data. Another is business partners vs end users (regular people) - there's a lot if reasons, including legal ones, why they shouldn't share code or data.

That makes sense, but is nowhere near universally applicable. For folks working in most startups, talking about different lines of business is non-sensical - there's generally one product.

It sounds like you're describing a monolith that actually contained multiple completely independent applications - which I don't think anyone would disagree with being a good case for splitting.

In most cases I've seen, the decision to split into microservices usually involves a fairly high amount of dependency between different services to achieve a common business goal (hence the concerns about things like referential integrity)

> > A huge part of my career has been splitting monoliths into cohesive smaller codebases

> Did that ... create business value for your company?

Not OP, but the big value of splitting into microservices is isolation.

In production, this isolation offers a limited blast radius in the case of an errant service. Also, independent scaling. Business value: improved reliability.

In code, isolation lets development teams have a smaller and more focused domain / set of concerns to reason about (vs. the entirety of the monolith). Business value: Increased dev velocity.

In my experience, it's easy enough to have services which have a large blast radius themselves and can become points of failure for your entire ecosystem. I don't find this to be a huge point of difference from a monolith, although of course it depends a lot on what you're working on. To pick on the OP a bit here (sorry!), they said that their entire legacy system "could go down because of issues to do with completely unrelated, non critical things like eg internal staff vacation hours." To me that sounds like poorly written software regardless of architecture. I can't imagine a scenario in any of the recent codebases I've worked on (microservices and monoliths both) where errors in what sounds like an internal CRUD tool would cause an entire production application to crash. I find it even harder to imagine if the application has a halfway decent test suite.

To add to that, when you have hundreds of services running around and something goes wrong, it ends up being a lot harder to track down exactly what's happening. So when you do get that critical error, oftentimes the downtime is worsened.

As for dev velocity, I find the claims of the microservice gospel a little bit exaggerated. Your layers of nested services all talk to each other, and any of them could be a point of failure. This isn't really all that different from calling another function in your monolithic app - you've just distributed that function call across a network boundary. You still need to know the callee's API, and you'll still spend a decent amount of time trying to understand the ways that the callee might fail. But you've also created a huge amount of additional developer work whenever you need to do something that spans the boundaries of existing services.

I think microservices certainly have their advantages but a lot of the simplistic claims made by their biggest proponents only hold up prima facie.

> In my experience, it's easy enough to have services which have a large blast radius themselves and can become points of failure for your entire ecosystem. I don't find this to be a huge point of difference from a monolith, although of course it depends a lot on what you're working on.

It's a huge different because if some core critical service starts causing problems it's almost certainly because the last binary push was bad, and you roll it back. You only have to roll back that particular service any everything starts behaving correctly again. Moreover, you probably detected the problem in the first place when the rollout of that service began by replacing a single instance of the updated service with the new binary. Monitoring picks up a spike in errors/latency/database-load/whatever and the push is stopped and rolled back.

Monoliths have inventive ways to address this problem without having to roll the entire binary back, like pushing patches or using feature flags, but few would argue that the microservice approach to handling bad pushes isn't superior.

> To me that sounds like poorly written software regardless of architecture. I can't imagine a scenario in any of the recent codebases I've worked on (microservices and monoliths both) where errors in what sounds like an internal CRUD tool would cause an entire production application to crash. I find it even harder to imagine if the application has a halfway decent test suite.

Easy enough with a sufficiently large codebase in C or C++. Somebody's parser encounters an input that was supposed to never happen and now it's off clobbering the memory of who-knows-what with garbage.

If you find yourself routinely pushing a bad binary that has (1) passed your test suite; and (2) passed whatever manual QA process you have on sandbox/staging deployments, then I would again suggest the problem is the process and not the architecture. Not to mention that if the service is indeed critical, you're not rolling out a deployment to every production server at once (or you're low enough scale that it doesn't matter). Either way, easy enough to roll back a bad deploy before things get hairy regardless of architecture.

Also, I'm not sure what kind of internal CRUD tools you're writing, but "malicious input" doesn't really seem likely to come from your coworkers.

Input doesn’t have to be malicious to cause problems, just unexpected. Separating memory-unsafe code into the finest granularity memory spaces possible is just good practice regardless of whether it’s microservices or just process isolation in a monolith.

When you're dealing with a giant piece of software that has existed since the 90s, with hundreds of developers working on it at any given moment, what you're describing doesn't work. You can read about a famous example here https://news.ycombinator.com/item?id=18442941

Well, you began by saying that "monoliths are universally bad in every imaginable way." You're now saying that a select few poorly-maintained, 25-year old monoliths are bad, which is an entirely different claim. That doesn't really prove that microservices are any better (are 25-year old poorly maintained clusters of services really going to be an improvement?). Let alone "better in every imaginable way."

How is a blast radius limited in the case where a bunch of things depend on that microservice? It seems a microservice can have an arbitrarily large blast radius.

The things that don't depend on it don't go down. Eg. Your email system doesn't go down because the building's elevator had a bug.

It also let's you choose which parts to pay closer attention to - the microservices that's depended on by everything gets the extra operations

If you have a high availability system, then one whole monolith instance going down is going to have less effect than one crucial micro-service going down.

That seems like a conveniently contrived example.

Why, in this example, would only 1 instance of the monolith go down but all instances of the crucial microservice go down?

Most of the time it's one dodgy parameter or variable (or one dodgy combination of variables) that causes something to crash. The majority of bugs I fix take a bit of effort to reproduce.

Exactly this!

Agreed, isolation is a good reason to opt for microservices. One app goes offline, others still accessible, and you can do isolated maintenance. That, and features that aren't directly related, are super lightweight (quick to build, run, low on memory and CPU) as little projects.

> Did that ... create business value for your company?

Image what a piece of crap Linux kernel would be if it was developed with this mindset. :-)

EDIT: The context is that there's constant code splitting code moving being done inside Linux.

> monoliths are universally bad in every imaginable way

In my experience, monoliths are simpler and often faster to ship a working v1.0. When that matters (which seems like it would be “quite often”), they are a short-term winner.

You have to survive the short-term in order to face the problem of the long-term.

> In my experience, monoliths are universally bad in every imaginable way.

I just imagined a different way. Monoliths can do transactionally consistent stuff.

Now you have a good rule of thumb for splitting services: stuff that must belong to a single transaction should live in the same service.

This is how you e.g. can split a bank transaction service from a bank balance-telling service, with different requirements for scaling, latency, SLAs, deployment schedules, etc. You can deploy a change to the balance-telling service and then roll it back if it exhibits a problem, all without touching transaction proceeding in any way.

These abilities are worth something if you're a big enough. But this is not a starting point for a new side project. You start with a monolith.

> You start with a monolith.

I think that is the HN syndrome; people start projects like they will be larger than facebook even though the project will probably never launch and if it does, get more than 10 users. Yet it runs all services aws has to offer and has 100 microservices and the little company(‘s investors) are paying through the nose in both dev time and hosting while a raw php script on a $1/mo vps would’ve been sufficient to validate the idea and get to ramen profitability and (far) beyond. Like someone said in another thread; focus on your market and acquisition channels first, build great stuff (much) later.

I remember that eBay seriously rewrote their backend three times — not because they kept doing it wrong, but because they grew.

An architecture that fits well for a small company is inadequate for a large company, and vice versa.

I bet there are orders of magnitude more cases where a ton of time and effort has been put into scalable architecture when it wasn't needed. Ebay is a serious outlier when it comes to scale of software, most people aren't going to serve anywhere near as many customers.

I've been through a couple of iterations of "MONOLITH BAD!" where none of the issues were monolith related but instead database design and usage. Also an iteration of "MICROSERVICES GOOD!" where there are more microservices than entire company employees (not just devs) and yeah, it's a bad idea at this scale.

I know it's a simple example, but wouldn't the balance of an account be critically important to be part of a transaction. Otherwise, how could you manage two simultaneous withdrawals if the transaction processing service isn't able to put a lock on reading the balance until it's complete?

The teller can show a slightly stale, eventually consistent balance figure, depending on its use. This would allow for massively better read performance, which is important when reads occur many times more often than writes.

Instead of a balance of a checking account, think about the karma balance of a reddit post, where this approach more emphatically belongs.

> Monoliths can do transactionally consistent stuff.

So can microservices, both internal to the service (via simple transactions) and between services (via distributed transactions, e.g., 2PC), though a good service design minimizes the need for the latter and is mostly guided by consistency boundaries as to where to draw service boundaries.)

(It's true that there are naive microservices architectures pursued that are essentially normalized relational designs with a one-service-per-table rule, but that's just bad—and usually cargo-cult—design.

Sure you can do transactions across services. Do you really want to deal with the problem of distributed consensus though? If software especially built to deal with it often gets it wrong (as I think Jepsen tests have frequently demonstrated), what are the chances your team is going to succeed and not build a distributed monolith instead?

I suppose in the end it depends on if you want to trade the hard problem of managing a monolith for possibly a much harder problem.

I find that for these discussions, the definition of a microservice is too nebulous. Personally I think that if you have "state" that crosses service boundaries (ie. a fault in either service causes the loss of that aggregate state) you have instead built a distributed monolith.

For sure, that's a factor that should be taken into account when deciding what to split and how. But monoliths always have completely unrelated things bunched up in a single codebase, storage etc etc etc.

Eg, in our case, there's simply no reason why we should have a single monolith service that has both our b2b partners user profiles and, I don't know, internal staff vacation hours?

They have nothing to do with each other, have vastly different load profiles, tolerance for failure etc etc.

I've seen monoliths screw this up. All it takes is some abstractions, maybe an ORM. Suddenly the novice programmer is too far away to notice they're running multiple queries outside a transaction without a simple way to fix it.

In fact, I think its quite common.

microservices can too, just most databases don't support this without an additional service layer. fwiw Spanner does.

> currently we have user profiles (user id > the users names, email etc) in one microservice, relationships between users (user id > other user ids) in another.

What do you do when you need to query data across these boundaries?

If most of the callers did need both pieces of data it would have made sense to combine them into a single service. But, overwhelmingly, callers are interested in one piece of data or the other, and the load profile, tolerance for staleness in caching etc etc for each of the two services are vastly, vastly different. This is why we chose to split the two into different services.

The few callers that do need to obtain both pieces of data just make concurrent calls to both and them zip them into a single result.

Sounds like you have a simple system, in which case the separation have few drawbacks. It's not uncommon for a customer to ask for a report that to be generated needs to span dozens of tables. Or to save a DTO object that will trigger multiple microservices (and a lot of validations, transactions, rollback, etc). Business rules can be very complex and entangled in some shops.

It's not a simple system, we have all those things and more. But when we make a service that contains partner user profiles data we optimize for that (=millions of partners accessing their profiles, has to respond immediately and scale infinitely, has to be easy and quick to capture new details like eg mfa preference app vs sms etc), not peripheral things like reports etc - such things have to live life being harder for them than they would be if we had a monolith.

Good thing if your data fit into a single database; sometimes they do not, because they are too big.

OTOH you can often export the data in different, more appropriate ways to make your joins more efficient.

I'd say it sounds like they have a distributed monolith :)

From your following comments I see that you consider a monolith a software that does "everything", from managing staff holidays to giving services to customers. But this isn't what monolithic architecture Vs microservices is about; if you have independent business processes, it's been completely normal forever to have independent "monoliths" to manage them.

Many proponents of microservices though request that you split your software into small independent services even when they manage the same business process, and this is where the complications come.

If it were up to Hackernews commenters, no software created after 1998 would be used - everything would run on a single monolith, use Postgresql, run with System V and written in, I don't know, Perl with CGI. It worked fine back then, why do we need to change it and make it all fancy and complicated?

I don't know the career histories of these kinds of people, I'm sure its varied, but I just can't imagine them working in very large public facing dynamic sites that update multiple times a day with monoliths thinking "this sure can't be improved! We have reached the peak of computing!"

I've seen microservices go really badly of course - I worked in a place where the devs insisted that two microservices need to go out "at the same time," as if such a concept existed, because they depended on each other. At the end of me working there, there were around 40 microservices, all Java so they needed at least 2GB of memory each, some 8GB, and at least 3 replicas for high availability. Cost a small fortune in servers for what really could have been like, 5 microservices, written in Go or Node and run on a handful of normal servers.

But microservices, like Kubernetes, are not _hype_, they are not a flash in the pan and they're here to stay because they are a good idea conceptually, even if they often aren't executed very well.

Having worked on both types of systems, old school way was far simpler, and bugs were faster to fix. Sure there are advantages to kubernetes, it will scale easily, but that is a problem that I have seen solved far more times than it has actually been needed.

If you're like facebook, it makes sense to split user profiles and relationships to two microservices, but if you handle a few tens of thousands of users it does not, so you're doing yourself a disservice by using a poor example to drive the point

You’ve given up on data consistency entirely, which is a significant problem with micro-services in general.

Do you run DR tests? I bet $TEXAS you will have orphaned relationships as your data stores will be restored from different points in time.

The issue (for me at least) is not the idea of microservices itself, it is the granularity at which services get defined/split off. What usually happens is that companies go all-in on the microservices idea, and force it upon all teams regardless of their size/head count. It is far easier for a 10 person team to maintain a microservice than it is for a 3 person team. Microservices impart some fixed costs to every team managing a service and thus make very little sense for small teams.

On the other hand, a 3 person team can much more productively contribute to a monolith. From my experience in the industry so far, taking into account the current quality of tooling, I would say it starts to make sense for teams above the 10 person mark to own their own service.

Maybe I don't understand microservices but what is to stop a team from declaring the whole monolith to be just one microservice?

I have interviewed at a few places over the last year and not one interviewer has given me a hundred percent guarantee that we will never have to allow access to the backing store (database) that isn't through our service.

Of course, microservice will fail in such environments. That's not a microservice fault. That's a defect in management.

> we have user profiles (user id > the users names, email etc) in one microservice, relationships between users (user id > other user ids) in another.

> Each has its own datastore

How do you maintain referential integrity between datastores?

For example, when a user is deleted, how do you update the relationships between users in a concurrency-safe way?

Reply on news.ycombinator.com