Comment by bradhe | Hacker News Reader

Comment by bradhe | Hacker News Reader

I work on a massive monolith that has about 800 contributors and its just as complex to add something as simple as a user’s birthday, just not all the complexity is technological. It requires “organizational alignment” since you’re touching everyone’s code.

There will be endless iterations on design and review. Sign off required by at least 2 architects. It will get added to multiple planning iterations. The actual code will take an afternoon or less. We’ll have to ensure we hit out 90% test coverage during code review but because of all the tests, it’ll be too big for one PR so it will need to be broken up into multiple PRs probably landing over multiple weekly releases. To facilitate that, we’ll put it behind a feature flag (of which there are currently 13,000). Once it hits production, and the dashboards/monitoring are put in place, it will get enabled and disabled over and over again as we’re not totally sure why our birthdate feature broke the metering service, but we think that’s the root cause—need to do a few weeks of analysis.

Then, finally, in a year the engineer who was tasked with it will get a good performance rating, maybe be up for a promo! Just in time for him to jump into another project that’s failing horribly headed into its 3rd year in development: Allowing the user to set their timezone.

This comment shows IMO that the real issue here is not really microservices or not microservices, but what the article calls "The apostles of the Church of Complexity".

Neither microservices or monolith are a golden hammer, silver bullet or whatever (nor the opposite). They are tools each with their tradeoffs, which combine with the many context-dependent tradeoffs of each organization. They are not the problem.

The problem is a) engineer's fascination with complexity, and confusing "simple" with "hack", b) how organizations keep cargo-culting tools, architectural patterns, and so forth. Applying architectures, design patterns, whatever naively, based on the belief that usage alone will deliver benefits. It doesn't.

I don’t think complexity is usually intentional though. Really, writing complex code is easy. It’s writing simple code that’s hard.

I’m in the process of wrapping up a reactor of one of our systems that I wrote a few years ago. Largely the refactor has been an exercise in just simplifying everything. The only reason I can effectively do this now is because we have a much better idea of our use case and how the entire system fits together. At the time, the complicity seemed necessary, though I’m not quite sure why. I think I just didn’t really understand what exactly this was really going to do when we were building it.

I think this probably applies to companies growing quickly as well. Beyond core product, there’s a lot of things you can do, but what benefit does it bring? How does it fit into overall strategy? It’s easier to build something if you just carve off your own little area to ignore the rest of the system and assume you’ll eventually understand better how of make it integrate nicely or add value.

Some people who advocate for huge monorepos, like Google has, tend to forget that Google has whole teams building tools just to wrangle the huge singular codebase for refactoring and testing.

Bingo. Monorepos work because tooling makes it work. Just putting all your code in one place doesn't make it a monorepo--just makes it a mess.

It was a mess before, but the mess wasn’t visible. Now the code is all visible to the IDE, it all compiles together, and tests together. You can see the mess and fix it. It’s possible to see enough of it to be able to change it and when you do the build stays green because you can make atomic changes.

It’s less about why monorepos are good, and more about why on Earth would you split a product up over hard repo / tooling boundaries. Perhaps a Big Brain could get those boundaries right. I can’t, so when I need to move the boundary I would rather it was done by editing code inside a repo than coordinating changes between micro products.

I’m obviously describing one scenario. What are the counter examples for where hard polyrepo boundaries are helpful? When your team are all IC1s?

> You can see the mess and fix it.

The trouble is, you're often left unwinding a half-dozen layers of abstraction, and in many cases experiencing what looks like some sort of quantum code interaction, where changes in one part of the codebase seem to impact completely unrelated bits of code which now fail tests.

The polyrepo + microservices approach helps enforce boundaries. It also makes rapid iteration easier with more limited tooling, because my code can no longer blow up someone else's code -- a problem which can be solved with dev toolchains, but those toolchains take a non-trivial amount of resources to support.

I do agree, when you need to move boundaries, or need to refactor across multiple packages at once, a monorepo + monolith is great. (A monorepo + microservices can introduce many of the same problems as a polyrepo + microservices, though.)

> The trouble is, you're often left unwinding a half-dozen layers of abstraction, and in many cases experiencing what looks like some sort of quantum code interaction, where changes in one part of the codebase seem to impact completely unrelated bits of code which now fail tests.

This perfectly sums up all of the microservice apps that I’ve had the pleasure of working on, with the exception of issues being caught by tests. Typically you get either silent production data corruption or a seemingly unrelated error in the UI x weeks down the line.

People seem to think Google’s monorepo is amazing, but I was often blocked from deploying anything because some team I never heard of submitted a changelist that broke the world. We should be using known-good dependencies rather than alpha testing the bleeding edge.

Bingo.

At $work i see lots of projects leaving the monorepo due to that very mess.

The problem is that services outside have other CI and maintenance issues that wouldn’t happen in the monorepo. But monorepo got so messy that it got unbearable.

The proper solution would be to have a dev teams dedicated to tooling for it, the org has the resources, but it’s just not investing enough in that area.

This sounds like you work at a place that’ll bungle any tech stack paradigm.

It sounds like this company doesn't have a competitive need to ship code changes, but when it does (i.e. a challenger appears or the moat disappears) it'll be trouble.

I've tried--with varying levels of success--to really take to heart that software engineering is a never ending battle against complexity. You have to do it all the time and it has to be your paramount value, otherwise stuff like this happens. I don't think there's an architecture or ideology that ends the war; this is just the nature of the job.

Having 13,000 feature flags is just insane. I'm so happy that at my previous job I insisted that we use "release" flags for every new feature were there wasn't an explicit requirement to be able to enable it for some customers and disable it for other ones. We made it so that you couldn't enable one of these release flags in production, and developers were required to remove the code for the flag before final tests and release. It requires a little more work for every feature which requires a flag, but it really pays off in terms of reduced code base complexity in just a few months.

It is indeed insane, but there is some value to leaving coarse feature flags in production. I've seen outages resolved by flipping a feature flag that was accidentally left in production months after deployment, where without it we would've had to resort to more drastic or hackish measures to restore service. (Eg. If one query is overloading the database, you can disable the feature that runs the query and investigate the real problem at your leisure while the database stabilizes.)

That only works in moderation, though - with 13k flags, there's no way the particular combination that you're about to release directly on production has been properly tested.

This is also known as a "kill switch" which is technically slightly different than feature flags that get used for release safety.

You can bake kill switches into many features as a core part of the application for operational reasons.

One similar thing I like is having global "maintenance" switches so you can gracefully put an application into maintenance mode.

Yeah, fair enough. At some point my ex-employer wised up and implemented kill switches on purpose instead of by accident.

That sounds crazy. Imagine how much worse it would be if it was a microservices system!

"Why is it so hard to display the birthday date on the settings page? Why can't we get it done this quarter?"

"Look...I'm sorry, we've been over this, it's the design of our backend."

https://www.youtube.com/watch?v=y8OnoxKotPQ

Totally agree. I don't see how microservices would solve this. Also monorepository doesn't mean that service is monolith.

You (correctly) assumed that it's a monorepo, by the way. But yeah, you're being reductive overall. It's not a "microservices vs monolith" debate. I was using this example to show that people can fuck up any model.

I agree that devs can and regularly do fuck up anything they touch.

OMG this sounds painful ;) both in processes and state of the codebase

I can ship something in a couple of hours on a monolith where 3000+ engs work and deploy every day, but to be fair that's not what I've seen in any other company I worked at.

Where do you work at, and, are you hiring?

Ditto.

Ugh that would make me hate software.

Sounds like DB2?

Sounds like DB2.

I think this is a great point, and shouldn't be just hand-waved away like you hit some bizarre edge case of Monoliths.

Monoliths can have crippling downsides -- just different flavours of downsides from Microservices. What you gain in network latency and DRYness, you can lose in Autonomy and Breadth of codebase.

Microservices vs Monoliths - just like everything else - is a question of tradeoffs, and making informed choices about when to apply them, and how to mitigate their downsides.

The slightly more nuanced point in the OP's article is that adopting any engineering practice and blindly following as though your identity is linked to it, is a bad move.

Reply on news.ycombinator.com