With microservices, it is easy to see services which are down or have high error rate or latency, have clear API contract and call out the team for breaking API contract, and assign cost for which the teams have incentive to reduce, or at least not increase it.
Large monolithic repos with many independent targets for testing and deployment work the best at huge scales. If you are only a few hundred engineers, monorepo with monolithic deployments and tests work fine.
This is not a commonly known fact. Just to take an example of GitHub, this check is disabled by default:
> Require branches to be up to date before merging
> Whether pull requests targeting a matching branch must be tested with the latest code.
I'm sure this happens occasionally, but I've never experienced it, and it seems to be rare enough that it's not that big of a concern. Especially since it'll be easily remedied by either just fixing the error or just reverting one or both of the changes.
Because this really doesn't scale well with the number of developers.
For larger teams the solution is to use a merge queue, e.g. https://shopify.engineering/successfully-merging-work-1000-d...
I haven't tried it, but GitHub is now offering such feature in public-beta: https://github.blog/changelog/2023-02-08-pull-request-merge-...
When committing, do a ff-only of ‘main’ to your branch. Yes, this forces everyone to rebase before “merging” but in practice, this results in the least amount of failures, tests being run after you resolved any conflicts, etc.
If you can use GitHub merge queues, that solves a ton of this, and you can run tests on the final merge before actually merging instead of relying on rebasing.
This. It makes life so much simpler. With teams that don't have a lot of experience with git, however, I tend to use the "Squash and Merge" feature, coupled with forcing a linear history.
Alternatively, if you have a low enough merge volume, requiring mergers (by policy) to squash and rebase (and re-run tests before attempting to merge) can work too, as others have already mentioned.
2. Add a test with a network dependency, and when that dependency is slow / down / turned off, the test starts failing.
3. Add a dependency on a third-party Github repo that clones from `main`, and the next time some dev touches a file in that repo your test starts failing.
4. Add a test that allocates memory in proportion to size of the codebase (e.g. because it tries to build a giant in-memory tarball of all the .mp4 assets). Eventually it will get flaky when it starts scraping up against the build machine's limit. Extra fun if your builds run without defined memory limits on machines of different sizes.
In a monolithic build, there's all sorts of ways for a single person to cause other teams' tests to fail, even months or years after they've left the company. Some of them can be prevented mechanically (such as by running tests without network access), but a lot come down to "tell them to stop doing that".
That's why big companies never run one build per repo.
IRT (2), network dependencies were forbidden in-general. Over any long enough timespan, the rate of failure is 100%. If you wanted to use the network, you had to consider the failure case and handle it in your tests.
For (3), all dependencies were committed as part of the repo. All dependencies had to be reviewed for any issues before being used, so this made sense. You simply weren’t allowed to just randomly include a new dependency without a review/PR to add it.
For (4), our dev environments had less memory than build machines and the same as production. If you couldn’t build it in a dev environment, it wasn’t getting committed without special treatment from dev ops (and a really good reason).
A monolithic build means that your ability to develop and deploy your team's code is dependent on every other team. As the number of teams gets larger, that multiplier really hurts.
There were a thousand automated checks to prevent you from doing the same thing as someone else that caused downtime in the past. It was virtually impossible to commit code that deleted/truncated a table, for example.
The last thing I want to do is have my build broken by someone on another team and then have to track them down and babysitting the revert. That is easily an hour of my time wasted.
2. How about an old way of blocking deployments less: parts of this monolithic apps developed as libraries with a stable API, a new version of a library released only after its own tests has passed, next you can increment dependency version in the monolith and run integration tests, if they failing you can revert to the old library version and still go ahead with the deployment (if you don't depend on something added in the latest library release). If you depend on this new feature and a component providing it is broken micro-services would not help you.
> If you are only a few hundred engineers, monorepo with monolithic deployments and tests work fine.
And here lies a very important problem IMHO - many (if not majority) of organizations (which do at least some software development) have less than 100 software developers but the industry best practices (which include micro-service architecture) are defined by FAANG-sized organizations and at least some of these practices are sub-optimal for small shops.
You can just... not allow code not passing test into master branch. They can fuck around in their own one, that's what branches are for
That's my experience at least. Things still break, you just notice it later.
That is a good point about reliability and cost though. I hadn't heard that before.
Google "continuous profiling".
I'm not sure why you would think that that reinvents microservices.
Microservices is just taking a monolith and moving the components into separate processes that communicate via RPC.
> Microservices is just taking a monolith and moving the components into separate processes that communicate via RPC.
Microservice architecture divides the responsibility much more than that. They have separate redis cache, local cache, tests, and even likely has different DB etc.
I also think it would be way healthier if teams acted as "maintainers" rather than "sole developer" of a service.
For example if team A wants feature from service team B manages they should be free (after communicating that so there is no confict/work duplication) to just make that feature and submit pull request to the team B.
Then team B can make sure it's up to their standard but that's shorter work than getting the whole machine of "submit a ticket for team B to add feature, find manpower to do it, and schedule work" running.