I'm pretty convinced that microservices are one of those things that make sense 5% of the time and the other 95% is cargo culting.
I haven't yet seen a project/product which would need microservice architecture for technical reasons. If you need to scale, you can just scale monoliths (perhaps serving in different roles).
The use case for microservice architecture is IMHO an organizational / high level architecture driven. I've worked in a big company (20K employees) which was completely redesigning its back-office IT solution which ended up as a mesh of various microservices serving various needs (typically consumed by purpose built frontends), worked on by different teams. There monolith didn't make sense, because there was no single purpose, no single product.
But if I'm building a product, I will choose monolith every time. Maaaaybe in some very special cases, I will build some auxiliary services serving the monolith, but there needs to be a very good reason to do so.
I did feel a bit embarrassed having to make a microservice after having argued against them so much over the years. Hopefully I can stop producing PDFs soon so I can delete the entire thing :P
Moved the monolith to .NET Core, kept the report service on .NET Framework. A win for everybody.
It's fine to just have plain "services" to do things like this where you need to leverage another OS/framework/whatever and just hive off something like PDF conversion while your core application remains a monolith.
Non-copyable build outputs sound a bit wild - you're thinking of builds that encode absolute paths into the output binaries?
"Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure." -- Melvin E. Conway
If you have a single team, you shouldn't be doing microservices.
But that just puts into perspective how silly this argument is because I have no idea what a project means to other people.
A single team managing 10 microservices that actually make sense to be microservices (like the PDF renderer example above [1]) is kinda good and perfectly manageable.
A team with one single microservice that would actually work better if it was part of a monolith is already in the "creates more problems than it solves" territory.
Having hundreds of engineers work in a single monolith in a single repo without any kind of (enforced) boundaries is a one way ticket to a big ball of mud. You need to invest heavily in tooling to make it work, and e.g. Google does so.
Having a network in between teams is a relatively easy way to enforce boundaries.
Naturally it could be a single container taking care of all those integrations.
With microservices, it is easy to see services which are down or have high error rate or latency, have clear API contract and call out the team for breaking API contract, and assign cost for which the teams have incentive to reduce, or at least not increase it.
Large monolithic repos with many independent targets for testing and deployment work the best at huge scales. If you are only a few hundred engineers, monorepo with monolithic deployments and tests work fine.
When committing, do a ff-only of ‘main’ to your branch. Yes, this forces everyone to rebase before “merging” but in practice, this results in the least amount of failures, tests being run after you resolved any conflicts, etc.
If you can use GitHub merge queues, that solves a ton of this, and you can run tests on the final merge before actually merging instead of relying on rebasing.
This. It makes life so much simpler. With teams that don't have a lot of experience with git, however, I tend to use the "Squash and Merge" feature, coupled with forcing a linear history.
Alternatively, if you have a low enough merge volume, requiring mergers (by policy) to squash and rebase (and re-run tests before attempting to merge) can work too, as others have already mentioned.
2. Add a test with a network dependency, and when that dependency is slow / down / turned off, the test starts failing.
3. Add a dependency on a third-party Github repo that clones from `main`, and the next time some dev touches a file in that repo your test starts failing.
4. Add a test that allocates memory in proportion to size of the codebase (e.g. because it tries to build a giant in-memory tarball of all the .mp4 assets). Eventually it will get flaky when it starts scraping up against the build machine's limit. Extra fun if your builds run without defined memory limits on machines of different sizes.
In a monolithic build, there's all sorts of ways for a single person to cause other teams' tests to fail, even months or years after they've left the company. Some of them can be prevented mechanically (such as by running tests without network access), but a lot come down to "tell them to stop doing that".
That's why big companies never run one build per repo.
IRT (2), network dependencies were forbidden in-general. Over any long enough timespan, the rate of failure is 100%. If you wanted to use the network, you had to consider the failure case and handle it in your tests.
For (3), all dependencies were committed as part of the repo. All dependencies had to be reviewed for any issues before being used, so this made sense. You simply weren’t allowed to just randomly include a new dependency without a review/PR to add it.
For (4), our dev environments had less memory than build machines and the same as production. If you couldn’t build it in a dev environment, it wasn’t getting committed without special treatment from dev ops (and a really good reason).
A monolithic build means that your ability to develop and deploy your team's code is dependent on every other team. As the number of teams gets larger, that multiplier really hurts.
There were a thousand automated checks to prevent you from doing the same thing as someone else that caused downtime in the past. It was virtually impossible to commit code that deleted/truncated a table, for example.
You can just... not allow code not passing test into master branch. They can fuck around in their own one, that's what branches are for
2. How about an old way of blocking deployments less: parts of this monolithic apps developed as libraries with a stable API, a new version of a library released only after its own tests has passed, next you can increment dependency version in the monolith and run integration tests, if they failing you can revert to the old library version and still go ahead with the deployment (if you don't depend on something added in the latest library release). If you depend on this new feature and a component providing it is broken micro-services would not help you.
> If you are only a few hundred engineers, monorepo with monolithic deployments and tests work fine.
And here lies a very important problem IMHO - many (if not majority) of organizations (which do at least some software development) have less than 100 software developers but the industry best practices (which include micro-service architecture) are defined by FAANG-sized organizations and at least some of these practices are sub-optimal for small shops.
This is not a commonly known fact. Just to take an example of GitHub, this check is disabled by default:
> Require branches to be up to date before merging
> Whether pull requests targeting a matching branch must be tested with the latest code.
Because this really doesn't scale well with the number of developers.
For larger teams the solution is to use a merge queue, e.g. https://shopify.engineering/successfully-merging-work-1000-d...
I haven't tried it, but GitHub is now offering such feature in public-beta: https://github.blog/changelog/2023-02-08-pull-request-merge-...
I'm sure this happens occasionally, but I've never experienced it, and it seems to be rare enough that it's not that big of a concern. Especially since it'll be easily remedied by either just fixing the error or just reverting one or both of the changes.
The last thing I want to do is have my build broken by someone on another team and then have to track them down and babysitting the revert. That is easily an hour of my time wasted.
That's my experience at least. Things still break, you just notice it later.
That is a good point about reliability and cost though. I hadn't heard that before.
Google "continuous profiling".
I'm not sure why you would think that that reinvents microservices.
Microservices is just taking a monolith and moving the components into separate processes that communicate via RPC.
> Microservices is just taking a monolith and moving the components into separate processes that communicate via RPC.
Microservice architecture divides the responsibility much more than that. They have separate redis cache, local cache, tests, and even likely has different DB etc.
I also think it would be way healthier if teams acted as "maintainers" rather than "sole developer" of a service.
For example if team A wants feature from service team B manages they should be free (after communicating that so there is no confict/work duplication) to just make that feature and submit pull request to the team B.
Then team B can make sure it's up to their standard but that's shorter work than getting the whole machine of "submit a ticket for team B to add feature, find manpower to do it, and schedule work" running.
A lot of people here say...one service per team. But to me that is, or can be, a monolith. Often a team is a product line, so you have one service for that product. Is that a monolith? I don't know either, I guess.
I -do- know most people who go around promoting that sweet microservice life end up being the worst. They seem to want every db table to be its own service, introduce tons of message passing and queues, etc, for absolutely no reason. I think we can probably all agree that is about the worst way to go about it.
But doing it right, is nevertheless hard. Because cutting your business into chunks ... is not as easy as it always looks.
This is why microservice costs often far outweighs the benefits, but they rarely consider the cost in their crusade to 'break up the monolith'
In my org in Google we average over one microservice per engineer. I'll be adding two in the next couple weeks. With the right automation setup you don't notice them any more than you do server instances.
The conversation went as below.
- Does your app work fine?
- Yes.
- Do you have any problems?
- No.
- Why do you want to migrate then?
- Silence.
Your job is to figure out what they actually need even if they don't understand it?
Seems pretty par the course.
There are dozens of reasons to migrate to the cloud. Do they apply to everyone? No. Are they always worth the cost? No. But the whole "cloud vs not cloud" argument that happened, got settled ("cloud"), and is now being restarted by the DHH-like is not data-driven and full of exaggerations and fear-mongering from both sides.
Then you add on top of that that the main product of moving to the cloud is "operations" which is typically measured in "hours of human capital being impacted outside of core working hours". When the market is booming, tech humans are expensive and fickle, and don't want to undertake more operations than they should have to, and companies are forced to pay cloud providers.
But in today's 2023 climate, any company looking around to decide how much to spend on cloud just says "Why would we pay for something when we can just ask our engineers to work more hours, and invite them to quit if they don't like it, oh wait nobody else is hiring anyway"
No cost calculator of $$$ saved considers that overtime is free in our industry.
tl;dr the cloud backlash is overblown, more companies/businesses would benefit from cloud than not.
Everyone wants to do the new cool thing. Everyone wants it on their CV. To be followed some years later by everyone saying how awful it is, and moving on to the next fad. Rinse, repeat, round and round we go with no actual intelligence being applied.
If you ask me, if the time and focus is invested properly, it would be much more efficient to run a monolith instead. That's what some small number of great teams end up doing.
"lateley" as in "for the last 5 years"?
If I could trivially package them up and deploy them locally with intrinsically less effort and wall-time then the old way, that'd be amazing.
If I could somehow get the horizontal scaling promises and redundancy as some kind of built-in, like I can with say, memcache, that's be cool.
If I could do these kinds of "hard" things with them more trivially, that'd be really nice.
There's a lot of things I want them to do but it's a god-damn bull-riding rodeo every time I try to get there.
And before you reply, I know you're an expert and can do all these things trivially. That's amazing. The vast majority of the industry creates a giant fragile spaghetti knot with them and I am not a full time k8s admin nor do I want this to be a career trajectory. It should be like you know, wine, ffmpeg, imagemagick, virtualbox, lua, qemu, redis, gnuplot, lvm2, gdb, ssh, sqlite; tools like that. It's pretty easy to get them to do really nice things. Those things deliver on their promises and potential pretty nicely.
It's nice that nobody feels a need to hype curl or squid. They just work. Isn't that nice? I mean look at gdb's website: https://www.sourceware.org/gdb/ it doesn't even have CSS animations --- in fact, it doesn't even have CSS.
I still recall the day when my local Docker builds necessitated a new router to properly manage streaming traffic at home while I downloaded a few GBs of layer images. Or the time I wanted to setup a 'simple' hosted Kubernetes cluster of my own in my lab for testing, only to discover the nightmare that is networking on it. Then there was the grim discovery that Docker containers were much more sensitive to the hosting environment than I had assumed, resulting in some fun "but it worked and tested fine" moments.
Did they all work eventually for me? Yes. Was it simple? Not by my standards.
If you find all that simple, kudos to you.
Like, compared to implementing hitless rollback over bare metal services k8s way is "easy", just set some stuff in YAML and have proper healtchecks in your app.
I wish I had infinite time to document all the issues. This isn't a small nuanced detailed thing - it falls deeply, systemically fundamentally short and in practice you still get the magical monolithic system it tried to kill but now with more obfuscation, complexity and a theatrical slight of hand to convince yourself it isn't that.
Instead of the server being configured for the monolithic app, it's now extensively and carefully configured for the myriad of containers, hostnames, configurations and connections of the containers running the microservice app.
It's in practice the same problem with a different costume.
The other promise of it being a collection of smaller constrained services running on tcp ports talking to each other ... that's nothing new. You've invented the idea of computer networks.
But then for perfect decorrelation you'd also need independent databases behind the microservices, and queues between them for horizontal communication,and few are actually going all in with that, and so fall in the 95% where they go trough te motion and the effort of splitting microservices,but reap no actual benefit from it.
https://en.wikipedia.org/wiki/Cargo_cult_programming#:~:text....
Currently supported on all non-Firefox major browsers. https://caniuse.com/url-scroll-to-text-fragment
Seems Brave is an exception amongst Chromium browsers. They don’t implement it for privacy reasons.
If you have a microservice first architecture, the perception is, it's easier to describe effort to re-write an individual service or split it into two services as there is a clearly delineated body of work. Bizarre service-to-service dependencies may still exist and a poorly implemented microservice architecture is still a potential challenge.
Point being, organizations incentivize bad economic decisions on the part of engineers through the inability to recognize that rework is a necessary aspect of developing software and by constantly eschewing rework in favor of feature delivery it sends a strong message to the engineer about what to prioritize.
The "pain" of figuring out how to deploy your "normal app" quickly amortizes over just how much easier and more reliable code is.
A simple example: I have a SPA that has the following features: auth(login, logout), dashboard, feature a, feature b. I can write a few very simple lambda functions and deploy these the same way (IaC). What do we (my team) win? We can implement each function in a language we want. You have a feature that is too slow? Rewrite it in Rust. You have an amazing Python lib for feature a? Use Python. What else? We almost never touch auth, so if a feature has a bug it does not impact the entire application. Security is better because we can allow individual functions to access part of the infra they really need to access. Lambda functions can call other lambda functions as well.
Downside is that we cannot use a shared cache that is easy with a monolith. People need to design the boxes well which functionality goes to which lambda function. We have to use distributed trace ids to track requests.
Basically cut down the cruft when deploying another small self-contained feature but still keep the code running (savings of few MB memory are meanigless if you just have few dozen features that might run at the same time anyway).
Then I realized it's basically reinventing the ancient idea of "application server" like JBoss and EJB... which is kinda the case for lambda anyway.
That assuming they are big enough to warrant that in the first place.
IMO if it is a dozen unrelated endpoints but all of that still takes less than dev/month to manage it's probably entirely wasted effort to "fix it"
And if any of them grows to warrant dev/month or more.... separate that one and only that one out of that.
My biggest grief with microservices is the fact that it's effectively become a war on having a coherent logical normalized relational data model inside an organization.
I think it's actually quite rare for companies to have data so actually autonomous and unrelated that it does not logically relate to anything else in the organization.
However, I think there is something hiding inside the µservices movement that is actually much more generally applicable and useful: API-first development.
And of course good old OO.
Micro-services were invented by an outstanding software outsourcing company to milk billable hours and offload responsibilities in large org.
If you want to save cost and not-that-large business, go for monolith-first. Keep it modular.
You also have to consider things like it is now harder for people to see the system as a holistic whole (the tricky bugs are often in the composition of components) and a lot of subtle effects that beings. Even just increasing the friction for people to move between teams or friction for security people to apply consistent standards across all groups.
Separate services aren't a silver bullet, but as an fyi to the younger software developers, we tried "just have all teams work on the same code base and deployable artifact" for a long while and it didn't work very well either.
I have lost track of how many times I did a git pull on a Python based solution only to find I broke all the things when I tried to upgrade one package.
Now in one place such an engineer uses pandas 2 in other place pandas 1 but it is just one single app. What does it say about the quality of engineering and mental focus of such a solo developer that cannot accomplish same thing with the same API - OR cannot refactor the already written code for Pandas 1 to Pandas2.
Sounds to me like more of an engineering discipline and engineering mindfulness problem.
Fix is simple with a simple rule - everyone has to use the latest major version, always.
Micro services do not make any sort of people's communication go away, they move it to different boundaries. From dependencies to the business layer/interfaces which is lot harder to navigate and negotiate.
Imagine needing a field in your downstream service. They refuse because they don't see it their domain and you cram it on your side and what not. Ask anyone working in micro services environment and they'll tell you it is a recurring issue every quarter if not more.
Just press this button to start the upgrade build and....boom! 10,000 services and their dependencies being built on a ton of hardware; we can practically gurantee your change in dependency will be checked... Whoops, turns out your one dependency change cascaded into about 1.5% breakage....no, I don't know who owns those packsges; why do you ask? That's not my job!
/s
I have only seen this from the business side (I'm not a developer), but I have seen teams start coding in another teams service just to be able to proceed.
It's not always good to create silos like this either.
As a developer, I have certainly seen the same. Pretty sure this very scenario is where I heard the term "away team" used in the industry: send your folks over to change things, and under our guidance they can check in the code.
You need truly gargantuan scale before things become logically separate code-bases.