Microservices: Why Are We Doing This?
michaeldehaan.substack.com
michaeldehaan.substack.com
As oxidation and reality set in, we realized the shiny thing was actually a horrific distraction from our underlying business needs. We lost 2 important customers because we were playing type checking games across JSON wire protocols instead of doing actual work. Why spend all that money for an expensive workstation if you are going to do all the basic bullshit in your own brain?
We are now back into a monolithic software stack. We also use a monorepo, which is an obvious pairing with this grain. Some days we joke as a team about the days where we'd have to go check for issues or API contract mismatches on 9+ repositories. Now, when someone says "Issue/PR #12842" or provides a commit hash, we know precisely what that means and where to go to deal with it.
Monolithic software is better in literally every way if you can figure out how to work together as a team on a shared codebase. Absolutely no software product should start as a distributed cloud special. You wait until it becomes essential to the business and even then, only consider it with intense disdain as a technology expert.
If you 'think' you need NoSQL or a MicroService architecture - chances are you don't.
(Ironically, I agree very much with your general sentiment, just not the example of illustration you chose)
You don't have to. You can just use JSONB.
Likewise, people who only know relational DBs shouldn't assume they need an SQL solution. It's remarkably rare that you encounter engineers with real thoughtful experience with both and they have the experience to know which tool to pick up for different problems.
That is kind of the problem, DynamoDB - and other NoSQL database - allow you to just start building your app, without any real thought about what data you will need and how it will be used - but you are often shooting yourself in the foot to not map this stuff properly out in advance.
Meanwhile a lot of the industry is trying to tell them that their problem is they haven't separated things enough and should be using lambdas for everything.
The only practical solution I have found for this: Carefully select a Padawan and guide them gently away from the forces of evil until they have gained enough situational awareness to spot these patterns and defend themselves. Not everyone can be saved, and at this point I fear that it is a majority.
If someone goes too far into cloud insanity, there is (in my experience) very little you can do to bring them back down to reality. At least, not on time frames the business owners seemed interested in when looking at new hires. I have had a much easier time taking in someone totally green and getting them happy/productive on monolithic software than I have with uService/AWS-certified 'experts' (et. al.).
"No, stop, don't."
It's like I'm talking to children, they think I'm just old and stuck in my ways and that I just don't understand.
Eventually my prediction comes true but none of them have ever said anything, they just go implement what I suggested and act like they solved the world's greatest mystery.
I guess it's easy to forget my comments from months earlier, it's almost as if I've seen their code before and know how it ends.
1) A direct method invocation resolved within the same L1 cache contents.
or
2) One quick network hop just 5 milliseconds away?
Assuming worst case processing semantics (global total order), you would be able to process ~10 million times more customer transactions per unit time with option 1 vs option 2. This is seven orders of magnitude.
In my experience, not a whole lot is truly worst case, but most complicated & important business systems (banking/finance/inventory/logistics/crm/etc.) are pretty close if you don't want to be chasing temporal rabbits around all day.
Nobody else saw the problem here. Establishing a http connection, serializing a JSON document, deseralizing it on the other side, then doing "14 == 15" anyway before going back the way it came. Lighting up a hundred thousand lines of code vs one.
This was done as it iterated over large files, millions of items, millions of requests.
They were genuinely confused why it was slow.
For example, ValidationService as described above could be replaced with OpenPolicyAgent and would scale much better.
That said, I could see someone pointing at OpenPolicyAgent and asking why they didn’t separate distributed updates from the policy spec as two distinct projects, this way you could re-use parts of OPA to distribute config files or feature flags. You can now too, but it requires the extra hop of compiling a function to answer.
Come to think of it, distributed state updating is roughly the same general problem that Kubernetes Operator pattern solves.
Another way of putting it, we have cloud primitives but not enough of them, maybe - or they aren’t well enough explained - to make the choices or restrictions of different architectures a bit more obvious? I especially look forward to a future when we get “design systems” for the programming we do with business logic in the cloud.
I’m over-emphasizing cloud here. I mean code you don’t directly have to maintain and write regardless of where it runs or who wrote it.
That said, depending on the type of validation performed, zero trust systems for example are often built using centralized validation endpoints, so it’s not entirely a bad practice.
These days when I do architecture interviews, I always make sure the problem can be solved distributed or monolithic and when they do distributed, I asked them why and to explain the trade-offs. If they do monolithic, I ask them why not distributed. Etc. For senior engineers, it's such a good indicator of whether or not they've truly worked in distributed systems and understand the human side of the problem.
The big O stuff is great, but the human and organizational costs are easily as important to assess as the computational and memory costs.
Micro-services were promoted for a variety of reasons. One of which was facilitating rolling out projects by consulting companies given the economic incentives for such businesses. Related was the VC driven development model that required rolling out features at greater speed than that of thoughtful development. Another was (remains?) the prevalent lack of competence in designing schemas (required for the basic n-tier system culminating in the database tier).
Microservices don't even have to run in separate binaries, let along separate machines.
If you're not running separate binaries, what you have is a monolith with well-defined module boundaries, not a set of microservices.
More commonly the binaries would differ but share a lot of code in a library of the monorepo.
What I don't want is for people to buy the microservices hype so thoroughly that they start thinking "microservices" just means "well-architected system".
Unless you're handling millions of lines of code and 1000s of developers, you won't need to abandon a monorepo or commit to expensive custom dev-platform investments.
When including our vendor's WSDL generated reference sources, we are well in excess of 1mm lines of code. I think even omitting that we are very close.
I cannot imagine us ever getting too big in those terms. Our full checkout is ~400 megs right now. If some day github says we are too big and need to break shit up, Ill move us to perforce or something even more ridiculous.
Our main product does a full re-build in ~45 seconds on a powerful developer machine. Incrementals for more likely work areas are closer to 8 seconds. Full monorepo checkbuild takes about 5 minutes. For reference, we do use .NET so JIT is an ally here.
That's cute. How about the test suites?
I still mostly prefer the libs + monolith approach though.
I prefer to use the term “Local Modular Application” in lieu of anything better?
https://shopify.engineering/deconstructing-monolith-designin...
I would say if you begin a journey to shim legacy, it should also be a journey to entirely deprecate it. You can gradually consume an old monolith with a new one over time if you are deliberate enough.
It's not a surprise that most younger large enterprises use microservices, with Google being a notable exception. Google however has spent 10s, possibly 100s of millions of dollars on building tooling to make that possible (possibly even more than a billion dollars!).
All that being said, every startup I advise I tell them don't do microservices at the start. Build your monolith with clean hard edges between modules and functions so that it will be easier later, but build a monolith until you get big enough that microservices is actually a win.
But honestly I'm not sure there is much of a line between the two. I've seen microservices that just return True/False and ones that return 100 lines of json, which are arguably more web-services than microservices.
I honestly think it's a distinction without meaning.
A CLI not a service: there is no operational complexity to "keep things running" in a CLI: you just chain some things together with pipes and that's that. The nice thing about that is that the text interface is generic and you can do things the original authors never thought of. With microservices this usually isn't the case and things are extremely specific. This is also why "do one thing and do it well" doesn't really carry over very well to GUIs.
A lot of microservices I've seen are just functions calls, but with the extra steps of the network stack, gRPC, etc. Some would argue that this is "doing microservices wrong" – and I'd agree – but the reality of the matter is that this is how most people are actually using microservices, and that this is what microservices mean to many people today.
Instead of "microservices" we need to think about "event-driven logic", or something like that. Currently the industry is absolutely obsessed with how you run things, rather than how you design things.
Google famously has a monorepo but that's different from a monolithic service architecture.
I.e. you can put your monolith in multiple repos, and you can put 100,000+ services in 1 repo.
I think the two are completely orthogonal.
At Google, when you check in code, it tests against things it could have broken. Not all tests in the system. For most services, that means just testing the service. For infrastructure code, then you have to test many services.
* unless you’re changing a very common dependency, of course, and Google has tooling for this.
Can you speak more about the criteria here?
You may be implying that microservices enforce Conway's law. If so then when the monolith divides, it "gives away" some of it's API to another name, such that the new node has it's own endpoints. This named set is adopted by a team, and evolves separately from that point on, according to cost/revenue. The team and its microservice form a semi-autonomous unit, in theory able to evolve faster in relative isolation from the original.
The problem from the capital perspective is that you get a bazillion bespoke developer experiences, all good and bad in their unique and special ways, which means that the personal dev experience will matter, a guide in the wilderness who's lived there for years. The more tools are required to run a typical DX, the more tightly coupled the service will be to the developers who built it. This generally favors the developer, which may also explain why the architecture is popular.
At most companies that do microservices well, they have a dedicated platform team that builds tools specifically for building microservices. This includes things like deployment, canaries, data storage, data pipelines, caching, libraries for service discovery and connections, etc.
This leaves the teams building the services focusing on business logic while having similar developer experiences. The code might use different conventions internally and even different languages, but they all interact with the larger ecosystem in the same way, so that devs at the company can move around to different services with ease, and onboarding is similar throughout.
But big enterprises inevitably lose "stack coherence" over time, through drift but also acquisitions. Finding the lowest common denomenator to operate and modify it all, while maintaining a high level of service (uptime, security, data integrity, privacy, value), turns out to be a tricky problem - just defining the product categories is a tricky problem!
Well I for one would love to see such a thing properly functioning. I've seen two attempts, but neither were successful.
Which ones? Amazon uses roughly 1-team-1-service, not 1-team-100-*micro*services.
Facebook famously built their main service as a monolith.
Edit: and don't get me wrong, I'm not saying services are bad - as long as they are the right size and with the right design rather than tiny.
I'd say you're letting your biases blind you. Netflix writes mature robust code, mostly in Java, and uses microservices.
I think you nailed it. Microservices are a solution for organizational problems that arise when the company grow in size, unfortunately it's not rare to see small startups with a handful of engineers and 5 to 10 times more services…
This is unfortunately very easy to override. Oh the rants I could write. If I could go back in time we would've put in a ton of extra linting steps to prevent people casually turning private things public* and tying dependencies across the stack. The worst is when someone lets loose a junior dev who finds a bunch of similar looking code in unrelated modules and decides it needs to be DRY. And of course nobody will say no because it contradicts dogma. Oh and the shit that ended up in the cookies... still suffering it a decade later.
*This is a lot better with [micro]services but now the code cowboys talk you into letting them connect directly to your DB.
(Which sometimes you also need even when using microservices, if you use a monorepo, for example)
I'd like to see software ecosystems that make it possible to develop an application that seems like a monolith to work with (single repository, manageable within a seamless code editing environment, with tests that run across application modules) and yet has the same deployment, monitoring and scale up/out benefits that microservices have.
Ensuring that the small-team benefits would continue to exist (comparative to 'traditional' microservices) in that kind of platform could be a challenge -- it's a question of sensibly laying out the application architecture to match the social/organizational structure, and for each of those to be cohesive and effective.
https://dropbox.tech/infrastructure/atlas--our-journey-from-...
With good defaults, you can have a dev tools / platform team create a blessed path that most teams will easily adopt so you get a mostly standardized internal architecture (useful for mobility). It's harder to allow for lessons learned from one service team to transition to the org as a whole, but if the dev tools / platform team has great Principal SWEs, it'll work. It does mean that you need great people on the platform team, though, since mediocre people will attempt to freeze development to fixed toolchains and will be unable to see the big picture.
I think Amazon does a good job with their Principals here.
You could do it with docker-compose I guess, but optimally your end result would be a single portable application.
all of your foo-service json endpoints can be lib-foo apis (the original meaning!)
Something like this would be the architecture I imagine.
But given that we already use libraries for everything, all of that stuff has already been figured out.
Do you use some sort of contract based testing between service libraries? Put all integration testing in the host application?
It's not obvious to me what the best approach would be.
With libraries it's easier than with microservices, because I can scan all the dependency files of all projects and immediately see which project relies on what library, which is much harder to do with microservices as a library author.
With those two things, and the fact that you can just keep using an older library version if you need to, whereas you can't easily keep using an older microservice version if it's been upgraded, I think libraries have lots of advantages in this.
The business revolved around filling out a form and PDF generations of said form. I felt like I got no work done in a year and so I left
I was taught that sharing code between services should only be done in circumstances in which there is already a strong connection between them - for example by being owned by the same team.
In the organization where I initial learned how to deal with microservices even direct calls via HTTP between services was a no-go and only used in rare circumstances and if then only temporary.
I am not sure if this is the way to do it since I have a sample size of one - but we had about 500 services to manage and in the end it worked okay and I never saw something you described. That’s why I wanted to know if you have considered just not sharing code.
It's a fantastic INDIRECT way of telling someone they did something stupid. And because stupid actions map so well onto their doers, it's also a fantastic way to insult people.
As an aside: this sort of indirection - asking a question to assert something - it's not peculiar to English, but it's suuper common in English-language expressions (which is, in fact, due to the habitual indirection of the English - at least according to a linguistics professor I once had.)
The kicker is: because it's an ironic construction, it’s likely going to require more effort to process: the presence of irony means that what is actually meant by the speaker is not encoded literally, but rather must be interpreted - derived from what WAS said - and irony (and other ways of flouting literal language) yields a disjunction (x || y || z). It puts the listener into a role where their recognition of intent and ability to joke around are tested - and the correct interpretation is up in the air. So while it's possible to say this expression to a friend when they screw up ("Have you considered NOT showing up to work drunk?"), it's also a great way to make an enemy ("Have you considered NOT being a fuckup that nobody likes?"). Same construction. (In fact, this later example is especially sinister, because any answer you give just makes YOU assert what you're being slandered with. "No" - I haven't considered not being a fuckup? / "Yes" - I did in fact consider not being a fuckup? You get the idea. It can be wielded in a very mean way.)
And your comment wasn't intended this way at all, and in fact it's obvious when you read it that straight talk and no irony is meant, but it shares enough of the signification with the ironic phrase that it sets off alarm bells. Especially in an online reply to someone, where trolling is sport! The context is working against you here.
And the context is the first tool we have when interpreting communication, because it's already available to our cognition before we receive any given message. And brains are energy misers, using heuristics to filter shit out that doesn't look like a good reward for the necessary energy expenditure.
So I bet that the downvoters saw 1) a short comment, and short is by the way much more likely to be low effort and unconstructive; and 2) all the native speakers' downvoting brains instantly recognized the same construction as the ironic idiom talked about above, and then they instantly said fuck it, this person is being a dick, without even really reading it. Since irony (unless it’s expected) requires more processing effort (both in detecting WHETHER irony is present AND what the ironic speaker means, just to get to the point where you can start assessing the semantic content), these two things were enough to justify jumping the interpretive gun. I wouldn't be surprised if your question was flagged by MOST native speakers’ brains’ asshole detection systems. And so, the actual substance of your question, which, processed in its entirety, have indicated that you were asking a bona fide, relevant, even microservices-essential question - got booted out. And then the overzealous downvoters succumbed to the urge to smash the downvote button without a second thought and never looked back.
I found myself in the same place, actually - thinking "they're being a dick," then moving on. Probably took less than a second. Often when people in online conversations say why the downvotes? they're trolls acting in bad faith, which pisses me off because I always go reread what they wrote. So I did. You weren't calling anyone stupid! Still, I had to stare at it a bit before I realized it was an unintentional collision with the syntax of sarcasm. Maybe like if I wrote "Was hast du denn?" and really truly meant "what do you have?" instead of "what's wrong with you?" If you don't recognize it for what it is, it might look like an honest question which expects an honest answer.
Oh. Another thing to consider: after a while spent in any given semiotic context, our cognition adjusts and those cognitive heuristics that reject hard things become even lazier. Our pattern of behavior when reading on screens starts with reading, but then it turns to skimming and scanning if given long enough. I'd bet money that if your brief, but bona fide contribution had been one of the first comments, your downvote percentage would be a LOT lower. Halfway down a long page, the skippers are skipping and the downvoters downvoting. That doesn't mean that they intended to or are even aware of the switch in reading modes. At the top of the page, information is still shiny and new, and worth interpreting, speculating on the speaker and their intentions, etc. After a while, nobody gives any comment a second chance.
Anyway, I don't get to revisit all the shit I learned at university enough, so it felt good to write all that. Are you still here? Maybe everyone's already moved the fuck on. Brain got bored? Would serve me right, ha! So, here I go, back to my job, where instead of working in cognitive linguistics, text semiotics, pragmatics, philosophy of language, etc., I'm writing boring-ass JavaScript all day, for an application ... being built on microservices.
My company uses microservices; deploys restart one service and PRs are one repo at a time. There's a shared library, but it's versioned and there's nothing compelling you to keep on the bleeding edge.
If a coordinated rollout is requires for anything but a change of service API (and then only the service and it's direct clients should be impacted, and even then a decent deprecation policy should eliminate the need for close coordination), you aren't doing microservices, because loose coupling is part of the definition of the pattern.
Once it gets popular, people start “implementing the pattern” based on rumor, bad descriptions from unqualified (and sometimes ill-motivated) intermediaries, and some boss and/or tech lead’s fever dreams, and blaming it on the pattern.
(If the pattern is useful enough, it will be later invented under a new name to escape from the accumulated cruft, which will briefly succeed only to succumb to the same process until another cycle passes.)
What if I suggested an architecture where every 5th call had the overhead of serialization and network time. Also the reliability and concern issues that go with multi machine calls. Data syncing issues and network partition concerns.
Pretty sure most people couldn't reason about this system. They'd suggest instead of having network calls on every 5th call, lets really look at our use case and only insert them where we may have load issues and have to scale. Data state becomes a big concern, make sure we know where our data is at all times and is only passed when needed to be correct and not over pass it to be efficient.
That of course leads you down the path of creating an "oddly specific" API that caters exactly to the needs of the clients, down to the point that small changes in the client's needs require changes in the upstream service(s) - so your "separation of concerns" has become a joke.
I've worked on "microservices" systems before that do this. They were mostly shit.
The system I'm working on at the moment has each service subscribe to events and maintain its own database. The only comms is via events. It works pretty well and everything really is nicely separated, but it feels a bit haphazard.
This is an engineering process failure, not a failure of microservices or shared dependencies. You should be versioning your shared library, that way you only need to make a deployment to the service that requires the update, leaving the others pegged at the previous version until a business or engineering need motivates the upgrade.
The whole problem, I think, comes from the "split the code" cargo-cult. We need to think about why we're splitting the code, and use that why to figure out when to split code.
IMHO, code separation arises naturally from modular programming - once your code is mature enough, it becomes just a piece of glue around a set of libraries that you can just rip out and put in their own repos, provided that they're useful enough.
The 7 repos are gone now and a k8s cluster has replaced the swarm service. Deployments are a bit easier to manage now. It's still such slow development process to add a simple API endpoint, especially if that API has to call other DAOs. It's crazy because I feel like it's completely normalized here that dev work takes 300% longer than it should for simple features.
The market has been distorted by endless amounts of VC funding for very dubious ideas that would never be profitable to begin with, so now you have 2 solutions: you can spend a few hundred grand building a boring solution with a slim amount of engineers, realize it doesn't work and quit, or you can keep over-engineering indefinitely, keep raising more and more millions, enjoying the "startup founder" lifestyle while providing careers to unreasonable amounts of engineers with no end in sight because you're too busy over-engineering rather than "solving" the business problem, so the realization that the business isn’t viable never actually comes.
Which one do you pick? The market currently rewards the second option for all parties involved, so it's become the default choice. Worse, it’s been going on long enough that a lot of people in the industry consider this normal and don’t know any other way.
I've commented/ranted about this before, see https://news.ycombinator.com/item?id=30008257, https://news.ycombinator.com/item?id=24926060 and https://news.ycombinator.com/item?id=30272588.
I would guess that both PHP and Rails/React (and Java, C#, Python) folk will have all the work they can stomach, if they can stomach it. Consider how in-demand COBOL programmers are these days!
I keep hearing this but I have only ever heard bad things from the people who work there. Considering how in demand all developers are right now I wouldn't even think of getting in to some obsolete tech. You'd have to pay me enough to retire quickly.
If I claim you felt angry when typing this, can I falsify it?
If you confine yourself to falsifiability, you will never be able to understand people.
You actually can't.
Like, you can make _arguments_ based on non-falsifiable points. But if the points aren't falsifiable, then there's nowhere for the discussion to go. How can I respond to them? What can I do to counter your arguments? Nothing. It's dull.
However, in a lot of cases they could be considered premature optimisation.
A bulldozer is also a useful tool, but in the real world nobody uses one for small jobs better suited to a shovel because of how expensive it is.
In the tech industry however, VCs will be happy to bankroll the bulldozer for you so that it becomes cheaper than using the shovel. All else being equal, I too will pick the bulldozer at least as long as I’m not the one doing the maintenance it will inevitably require.
Worse, give it enough time and the skill to use shovels will disappear, and now everyone will be using bulldozers for even the smallest jobs, with all the negative externalities attached to them. The only winners are the bulldozer manufacturers.
React is about a decade old.
Microservices is between two and seven decades old, depending on your definition.
None of this stuff is new.
If it can be done better for cheaper, VCs will tend to fund that team.
AI and ML is one thing - it can mean a product that couldn't exist without it. There's nothing about MERN vs PHP that indicates that though.
I'd note that today putting that workload on AWS would probably be a pretty pleasant experience - but yeah, sometimes investors absolutely -do- care in ways that you'd really rather they didn't and you just have to suck it up.
But for the rest of the "crap", AI- & blockchain-powered crap is more shiny than other, more boring crap, at which point the tech stack might be more important.
You would get a similar read from VCs if you said that you use Haskell, Erlang, or up until quite recently - Rust.
There is something pretty comforting to an early stage investor of "We use Java microservices on mainstream cloud provider". Says that the team isn't ancient, isn't avant guard, and that they will be able to hire people. Worst case, an acquirer won't mind buying the leftovers or acquisition hiring the team.
If their MRR/ARR/MAU/DAU or whatever are strong then absolute-f'n-lutely.
That might be true if there was a shortage of VC money so they had to choose their investments very carefully. The reality is there's a huge amount of money sloshing around with few good investment opportunities.
Cheaper solutions would be a good pick for something that has a very good chance of success and thus doesn't need a "next sucker" nor media attention, but those are very rare as typically it would be bootstrapped and not involve VC to begin with.
If it seems hard and has buzzwords then VCs will fund it; they use difficulty and buzzwordiness as proxies for being on the bleeding edge of technology because they think that's how companies succeed.
The only reason blockchain and ML for example are trending is because VCs will pump money into companies that use these buzzwords.
One of my previous employers really wanted to use AI for a mundane problem that the engineering side provided simple solutions for, but the "founder" kept trying to push for AI from any angle he could imagine. He didn't tell us this directly, but to me it was obvious that for him AI was a gimmick to get VCs to pump money into his company.
This is exactly the kind of thing you should tell the engineers. Get them on board, straight up.
It's a mimetic society.
VCs and CEOs, a lot of times, want more developers working at companies, because they believe that's how they're get velocity to experiment, ability to grow and ability to change course quick enough. Up to some point that's true! The issue arises when having lots of developers requires having too many isolated teams. That's when Conway's Law kicks in: a company with lots of isolated teams will probably have complex architectures to make the teams work, most probably using (micro-)services. Hence, complexity.
Yep... As someone who's been in this situation a few times: after a certain number, very badly. Things get done, but in slower and messier pace than at a smaller company.
The diminishing returns law kicks in together with sunken cost fallacy. With 10 people you're slow, with 50 you're fast, but then there and you're forever "chasing that high" of velocity increasing with headcount.
There's no way to escape what Brooks wrote in Mythical Man Month.
Unless there is a very clear-cut division of labour, like 20 teams working in 20 different libraries that don't talk to each other, there is no free lunch when it comes to team communication.
So you notice that there are 2.5 implementations of a bit of logic because there are bugs, maybe security bugs, that cross 2 of them or are different on each.
I have found that once the code exceeds the size of one brain, it accelerates because if I don't know code exists, I will write it again. Like driving down a mountain side with the brakes on the whole way, eventually the sanity leaves the system and you go careening down the mountain.
Tony Hoare used his Turing Award speech to state the following:
There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult.
The real reason that microservices are so prevalent is that they became fashionable, for exactly the same reason a particular item or brand of clothing becomes fashionable - influential people were seen using them and so regular people aspired to start using them too. At a certain point in the popularity curve, _not_ using microservices becomes a controversial viewpoint.
They also suffer from being what I call "conceptual crack". There is a certain kind of idea that really tickles some kinds of engineer's brains. Microservices seem like a such a clean solution, each service having its own single responsibility, so easy to draw on a whiteboard, so neat and tidy. Other ideas I place in this category are blockchain, redux, and V=f(s). Clean and tidy ideas that are compelling conceptually but result in nightmarish levels of hidden complexity when they're put into practice.
I'd say it's one of my favorite properties of component frameworks as opposed to raw element modification, templating solutions or classical MVC.
EDIT: Oh I get it, you aren't talking at all about the consequences for the implementers. Only the massive effort required to turn the alluring V=f(S) into the React we have today. Got it.
CTOs and engineering managers care about their own visibility at said cloud provider's conference so it gives them bonus points for their next gig.
Individual contributors want to take the place of the aforementioned engineering manager/CTO so they'll double-down and learn the current stack and "best practices". They may not even be aware that there's any other way or how unefficient it is, as they've started their career during this madness.
If any of those people have stock options, keeping low and going with the flow is a sensible strategy if they don't want to lose them by getting pushed out for speaking against the groupthink.
There's rarely explicit malice involved at any specific stage - it's a market-level problem. The music will stop at some point though and we'll see a readjustment.
Imagine you were the founder of a business you've now realised... isn't going to be the next Google.
Your product hasn't failed! By some metrics it's very successful! You've got investors who value your company at $10 billion, and they're ready to loan you $1 billion.
But also, you've never made a profit, you've tried the most likely routes to profitability without success, similar businesses have suffered sudden implosions, and you'd have gone bankrupt years ago if it weren't for the fact there's chumps who'll work for you for free. You've got no plan for how to pay back that $1 billion.
But so long as you keep your mouth shut about that last paragraph, and accept the $1 billion? You get a high-status job, a thousand-person empire, and a fat compensation package. 1000 of your colleagues stay employed too.
Why would anyone climb off that gravy train?
Decency?
Can you believe it? The audacity! A business charging business people, for a course of knowledge both sides know they will never use...
This scenario you've described is such a stretch that in all likelihood it has never happened.
If I actually believe I can't deliver the stress alone of feeling like a fraud and waiting for the hammer to fall seems worse than the interim reward for me, but I can certainly see people built different who would tolerate that. I think more likely is that people let themselves be convinced it will work out during the process of convincing investors.
This suggests either:
- the investors [think they] know something you don’t know
- the investors are profoundly careless with their investment
- the investors don’t exist, this is completely implausible
- you’re actively misleading the investors about that paragraph, not just keeping your mouth shut
Sounds plausible to me.
For example, consider WeWork - the CEO was literally buying office buildings with his shares, and having the company rent them off him. These investment funds are astonishingly careless.
Most founder either want to run a profitable business or make their way to a big exit. Company valued at 10b? Great, sell or IPO. If the founder really realized that there was no profit ever to be made and genuinely didnt believe in the future of the company then the best move for their own self-interest would be to sell and gtfo.
Not only if your premise impractical, but it's inherently illogical for your antagonist to act as you imagine.
Most people love to be looked up to, loved, payed nicely. They won’t quit.
The new owners have a different threshold, which you are about to learn. The exchange of money is fresh in their mind, they don't have a history with the employees, and they have a story in their head about how they can turn this money into a pile twice as high. They may well get the company to profitability, but if things were a little ridiculous before they may be a proper circus now.
There's some magic algorithm they use for figuring out options and vesting periods to retain staff, and there's some inflection point where if you stay this long then you get the most extra money per month. This is the trap they have set for you. It's practically an optical illusion, which you can only see once you've stepped past it. You will underestimate the relative value of month 3 versus month 13, and you will forget that 6 months may be worth $6N dollars but 5 months is worth $0. And so with one month to go you will be willing to put up with 4x as much bullshit, forgetting that most of that extra money was already buying the last 5 months. In effect, double spending your bonus. The moment you leave you will wonder why you didn't do it ages ago, even as you are spending the cash.
If it seems like you should stay for 2 years, then you will probably be happier with 1. If 3 years, then 2 years, maybe 18 months if there are increments smaller than a year. Of course, nobody can really tell you this, because your brain will keep asking you 'what if I had stayed' until it's happened to you at least once. But I think maybe you can tell people not to repeat that mistake. Trust your instincts from last time. This one won't be different.
Are they really? I think they're workaholics chasing an escape. True success is the worst outcome they could imagine, because then they'd have to tend to the rest of their life. Why be a serial founder, if its so horrible and stressful? Why keep working yourself to death after your first $10M, $100M, or $1B? Why is it never enough?
People have self-destructive instincts that are disguised in socially acceptable (or worse, praised!) forms. We shouldn't ignore the actions of founders when trying to make sense of their words.
If that's true, I'd like someone to explain Craigslist.
When you start, you choose between investors+growth=exit or profitable+steady=tidy
Greatly simplified of course but Craigslist chose the second path.
Every company was once a startup.
A small business is one designed to be tidy.
All business start but not every one that starts is a Startup.
Include tangent here about Lean Startup vs Startup.
Also, I'm using Startup which is different than startup - the Proper Casing I think implies the Venture-to-Exit part.
This is so spot on. The amount of unnecessary complexity is becoming ridiculous.
It's kind of embarrassing but whatever, we are going to bill a few million to get it done. And that's just my small business logic team. It doesn't count the two or three user interfaces.
In the companies I've worked for architectural discussions are taken extremely seriously, we research and discuss how different approaches to solving problems has turned out for other folks and we might even run a pilot or two where we do a small scale demo to feel out the pluses and minuses.
Complexity without sufficient justification can't survive in most established businesses - a research budget certainly can, and you can use that research budget to explore options - but if you want to rewrite the codebase into a different paradigm you need to clearly demonstrate what we'll be getting out of it. That might be performance, maintainability, hireability (i.e. most local new grads are trained up in this paradigm), feature support or something else, but you need to show the benefit - I think it's a good idea to be skeptical of historical decisions, a lot of decisions early on in a company's history are made arbitrarily and sort of assumed because the momentum is on their side - you definitely should fight against that momentum and reject bad conventions but... switching to a hip new tech just for complexities sake - I have no idea how I'd ever sell that to higher ups as a project initiative and I've sold executed two framework switches (no framework to a now defunct framework to a now healthy framework) and several major data model changes (one that took about ten man years of labour).
When you've got a green field you're making decisions left and right, and some of those will be made arbitrarily - but the big commitments should be well contemplated in advance.
Well, that all seems reasonable until that upstart competitor comes along and upends the market with simpler stuff that doesn’t require so much bondage and discipline and tithe to process religion suppliers, and the customers start to question the value you’re supplying and wham, all that orderly process and framing is like out the window, man, like man the lifeboats. It’s a pain to stay in the game when it seems like change for changes’ sake and there’s no time to sit down and reason through a business case before the next new shiny hits. Stupid but true.
I still remember Digital trying to sell me maintenance contracts for 20 year old PDP-11s still in service out in the field. We rebuilt our entire infrastructure from scratch on new software and hardware in less than a year for 2/3rds the cost of the annual maintenance contract...the next year those pretty wood paneled Digital offices with executive dining rooms had "for lease" signs out front. Digital had failed the paradigm shift.
For whatever the things of the year, a VC team will fund say 20 startups with the hope that 1-2 do well.. and are ok for the rest to flop. It's expected that 18-19 we're bad ideas, but they all still get giant sales/marketing/engineering staffs that 99% co's won't in order for the VC to have it get a go and then find out which is the winner. That is why you especially shouldn't trust speakers, marketers, and sales staff from a VC co: the financials show it's 95% chance to not be as advertised and then disappear on you in some financial shell game.
Why else would people spend hours trying to configure and customize linux distros?
To them it's a puzzle game and they very much enjoy the challenge of solving these puzzles. Having a dynamic system with many moving parts that all need to be configured just the right way so they finally fit together and come to produce the desired outcome.
This is an "epiphany" I got from playing "The Witness". I spent more than 50 hours playing the game, and I'm not even 10% finished. The puzzles in the game are original and require high level complex thinking to solve. But at some point it just got frustrating to me. I wanted to play games to sort of relax or perform low key mental activities. But this game wants you to spend a lot of mental energy to solve puzzles that seem arbitrary and pointless. The feeling I had when I was playing this game was very similar to the feelings I had when I was trying to wrangle with confusing Docker configurations.
That's when it hit me: people who love to play with Docker configurations treat it like puzzle games and they enjoy every bit of the mental effort it takes to get things just right for the system to work. It doesn't bother them that the system is fragile or over complicated, or that the mess they're building is hard to maintain.
Of course, I kind of empathize with that because that's also what got me into computers and programming in the first place. But to me, the complexity I want to deal with is in the code. Once I write code that solves a problem, I don't want to then struggle to get the code running. I want to just compile and run with one command. I want all the complexity to be contained in the code and to keep the environment simple.
But if your jobs is DevOps, you don't get to solve hard problems in the product's code base. So instead you solve hard problems in the environment that the code executes in. So you thrive in the complexity of microservices and dockers and all that buzz.
In other words, people in tech love solving hard problems, and if you don't give them hard problems, they will invent them.
I'm going to flip the narrative a little bit: whether its "tech is the end-game" or "engineers are bored and just want to have fun", I'm not sure I see the problem. I've grown out of the "maximally efficient business is the endgame" propaganda. My endgame at work is to have fun and produce enough value so I can come back tomorrow and continue having fun. Business as the endgame is a great motive for the CEOs who make millions a year. I'm not that.
Put another way, many people say that coding is as much art as science or engineering. Artists may do commissions, but ultimately their work comes from within; its not by-and-large prescriptive. The endgame is the art; the endgame is the tech; and, hopefully, that endgame is marketable. Sometimes it is, sometimes it isn't; that's not especially relevant to the process.
Or, you can take the stance that the endgame is the business, and spend your life unfulfilled, making your rich boss richer. I'd rather seek fulfillment from the tech; not the money.
I often equate a business to a ship. It's dynamic, navigates obstacles, and is captained by leadership.
Perhaps the two examples are extremes on a spectrum. And inside is the option to learn more about the ship, so that the right problems can be resolved and solutions found, to help it reach its destination. And perhaps fulfillment can be found there.
In fact, in doing so, sometimes, new courses become realized. Personally, I find getting one charted very fulfilling.
I have also talked about something similar before [1].
A company I worked for, after receiving way more money than they needed from Softbank, suddenly had to hire as many people as they could as a condition imposed by the investor. This got to the point we had large teams working solely on portions of a page of the application, even though the content (not only the look) of the website didn't really change for a few years. The constant rewrites and programming language changes (three full rewrites of the whole app) meant that there was an endless stream of work even for teams that only controlled half of a settings page.
Like a friend put it, "your 8-person team is a 3-day job in a normal company".
Even with the pandemic bringing customer numbers to almost zero, the website was still too frail to stay up. And the solution to that wasn't introspection about how the architecture was shit due to Conway's Law. It actually ended up in another rewrite using even more complexity and more division.
But you don’t get upvotes for saying this on HN.
Small teams were dealing with specific functionality, and they were largely autonomous as long as we agree upon the API, which was all done via Erlang's very elegant message passing system. Scalability was automatic, part of the runtime. We had system-wide visibility and anyone could test anything even on their own computers. We didn't have to practice defensive programming thanks to OTP, and any systemic failure was easier to detect and fix. Updates could be applied in hot, while the system was running, one of the nicest features of the BEAM, that microservices try to address.
All the complexity associated with microservices, or even Kubernetes and service meshes, are ultimately a way to achieve some sort of "polyglot BEAM". But I question if it's really worth it for all use cases. A lot of the "old" technology has kept evolving nicely, and I'd be perfectly fine using it to achieve the required business outcomes.
Are there complications? Sure. Are they manageable? Relatively easily with correct tooling. Do microservices (with container management) allow you better use of your expensive cloud resources? That was our experience, and a primary motivator.
I also feel they increase developer autonomy, which is very valuable IMO.
For me, it boils down to "this is absolutely doable but I'd still rather have as few services as possible while still maintaining useful levels of separation" - at least for the primary business services. Having a bunch of microservice like things serving pure infrastructure roles can be much cooler depending on your situation.
$100/service. Is that top level service? To cover a 100 endpoints at one service each is going to be 10k/mo?
I believe $100/service is top level service, not endpoint. But really not sure.
It's easy to say this is a lot of work to reinvent something you get for free with a (single language) monolith, but at least it's recognized as a problem worth solving.
I've worked on multiple systems with around 50 developers contributing fulltime to them, very practically.
Microservice fanaticism seems to be coupled with this psychosclerotic view that world can exist in state of microservices or as monolith.
From what I've seen in last 20+ years, if I had to pick one sentence to describe fit-all enterprise setup (and it's as stupid as saying "X is the best" without context) - it'd be monorepo with a dozen or two services, shared libraries, typed so refactoring and changes are reliable and fast, single versioned, deployed at once, using single database in most cases - one setup like this per-up-to-12 devs team. Multiple teams like this with coordinated backward compatibility on interfaces where they interact.
What your business does, how many people you have 3 or 10k, what kind of roles and seniority you have, how long you're in the project - 3 months in or 10 years in, how crystalized architecture is, at what scale you operate, how does performance landscape looks like, what kind of pre-deployment quality assurance policies are dictated by the business, are offline upgrades allowed or we're operating in 24h, which direction system is evolving, where are gaps (scalability, quality...) etc are all necessary to determine correct answer.
Building website for local tennis club will require different approaches than developing high frequency trading exchange and both will be different from approaches for system to show 1bn people one advert or the other.
Seeing world as hotdog and not-hotdog (microservices vs monoliths) makes infantile conversations. There is nothing inherently wrong with microservices, monoliths or any of approaches to manage complexity ie:
- refactoring code to a shared functions
- encapsulating into classes or a typed object
- encapsulating into a modules
- simply arranging code into better directory structures, flattening, naming things better, changing cross-sections ie. by behavior instead of physical-ish classes and objects
- extracting code to packages/libraries inside monorepo or its own repository ie. open sourcing non-business specific, generic projects or rely on 3rd party package/library
- extracting into dedicated threads, processes, actors/supervisors etc.
- extracting to service in monorepo or dedicated repository or creating internal team to black box it and communicate via api specs or use 3rd party service
...bonus points for:
- removing code, deleting services, removing nonsensical layers of complexity, simplifying, unifying etc.
etc
A lot of software engineering is about managing modularization, I've lived through structured programming, OOx, various distributed object schemes and now this. Basically all these mechanisms attempt to solve the problem of modularization and reuse. So, the fact that new solutions to the old problems appear just means that it's a hard problem worth working on. I'd say use every single technique when it's appropriate.
If we think about the fastest way to execute some code, that is a function call, not a web request.
If we think about the best way to make sure we detect problems at compile time, that is by using a library at a compiled language.
If we think about the best way to understand why something failed, that is a full stack trace.
If we think about resilient systems, the most resilient systems are the ones with the least number of moving parts. The same is true for the fastest systems.
If we think about deployment and management and upgrades, the simplest systems to deploy and maintain also have the least number of moving parts. """
You don't really need any data to back this up, it's all self-evident.
If we think about the easiest way to swap out pieces of functionality, that is by using well-defined interfaces, low coupling, and separation of concerns.
If we think about the easiest way to scale parts of the whole independently, that is by similar mechanisms.
If we think about the easiest way to build in fault-tolerance, that is by distributing work across failure points.
And so on...
Fault tolerance again is largely orthagonal to ms. It's a matter or system architecture, see the point above. e.g. A common situation where service one depends on availability from service two, having them as seperate ms doesn't help, what helps is architectural design to keep service two useful somehow regardless, and this is orthagonal to if the services are in process or done as ms.
Scaling is a whole other discussion, and microservices can have good impact here, but so can other techniques.
Microservices don't enforce good practices and monoliths don't prevent them either.
And others in monoliths.
Each has its pros and cons.
I've worked with with monoliths. The author must not have experienced them. I've worked places that had builds that took hours to run. We had git merges that took days. We had commit histories that were unreadable.
The developer experience working with it was one of CONSTANT frustration. The system was too big to make large changes safely. Incremental changes were too incremental and costly.
Note nowhere in here am I saying that microservice architecture should always be preferred. But the idea that its all just some sort of trend with no real underlying advantage is sort of silly.
Every company I've ever been at with a monolith tends to have "untouchables" of architecture and the original design schematics who understand the system orders of magnitude better than anyone else. That doesn't scale, and really messes with an engineering organization.
There's conways law where software will eventually reflect the organization structure of the company, but there's also a sort of reverse conways law - when you have teams dedicated to specific services you also get to be able to target investments in those teams when their services are not executing well enough.
The additional complexity - in terms of code and operations - is significant. You need to be very confident that it's going to pay for itself.
The only correct answer is to not waste time with the decade+ worth of pointless internet debates on the topic.
In comparison, building a good set of microservices is a minefield of infinite possibilities, with each decision about where a particular responsibility or piece of data should live being quite significant and often quite painful to change your mind about.
No, the fastest way to execute some code is a goto. Be careful with arguments from performance, that's how you get garbage like a former colleague's monstrous 10k SLOC C(++) function (compiled as C++, but it was really C with C++'s IO routines). Complete with a while(1) loop that wrapped almost the entire function body. When you need speed, design for speed, but you almost always need clarity first. Optimizations can follow.
> If we think about resilient systems, the most resilient systems are the ones with the least number of moving parts. The same is true for the fastest systems.
I suggest care with this argument as well. This would, naively interpreted, suggest that the most resilient system has 1 moving part (0 if we allow for not creating a system altogether). First, this is one of those things that doesn't have a clean monotonically increasing/decreasing curve to it. Adding a moving part doesn't automatically make it less resilient, and removing one doesn't automatically make it more resilient. There is a balance to be struck somewhere between 1 (probably a useless system, like leftpad) and millions. Second, there's a factor not discussed: It's the interaction points, not the number of moving parts themselves, that provides a stronger impact on resilience.
If you have 500 "moving parts" that are linearly connected (A->B->C->D->...), sure it's complicated but it's "straightforward". If something breaks you can trace through it and see which step received the wrong thing, and work backwards to see which prior step was the cause. If you have 500 moving parts that are all connected to each other then you have 500(500-1)/2 interactions that could be causing problems. That's the way to destroy resilience, not the number of moving parts but the complex interaction between them.
SOA works well if your teams are larger and have larger domain (or end to end, however you want to call it) knowledge.
Monoliths work well when the domain of the application is singular, the team is large, or if you're in prototyping. The big downside for monoliths is that their scaling model must be considered in advance or engineers can tactically corner themselves with architecture. That incurs big, expensive rewrites as well as time.
While Conway's Law may be reflective of the enterprises use of (or overuse of) microservices I think it really has more to do with a different enterprise habit: understaffing and budget constraint. Microservices and client-side applications, from my perspective, very rarely have long-term maintainers. Instead, things get done in cycles and then for most of the year a given service does not receive anything besides some maintenance updates or low-hanging fixes. That makes it look like a microservice is expendable and easier to replace to the people who manage resources, staffing, and budgets. Thus, things now look "modular" to the people who fund the ship that everyone else drives.
For instance, comparing Adama to the services needed to build similar experiences offered by AWS has interesting results. Adama costs 97% less than AWS ( https://www.adama-platform.com/2022/03/18/progress-on-new-da... ), and a key thing is that the microservice approach is amenable to metering every interaction which scales linear to demand whilst a monolithic approaches condenses compute and memory.
That means that the AWS service based option costs 33 times more. Not ten time more, but thirty plus times more.
"But one day, when we get massive growth, it will all be worth it", he says.
Alas that day may not come, since he is busy configuring load balancers and message queues instead of developing features.
He made the point that micro-services are a deployment method not an architecture. A good clean architecture shouldn't care how it's deployed. If you need to move from plugins to micro-services to be massively scalable your architecture shouldn't care. If you need to move from micro-services to plugins to make your app simple to host and debug, your architecture should also not care.
This strategy has been implemented in frameworks like abp.io very successfully. You can start your application as a single collection of split assemblies deployed as a single application and move to deploying as micro-services when it's necessary.
[0] https://www.linkedin.com/pulse/20140604121818-6461201-seven-... (I think that's it, original link doesn't work anymore, this looks like a copy of it)
[1] http://highscalability.com/blog/2014/4/8/microservices-not-a...
Regarding division of work, those advocates seem to have forgotten that libraries exists.
Or that you can deploy a monolith and still scale endpoints independently.
Microservices might be a good fit for tiny fraction of real world scenarios.
That said, the challenges of building such systems are real, and the developer experience is universally quite awful compared to our monolithic, single-server past.
It’s for that reason that I’ve been building [1] for the past four years. Would love your feedback if the OP resonates with you.
Over-complicating things to pad your resume might get you into FAANG but it won't make you a good engineer or a good entrepreneur. The sign of a truly senior engineer is one who knows how to keep things as simple as possible to solve real problems while maximizing power to weight ratio of their code. The resume-driven-development anti-pattern is pretending that the problems facing 1000 or 10000 person orgs are your problems. Those large companies got to where they are by solving the problems in front of them, and you won't get to that scale if you don't do the same.
Packing multiple concerns into a single instance/VM seems particularly cavalier. What if one of the services crashes the OS based on novel user input that is being sent over and over? I think it's naive to say that perfect testing and strongly/statically typed languages make this problem go away.
A large team iterating on a set of monoliths runs into other problems as well. Ready to deploy, but wait, you need to rebase for someone else's changes, oh wait, now your tests don't pass, etc...
A set of monoliths certainly simplifies things for some projects, but many of "us" have battle scars from managing large cloud-based projects in that way, and find the deployment complexity of having a proliferation of independent entities that can have their own life cycle to be crucial for velocity and availability. At least that is why some "us" do it this way.
I'm not sure this is 100% correct - or, at least, has never been the case in the 20 years or so I've been working with "microservice" architectures. There's always an architecture team dictating the form of the services themselves - usually much more so than a module in a monolithic application. Part of this standardization is usually a set of languages - you can use Python (with Django) or Java (with Spring Boot), but anything else has to be approved by the committee.
That said, I agree with is ultimate conclusion that microservices haven't lived up to their promise. The usual justification for microservices was and continues to be "we can update one component without disturbing the others". I've never seen that actually happen. Every time one component changes, everything that depends on it has to be retested just to be on the safe side (and most of the time, something is found).
Reminds me of the fun time a major financial institution had one of their internal services provide fully 6 historical versions of it's api. Interestingly, they had not a single (automated) test for any of these.
There are many companies still successfully using Rails, don't lose hope!
Then I remember that half the time we only notice a problem with service A because service B started getting slower, and that it can take a lot of servers to make sure that you aren't "wasting" servers.
Sometimes a little slack can be cheaper. Sometimes side effects shorten the feedback loop for a group that has a bad habit of letting problems fester until someone on another team notices.
But the real reason we use microservices is they're just more popular. Nobody wants to recommend something obscure or old for fear they'll be laughed out of the office. Nobody can hire tech people to work on 'uncool' technology. People like to follow trends. That's why we are doing this.
in some sense yes, in some sense no. a monolith with a denial of service vector in part of its functionality can suddenly take down your entire fleet because everything was exactly the same.
in much the same that domestic bananas are more or less one virus/bacteria away from being wiped out (again), a monolith very susceptible to any kind of correlated failure because the blast radius is most likely the entire monolith.
Just like a microservice can bring down the core functions of your service because a single component locks up under DoS
It can even be done dynamically on the fly. If the instances begin to struggle, divide them in half and see which half has the problem. Then divide that one in half, and so on, until you have isolated the problem. Then stop those requests altogether and have capacity over for all the people desperately hitting F5.
So now you have both DOS resilience and independent scaling, all without requiring any changes to the architecture or knowing in advance exactly what you should have factored out into its own service.
From my experience all of these resources tend to direct you to think in microservices / uber-distributed architectures. I can easily imagine this causing folks to consider this as "the way we do systems now" and taking it to extremes.
The low barrier as well for 1 engineer to decide they want to just get busy on a weekend on their own and build something is another way I've seen this proliferate.
People will drink rat poison, as long as simple, natural, local, small scale, etc.
People don't want what is technically best. They want what makes them feel strong and smart.
They LOOOOVEEE to add challenges and problems because they want to build a world where people are "tested". If something is such a good design that it just works without a lot of talent they get bored.
Note: lots of systems, lots of technologies, from SAP to Rust. Not all microservice by a long shot but those have worked pretty well.
This is non-sequitur: release frequency is not dictated by release process - although it is affected by it - but by business requirements.
"We never really had “monoliths” before in development that I experienced".
Good for the OP, I guess. I currently involved in development of 3 monolith services. All are 6-7 years behind on the framework they were based on and making any sweeping refactorings is cost prohibitive and tantamount to a full rewrite. Were they architected as microservices (which entails not just code, but infra), major refactorings would've been on the table.
Code review policy feelings doesn't make a lot of sense to me either. Company's policy is all code goes through review. How one feels about it is not relevant, it's a part of one's job. Microservices or not, code gets reviewed.
Currently, I couldn't be more excited to start my new position in which I'll be responsible for speed, reliability & security of millions of cloud transactions in a microservice backend.
That being said, let's see how I feel in 6 months. Experience need to be made by oneself.
Real microservices implementations don't deliver so much from the proposed statement about synchronous web-tier request handling vs asynchronous compute workers in my experience. A few easy rules to guide this development are even touched on in the article, but treated like they could only ever be a tenet of monoliths.
Very strange.
Given how many fail to implement it right, it would help if we had a paradigm that caused less failure, but I don't see the author recommending anything except for going back to monoliths. I understand their perspective but I'm not sure this is an either or scenario.
The solution is probably some new paradigm we haven't figured out yet.
Not, I’m not a software developer, and I work on some of the largest software pipelines in AWS so my view might be distorted by my lack of experience in the field and the massive scale we operate at.
I spend like 98% of my time dealing with people problems. I will trade technical optimization for communication optimization every single time.
A pure microservice ideally has no state, or if it does, its own data store. That's awesome, until the front-end team wants a joined result (query) from multiple microservices, and thus data stores.
What are you going to do now? Build a proxy microservice in front of it? With shit performance, no referential integrity, and hard dependencies? More likely, you won't do that, so the front-end is going to be looping calls. 50 network requests with each having parsing overhead just to render a simple list of things.
The autonomous team has their own roadmap, which in a connected universe (the one I live in) is a problem, not a solution. The business team really needs that mobile app to be shipped soon, but the micro service team has no room on the backlog for another 6 months to build the part needed. A business prioritization problem? Perhaps, but it just shows that autonomy is largely a fantasy. The reality is that as soon as something is "one team away" everything becomes dramatically slower and less flexible. That's the price of autonomy.
All of this is like a 100 times worse than the ridiculed traditional software stacks, but hey, I'm not complaining. In a perverted way I benefit from delusional tech choices. It keeps me paid.
I was watching as the industry went all in on something fundamentally wrong. It didn't make anything better, it made everything worse.
Now we are looking back and wanting monorepos, monoliths with simple code, and low complexity.
1. The Microservice platform itself is mature and stable. See: VMware Tanzu. 2. The devs implement traceability. See: CorrelationID 3. There is good cross-training between the Dev and SRE teams.
Actually. All 3 apply to Monoliths too.
The grass is always greener, as they say...
1. Because many developers have no idea that it is possible to produce a “local modular application”.
2. Plumbing, technologies and deployment pipelines have become more important than a business problem domain.
People from big companies are also doing microservices but they don't necessarily call it that. They use typed inter-process communication technology such as protocol buffer / thrift etc, and work in a monorepo with statically typed languages. This makes things much more likely to work.
I suspect that in general smaller companies doing microservices should move in the direction of what the bigger companies are doing.
There is no answer to that, because "microservice" is a meaningless buzzword at first place, like "cloud", "no code" or "serverless". These are mostly marketing concerns, not engineering concerns. Marketing has taken over web development.
I think this should be tought in schools
We managed to write more performant code even though we're calling off to 10+ services each request. Did microservices make our applications faster? Of course not, but they clearly exposed the issues with our monoliths. We could make the same applications even faster if we moved back to a monolith, but the worry is when does it get back to the state we were in before?
It was not uncommon to have multiple pieces of code calling into the database to grab the same data in our monoliths. The fact is that it's way too easy to just DI a service where it doesn't need to be and boom, a pointless DB call. Do it in a few more places, add a bit of a spiderweb here and there, and you've just amplified the number of database requests doing the same thing. Yes, a lot of this comes down to fundamental issues with the architecture of the applications, things that have been stacked over years and years of tech debt. It's not an issue with a monolith, but rather how developers often treat them. There's a sense that everything is on the table because it's all in the same code base. A lot of developers don't care to think of the future implications of injecting a service where it doesn't belong, it does what they were asked and the person reviewing it thinks the same.
With microservices it feels like the decisions and behaviour of them are more public and there's more eyes seeing what they're doing. If someone is doing something weird, it's easier to call out since these interfaces are public. Previously all the shit code got hidden in random pull requests. Now the shit decisions are out in the open. Everyone is interacting with the same microservices, a shit API is shit for everyone, a slow service slows everyone else's services down, people seem to care more and make better decisions. There's still those guys who just don't give a shit and make a microservice into a macroservice. But when that happens it's easier to see now, it's in our faces, it's not 500 lines of code hidden in a library, it's easier to call out.
As time goes on I do long for a monolith again because personally I've learnt a lot from breaking down our systems into microservices. I know what touches what, I know what shouldn't touch what. The domain knowledge gained from this project would indefinitely lead to a better engineered monolith. But at the end of the day, microservices force these decisions into the open and less architectural mistakes are being made, which is good.
This is also a big reason why I'm a fan of Elixir/Erlang, you're almost forced to think in microservices and that leads to better decisions.
One mistake I think a lot of people make is creating a web of microservices. You want to keep the hierarchy as flat as possible so that each microservice is entirely independent of another. When you want to actually do work, you write an orchestrator that calls into each of these services to carry out the work. This orchestrator is not a consumable service, it's console app, it's a website, it's a product.
I've used "microservices" as an argument before now in situations where switching to a microservice architecture for that part of the code was in and of itself not really a significant advantage (though not a significant disadvantage either) but it bought me the opportunity to clean up everything else about that functionality with an extremely good end result that I don't think I could've got without selling the change that way.
What are the odds the companies aren't already doing the best hiring they can?
I think the author has correctly identified the reason but the given explaination is wildly off base. There's a much better answer to this and interestingly it predates the term "microservice" by around 40 years!
In 1968 the Conway, Melvin E. had his paper "How Do Committes Invent?" published, you can read it here: https://www.melconway.com/Home/pdf/committees.pdf (since the article author mentions Waterfall which also has an excellent paper behind it; this one, like that, is super readable and accessible).
TL;DR "organizations which design systems (in the broad sense used here) are constrained to produce designs which are copies of the communication structures of these organizations"
I built my project in a microservice architecture, so when the author states that microservices were built so that teams can enjoy their independence, it doesn't apply to me as I am currently a 1 man team.
I split my services up for many reasons, but primarily two, the ease of migrating from python to go as the ease of rebuilding one service at a time differs as opposed to a full blown rebuild, and also for performance and scaling as parts of my application will be hit harder by outside requests than others.
>Web requests can be managed by one type of instance, that results in one EC2 image or whatever. Anything that can be handled within the lifecycle of one request can be handled there, and these instances are horizontally scaled behind a load balancer.
The author gave an extremely simple and generic system design, and while this can work for a sizable amount of applications, there are still a significant amount of applications that require and demand a more complicated structure.
One of the services that I have split, almost exclusively deals with real time connectivity with websockets which requires a towering amount of performance as opposed to the other services that I have - to place these in a monolithic structure, scaling would be incredibly awkward - imagine adding 10 more load balanced boxes just so you can handle your websocket requests, but now the part of your app that deals with all your http requests is now also horizontally scaled when it didn't need to be.
>On the communications front, internal web services are often doubly inefficient by using REST rather than binary transmissions. There’s no reason for any of this, and if multiple microservices hops are used, this all adds up and slows down the system. Even just a conversion to JSON and back is a wasted effort, more so if done dozens of times.
This is true - while initially developing internal communication with GRPC, I've reverted back to http despite the 30-50ms TLS/SSL handshake simply because its a tired and true technology.
GRPC is still relatively new, with seemingly insubstantial development, and I was afraid to proceed further as roadblocks and technical debt may be accumulated in the future.
However, in a smaller mesh, in my opinion, anything shorter than ~150ms is insubstantial in my opinion - A blink of an eye is just about 150ms, and if it is the case that the internal requests are hitting multiple endpoints before resolving the original request, then it's more of an architectural problem, not that "microservices" are a problem.
Not everything has to be finely chopped, but breaking some portions down can make it more digestible
This is a ridiculous strawman. The point of microservices is higher-level architecture and design: to have discrete components in your system that fulfill a single responsibility — do one thing, and do it well—, so that reasoning about the entire system as a whole becomes simpler. It allows the microservice to expose its implementation behind a well-defined API, and thus keep its privates private. Additionally, microservices permit those (sub-)systems to scale independently.
Monoliths can do some of that, but often the "keeping the implementation private" is the hard part. When every part of the system has access to a database, people will reach behind the API & just get the data they need. It's not impossible to prevent, per se, but having the service entirely separate makes for a much better, stronger separation that forces the API design & planning that would not otherwise occur, as it would otherwise require a level of discipline that I don't think today's PMs and "agile"/scrum permit to exist.
Now, the article tries to address one of my points,
> In the usual web application, this is not a problem, because load and tested ensures each VM will be tested to it’s autoscaling parameters, and then it will grow.
No, it most certainly does not… I've seen plenty of VMs in my career running monoliths that were 90+% idle and most RAM free, because the application was bottle-necked on the database. And even if they're "monoliths", there's inevitably some other service not part of the monolith (either b/c it is third-party, or what) that then gets its own ASG, it's own set of VMs for redundancy … and it is waste. Never have I seen exactly 3 VMs running exactly 1 monolith.
(& VMs are like the worse case, too, as they are inevitably hand-crafted snowflakes. But worse, if a dependency, such as a package, is required… what part of the system required it? If I remove a use, is the package still required? Answering these requires reasoning over the entire monolith, something that, once the codebase is big enough, becomes effectively impossible.)
> On the communications front, internal web services are often doubly inefficient by using REST rather than binary transmissions.
… this is beyond wrong and it doesn't make any sense. You can serve Protocol Buffers over REST … is that not a "binary" transmission? (Not to mention that HTTP/2 & later is a binary protocol…) Sure, many people use JSON today, but there's no requirement to do that, and I've written several RESTful endpoints that didn't serve or consume JSON. (Generally because the requirements were such that that would make no sense.) The protobuf vs. JSON is a whole different debate, and each format has its pros & cons, but it is certainly orthogonal to the question of whether microservices are good or bad…
> Code that needs to be shared between the asynchronous services and the web tier should be kept in libraries used by both of them and is not a service call.
If you're going to do a monolith, yeah, this is what one should be doing. I've just literally never seen it done. (In fact, I've suggested it, mulitple times, when the second, third, fourth, fifth use case comes up: "we have code for that, but it isn't in library form. Let's solidify it into a library, & then change the existing consumers to use it, and then your use case is just another consumer" is inevitably met with "but I just want to do $whatever_it_is_thats_the_use_case it's just one more instance of this code, what could it hurt?" … followed by "why am I hitting $corner_case" and "well, that's some old organic growth; the original code, that we would have turned into a library, handles that…")
> The number of which does not really matter, but in a world of 200 microservices
This is strawman is repeated in every "I hate microservices" article. I've never seen microservices taken to that extreme, and yeah, if that's what you're doing, I expect you're in for a world of hurt. But that's not the point, and I doubt you actually have 200 well-defined systems with well-defined boundaries & APIs. But yes, if you take something to the absurd, it breaks down?
Like evolution, you would think the technology gets better through natural selection. However, selection pressure in the real world is vague. The best doesn't necessarily win (just what works) and microservices is a form of genetic drift.
The problem you're trying to solve, the size/expertise of your team, scale, customer expectations, legacy integrations.
As long as there's no catastrophic failure nature doesn't always necessarily choose the most fittest mutation. Just what works better than the competition. And there are multiple factors at play here. A company with better marketing could do better than a company with better technology, hence selection pressure on technology is negated.
What ends up succeeding is the cohesive whole. Every metric of the company including the CEO, funding, marketing, luck and everything else being the best defines "fittest". This means if everything else is the best but your technology is the worst you still succeed. Hence bad technology continues to propagate and exist within the industry.
It's sort of the same reason why cancer still exists. Why hasn't natural selection eliminated cancer?
The only time where microservices truly exist is if you're part of a big company that has a multitude of initiatives. But within a scope of a single project, there's almost always a mothership.
Not the apt, "The cargo is from a plane!" but rather, "We never needed any of these supplies to begin with! We were better off starving!"
It's wholly unsurprising that the next series of topics the author plans to discuss are mental health. That makes perfect sense, given the rest of the article.
The same people who said to me that the pandemic would just last a few weeks.
The same people who though a PT Cruiser was a beautiful car.
The same people who believed dogecoin and NFTs would make them rich.
People keep willfully failing because there's no incentive for them to be credible and accountable... There's just all the money they can make from perpetuating lies.
Sometimes we need to unplug the microphone, or ask the more quiet individuals what they think.
HOWEVER, I don't understand why people get religious about methodologies. Every time a new fad comes out, I don't look at it as the "One True Way", I just evaluate its strengths and weaknesses and add it to my toolbox. I also am not religious about following all the precepts of the new religion. I'll mix agile and waterfall if I want. I will conduct my daily status meetings how I want and call them scrum, even if it's just to piss off zealots. I will sit down during standup. I'll mix microservices and monoliths. I will figure out the value I want from a methodology and be happy when I get that; I don't believe in utopia anymore.
If you know the spec for inputs and outputs, wouldn't you be able to do the exact same in a monolith?
For microservices you need a good idea of how you’re going to do IPC and how you’re going to maintain isolated state. In most cases there are reasonably easy solutions to both problems, but in general there is a little bit more more up-front work to do when starting with microservices.
For monoliths you need a good understanding of how you are going to upgrade a running system without downtime, and how you are going to stop developers from taking stupid shortcuts that create invisible internal coupling. IMO, if your team is big enough to have the necessary processes to get this right, it’s big enough to deal with microservices too.
So I honestly don’t get the hate from some on HN for microservices. You can fuck anything up if you try hard enough, but microservices epitomise the principles of good systems engineering, most particularly around separation of concerns. I honestly don’t understand why anyone would choose a monolithic architecture for a new build in 2022.
The people who (at least initially) advocated for microservices were the first to point out that it is better to start as a monolith and then refactor into microservices as required by external factors.
I know the advice is to build a monolith first, and I used to think that too. But I now think it’s probably not good advice.
Each to their own, but I won’t be building a monolith again.