Microservices – Please, don’t
blog.komand.com
blog.komand.com
I've worked with an app that had been factored into half a dozen microservices. Each of these needed two EC2 instances for redundancy. They weren't using docker or any other container infrastructure and were treating EC2 instances as pets instead of cattle, which meant that a LOT of effort was spent maintaining the dozen or so instances. Each of these instances was running at < 1% load average! So in this case the microservice architecture was also costing a significant amount of money.
My takeaway: don't build an app from scratch using microservices, especially if all you're splitting apart is web handling which is almost never going to be where your application spends a significant amount of time. Instead, build apps in the straightforward monolithic way, and only split things out into microservices when there is a very clear reason that has been identified from performance monitoring or some other metric gathering.
If your devops is inadequate, so that you have low visibility into your product's performance, can't degrade gracefully, and have trouble deploying things without significant downtime, investing in proper devops infrastructure and processes may directly help your bottom line.
The set of companies that have any sort of reliability or performance-oriented need a "microservice architecture" is three, maybe four orders of magnitude smaller than the set of companies building web applications. They--and you--have been sold on a trend that is actively antithetical to the goals of delivering a product.
Build your application. Use an auto-scaling group and an ELB. You have more important things to do until you reach the scale where you can pay someone (hi) to do them.
I think most of your points are valid (wherever my experience is enough to judge).
A pre-requisite to "spend as much energy as possible on end user features" is to have some sort of foundation to build on that can grow with your platform, or you might as well build on quicksand.
I think there is a middle ground. Especially if you can start writing common libraries for the other components to use.
Unfortunately, this requires an actual architect to think about the design of the monolith ahead of time, and most startups either have trouble recruiting anyone above the junior level, confuse the responsibilities of the project manager (who, in the early days, is typically the CEO/CTO) with the responsibilities of the architect, or simply cannot afford to put an architect on payroll.
There is sooo much truth to this statement. Never be afraid to put down an instance, and if so your doing it wrong!
Doesn't EC2 already have redundancy built in?
2. People who have problem P pick up X and say good things about X
3. People, usually junior devs, read blog post about X. They start treating X as solution to every problem they encounter and ofter take it to the extreme.
4. After realizing it's not helping, they blame X and write posts like this.
You can replace X by many things, starting from methodologies like microservices or TDD, ending on things like nosql databases.
A guy with 30 years of experience that has only exposure to imperative programming, and never really learned OOP, networking, functional programming (it's not new: lisp, scheme, haskell, etc. have been around for decades), never really learned about operating systems, never really cared about automation, testing... unless you work on implementing software at a low level, which is very rare these days... guess what, that guy has a lot to learn from a fresher engineer, or engineers that actually cared about learning.
Since you brought up functional programming, it actually fits the same framework.
In that case P includes problems with a "pure" domain (e.g. compilers) or problems that require lots of multi-threading.
It does not work well for problems that are fundamentally state-full or sequential. It also does not work in "messy" domains - e.g. data mungling.
Someone who is biased towards "getting job done" and seeing technology adoption as a mere distraction, is simply too biased to be of any help during decision making. When evaluating technology adoption, you need to see the pros and cons (including adoption costs), and evaluate things seriously.
I think it's junior ones who lack understanding of how is OS and networks are operating, as they don't needed most of the time in more popular industries (mobile development, web, etc).
source: by the chance I conducted few interviews for software engineers and devops in web industry - and instead of fizzbuzz I started with asking what's the difference between TCP and UDP. Results will surprise you (also too many people want to exploit current hiring system by just cramming algorithms).
I can only think of a couple examples: data structures and some APIs.
Whether data structures should be objects rather than primitives depends very much on the data.
I love both FP and OOP and especially love using them together. I honestly don't get these arguments.
Objects are merely collections of partially-applied functions (or often procedures): methods are bound to `this` aka `self`, the instance. When you have a language where partial function application (currying) is a norm, you have little trouble implementing this approach.
What made OOP popular, things like good modularity and hiding of implementation details, has little to do with the "inheritance + encapsulation + polymorphism" trio proper. They can be and often are achieved without these.
What makes FP powerful is higher-order functions / function composition, immutable data / pure functions / referential transparency, and isolation of effects. Technically, all of these are easily available in most popular OO languages, and you can reap many benefits of FP writing in C#, Python, or even Java 8. But the standard libraries of most of them actively resist such usage, and the translators do not rely on the functional features heavily for optimization.
Where pure FP has trouble is e.g. large mutable structures with constant small mutations, like editing a bitmap in a paint program. Real-time stuff can also be harder, because the shape of the machine code is much less obvious than when using a lowest-possible-level language like C.
Since the real world is inherently effectful, there will always be a mix of approaches. Pick the right tool for the job; a job of any serious complexity will likely require multiple different tools.
> junior devs
mostly don't make architectural decisions. It's a bit easy to keep on blaming "junior devs". Who keeps on hyping up and marketing useless stuffs ? that's not "junior devs", that's big consultancies that want to sell their crappy services, that's PAAS that want to sell more servers, that's vendors that want to sell more useless stuff. Enough with blaming "junior devs". Junior devs don't lead teams and have very little say when it comes to architectural choices, they don't contract this or that consultancy, they don't decide whether a business will run on Amazon or Azure. Junior devs didn't invent NoSQL, Microservices, "Serveless" and all that bullshit. Well established players sell these services to others, and the people who drink the kool-aid are managers, not junior devs.
Do you really think "junior devs" want to maintain 50 micro servers just because it's trendy? when they could get away with a single one?
Another possibility: senior dev / architect wants to try it on a relatively small project, as a pilot to get a better feel for how it works and where it might be useful. Sure it might cause headaches and extra cost on that one project, but the knowledge gained might be worth it. The resulting blog posts are either public dissemination of that knowledge, or griping by people who either aren't interested in that big picture or who think the cost/benefit call was made incorrectly.
If there is an issue in one those services it takes a long time to figure out which service was it and why it wasn't throwing a 500 error. Even if they throw 500 status codes you still don't get the nice line number and stack trace in a large application gives you.
I think the whole microservices thing is a mistake, I as a developer should not care how this application is running in production. I want to work on a large app and leave the deployment business to someone else. I get it that it's hard to scale a big application but it shouldn't be my concern.
I miss the old .NET days where I could run the app on my machine...
People seem to forget that microservices isn't new. It's just SOA all over again.
Heck, I'd bet a lot of enterprise SOA is just a giant crufty monolith with a bunch of its functionality exposed via service endpoints. I suppose that a pretty decent approach - add endpoints to a monolith. As long as the service interfaces remain the same, you can bust it up into as many medium or micro sized services as are necessary on the back end and the users of your services won't notice the difference.
(Sub in any other reason to separate a service for 'heavy stuff' - written in a more appropriate language for the function, etc)
And in your case I don't know what you can't get the line number and stack trace from a microservice. Likewise you can get tracing between microservices using something like Zipkin.
Personally I think the "truth" lies in between. Bigger services that cover key functional areas.
This. What the GP was talking about is something like nano-services. Who really writes a whole service stack for every function?
In addition to functional areas, I also like a service when a single job needs to get done. For example, read message from queue, do some processing, put result somewhere. The beauty of the service here is I can write it once, test it, deploy and it just works. I don't have to worry about it randomly getting broken from code changes in a larger monolith.
I can't imagine being in a place where I don't care about how my code runs in production or scales.
Whether it's monolith or whatnot, what you really want is the "good dev team" part.
Then, with .NET it is much easier to end up with gigantic spaghetti solutions with projects that are all coupled to each other.
Coupling is the problem, learn to control coupling. If everything needs to know about everything, how do you expect to have a team that is capable of maintaining that?
Our DARPA Grand Challenge vehicle in 2005 was all microservices. There were tasks for each sensor, a location task which integrated AHRS, GPS, and map data into "where are we now", a steering task, and a planning task which took in the sensor data, built a local point cloud map, and issued steering commands. There was also an emergency stop task monitoring the radar and able to stop the vehicle if it detected an incipient collision, independent of the mapping systems. The non-hardware tasks could be run separately, or in a simulator, and we could tap in at a boundary and log.
This was all QNX, hard real time. This part of the system worked quite well.
Synchronization was unusual, in that the sensor data all came in at different times depending on the sensor. So we timestamped everything at the millisecond level, and interpolated between sensor frames to get values for all sensors at the same instant. The overall update rate for planning was 100ms, rather slow by modern standards.
I feel as though about 80% of the programming blog posts that breathlessly pontificate about why everything you're doing is wrong and what you should be doing instead would all be better if replaced by a phrase along the lines of "understand the problem you're trying to solve, understand the possible solutions at hand, and pick the one most appropriate for your problem and your circumstances."
Instead, we often see dev teams chasing off after the newest and shiniest options available. And sometimes, the newest and shiniest option is the best one, which is great! But often, it's a far from optimal. However, in an industry where so many companies hire largely on the basis of buzzword compliance, I really don't blame developers at all for choosing the new, hot, and shiny solutions over proven solutions that are better solutions for a wide class of problems.
In theory you could avoid the integration hassle if each service is so well defined and "loosely coupled" that I can be assured it will work without having to test with all the other services, but in practice the architecture is never that well defined.
I think if someone would dislike microservice if they don't understand the following points very well: 1: The business model of the system you are working on. 2: how to design/develop business models – write a bunch of CURD services on top of database doesn't count. 3: define business boundary is far more important than you think, especially you are working on some projects for computerizing real world business operations.
Strongly recommend reading Domain-Driven Design book.
Yes, it's easy to get burned. Yes, it's often the absolute wrong tool for the job as a young startup. Further, its all too easy to just light cash on fire once you get some real serious traffic hitting all these things that may autoscale individually. We have seen it all before, and we will see it again.
But don't throw the baby out with the bath water. In fact, I think the healthiest approach is to drop the "micro" and think of classic Service Oriented Architecture (SOA).
When you hire your 50th engineer, and you don't have this yet, its not the end of the world, but slowly moving from a monolith to a SOA, is pretty much a very well travelled path, and at a certain scale is how you maintain sanity.
It's like the old saying, the first code you write is never the best. Try the 2nd or 3rd rewrite, and you finally know what you want. So start with a monolith, but don't back yourself into a corner, and move one obvious service out at time.
The main benefits are not even what the article gripes about. It is compartmentalization between teams, and honoring stable internal functionality between teams that it makes the biggest difference. Sure scaling CAN be easier, but it can also be harder. Things like managing connection pooling becomes easier with an SOA. But not to the tune of doing it concurrently across 40 services, that is irresponsible to design without knowing exactly how they are all going to get hit when you finally get insane traffic.
One thing the author said that I agree should be more obvious to people is that somewhere in-between hiring engineer 12 and 50, you will end up running your monolith a bit like an SOA, where even if its the same codebase, you can run in different clusters/pools and service a specific part of the application. This is a wonderful way to get the benefits of SOA, with out the nightmare, furthermore, you will quickly learn which of the parts of code, when your rewriting it the 2nd time, you actually want to divorce from the monolith, and you already have patterns, load balancers, and metrics for what that code should do.
When you hire your 200th engineer, it's easy to get on HN and say SOA is the only way to get the job done. But people need to realize the importance of the journey of getting there. Otherwise your just taking a shotgun to the skateboard your cruising down the road with.
Sanity is the key takeaway. Always optimize for sanity.
It's an approach to organizing the responsibilities of a code module, and abstracting implementation details such as what technology stack is implemented in, what operating system is running on, etc.
By making services depend on each other through a protocol like REST (or Thrift, gRPC or whatever), you can then replace that module by another one that respects the same interface... and you can reimplement it in whatever technology you want. I can have a module implemented on Elixir running on FreeBSD talking to another one running on OpenBSD implemented on C, talking to another one running on Linux implemented in node.js...
Similar to SOA, but SOA was much broader in scope. You had service composability, discoverability, etc. That's in most cases overkill for most companies.
> By making services depend on each other through a protocol like REST (or Thrift, gRPC or whatever), you can then replace that module by another one that respects the same interface... and you can reimplement it in whatever technology you want. I can have a module implemented on Elixir running on FreeBSD talking to another one running on OpenBSD implemented on C, talking to another one running on Linux implemented in node.js...
I said:
> It is compartmentalization between teams, and honoring stable internal functionality between teams that it makes the biggest difference.
Maybe we agree more than disagree?
You also say: > Similar to SOA, but SOA was much broader in scope. You had service composability, discoverability, etc. That's in most cases overkill for most companies.
Not sure if the normal way of thinking of micro-services does not include this. If so, that is a new definition to me. To me micro-services mean services which perform less functionality, therefore you need more of them. If you have a platform that has 40 functions-as-a-service running in docker containers, you will need discoverability, for example.
Now if you need more granularity than that, there's for instance Zookeeper ephemeral nodes, that get deleted when the client stops emitting heartbeats.
MVC outlines very clearly the areas of responsibility for the framework. But many of the popular MVC frameworks take it a step further, declaring that MVC should be the primary dividing lines for your app code and business logic, with little thought given to separation along functional/domain lines. (Or in DHH's words, "The framework is the application!")
So is it any wonder that 10 years later, the industry is freaking out because our nice, clean MVC apps became business logic casseroles? Microservices can help with that, just like MVC can help with the PHP spaghetti code that came before. But none of those things can substitute for basic design principles.
I wonder how things would be different if 10 years ago, instead of app/models, app/views, and app/controllers, Rails had promoted app/auth, app/billing, app/spline_reticulation, etc. Might such a simple thing have pushed us in a better direction, leaving microservices to the 10% of apps that really need them? Or would we all have followed that pattern to illogical extremes as well?
The same with Facebook and php or stackoverflow and .NET: a successful start-up doesn't prove it's tech stack is the best, it just proves it can work. Someone who left to build a new ambitious competitor is more likely to choose a better tech stack, e.g. Python for Quora.
But just because they may put all their code and commits in a giant common repository, does not mean that its hosted, scaled, and deployed as a monolith in current day.
From the grapevine, I know that some of their stack is c++, and some is java, as an example.
Also, google losing all their code, made me laugh. Maybe that is a possibility, but seems like they have to have so many backups and failsafes. But I get you said that to illustrate a point :)
Microservices make sense to me when the task is clearly abstract from the application using it (which allows the service to be used more widely), the nature of the service is such that it has significantly different resource requirements than the rest of the application (allowing it to be scaled independently), and the problem it solves is not trivial.
I would be happy indeed as a developer to walk into a project and have a handful of tools at my disposal for doing things like sending emails, event notifications, logging, encryption key rotation, etc. I just consume the API. It's the reason I appreciate AWS; they provide a load of features (i.e. microservices) that let me focus on domain problems and not things that have been solved a thousand times already. So as a thoughtful developer, deploying something as a microservice can be a prudent contribution that brings a great deal of business value down the road.
But it starts to get out of control. After a bit of success with a legitimate microservice or two you become a hammer and every function starts to look like a nail.
If you can't visualize the microservice you're building ever becoming an AWS product, popular open source project or service offered by some SaaS company, likely it's too domain-centric and should stay in the monolith.
"Better learn balance. Balance is key. Balance good, everything good. Balance bad, better pack up, go home. Understand?" - Mr. Miyagi
To me, Microservices are a pragmatic solution to build software in an age of complete chaos. Clearly it's not a silver bullet - but what ever has been?
As someone who has to manage a complicated application architecture with lots of pieces, very little time to think, a flat organization of full stack developers, and a bunch of executives who pivot every other week, it's the only scheme that works.
Would I like to go back to the old days when a team of 20 and six months would do what a team of 3 and two weeks does today? Hell yes. But, that's not reality - the world has moved on, and it's not slowing down anytime soon.
Startups these days are often about taking on as much risk as can be tolerated in a race toward dominant market share and exponentially positive revenue. And, Microservices help get us to that goal. It's often not pretty, no doubt. But, it's way easier to manage risk in that kind of environment. With proper monitoring, alerting, and automation - the risks go down a lot.
Some might say that I'm crazy to work in such an environment. Maybe they're right. Generally, this is all completely experimental. But, it's helped me scale to 25 developers and $25M revenue in 8 months. Is it sustainable? Who knows - but, I know that the alternatives certainly wouldn't have worked. In another 8 months, we will either be at $200M in revenue or will have died a horrible death - but we will have died because of the business model, not because of the software.
This is yet another race to the bottom, which as programmers , we should advocate against whenever we can.
When considering only production systems, anybody that has a nightmare microservice deployment that is unruly to upgrade, version, or maintain always had some story about getting to market in record time before hand. Shipping services as quickly as possible to get the feature out the door without considering the downstream impact. (my absolute favorite... "we can solve auth later with a microservice in a container")
I don't care if you have micro, macro, monolithic, or (pick a buzz word) architecture. If its not designed properly, it will always be a nightmare.
I want a framework/container that lets me set service boundaries in code, and if my services are deployed to the same logical cluster, a local call is made if there is a service to service call. This way, I can develop my services as a monolithic codebase, with sharing of common code, but at deployment time, service modules can be split up and hosted separately (or not.)
Also we need some forwards/backwards data type compatibility like you get with protocol buffers.
Also some simple human and business process workflow support built in.
"Given a choice of solutions, pick the least powerful solution capable of solving your problem" Be it microservices, OOP or whatever... keep this principle in mind.
- distributed transactions are hard to do well especially in write scenarios
- microservices often solve team issues better than technical issues
There's a fundamental mathematical truth that if you have N moving parts in a system, you're looking at possible O(N^2) ways the system can break. Now that doesn't mean that you build everything as one gigantic binary but it certainly means that system architects should be always be looking at reducing the number of moving parts not increasing them. The whole microservices craze is exactly the opposite of that.