GitHub CTO – Biggest architectural mistake was going full microservice
twitter.com
twitter.com
Why did they care? Because their particular hosting model at the time really couldn't handle complex architecture, so microservices speaking to each other over HTTP was the only way to do things.
The arguments in favor of microservices at least sounded good enough that they were adopted at a lot of orgs before anyone actually needed the supposed benefits, and it turned into a big mess.
I've especially found anyone that is an "enterprise architectural consultant" to be especially suspect.
I fail to the the point in this cheap ad-hominem.
Uncle Bob is arguably best known by his seminal books: Clean Code, and Clean Architecture.
Clean Code talks about principles such as "you should name your variables well". Why do you believe scale is any relevant?
Clean Architecture talks about a general principle to organize software projects which has traits that are highly benefitial to maintenance, refactoring, and even re extensability. It's a nuanced take on the tried and true layered architecture. Why do you believe scale is remotely relevant?
I know personally Principal Engineers of a couple of FANGS who are great at selling ideas but are outright awful at writing code. One even famously handed POCs to SDE1s to have them rewrite and refactor his code to get some quality in it. Do you really believe in this appeal to authority fallacy to the point where the number of customers hitting a server are the measuring stick for competence and the quality of implementations?
Also, some of the advice is just provably nonsensical. Like "uncle" bob says the "ideal argument's number for a function is zero (niladic function)" which is absurd. It either means your function is impure, or is literally a constant.
Most of it is just very shallow stuff that's mostly applicable to Java/C#/OO languages. A lot of the patterns might even be considered antipatterns in other languages.
I just don't find stuff written by these guys very insightful or useful.
you shouldn't start a new project with microservices, even if you're sure your application will be big enough to make it worthwhile.
Thanks for the awesome advice!
He plays a lot with the Heavy Cardbord group. [1]
I haven't paid attention to his essays as I found them to be way too long and with too much prose and background.
This take is pure nonsense. It's highly disappointing to see this blend of ignorant and uninformed take in HN of all places.
Amazon's serverless offerings are the ideal tool for many use cases, specially internal services. Instead of being forced to write, deploy, operate and maintain a dedicated service to handle events, just write the event handlers you need, get it to run in someone else's computer, and forget about it.
Internally, things like AWS Lambda saves a lot on wasted allocated capacity. You don't need to waste allocating a whole box to run a handler each time, say, someone uploads a file to S3.
Developers don't need these "tools" pushed by marketing departments at first place. These aren't technical solutions, these are marketing considerations, solutions looking for a problem AKA "hype driven development".
When losing an argument, some people resort to the "it's just a tool, use the right tool for the job" cop-out instead of admitting it's a bad tool and moving on. I don't think it's a good argument at all. It's a phrase you can use to defend anything.
What is defined as complex architecture/could you elaborate?
Isn't it usually "a bunch of microservices talking to each other" is complex?
> What is defined as complex architecture/could you elaborate?
Earlier in its lifespan, Heroku didn't have an easy way to add common components of infrastructure to an app. If you wanted to use Postgres + some Postgres plugins + Redis + some kind of queue manager, you basically had to split those out into separate apps that would run on their own "dynos" (to use the Heroku terminology).
That makes sense, right? We all do that now. We don't usually have Postgres running on our application server.
The difference is that Heroku only let those things talk to each other over HTTP at the time, so you essentially had to put a REST API in front of every component of your system.
It was, predictably, a huge mess that I honestly didn't see many companies fully adopting.
The right answer is pretty much always medium sized services and medium sized repos. I guess that isn’t edgy enough to make people feel like they are doing something cool
Saying medium-size services are best means that the engineer needs to use their judgment to determine where that threshold is. It becomes more of an art than a science, which many engineers are deeply-uncomfortable with, so they stick to their dogmas.
If the decision to use metric or imperial bolts on a bridge were up to the individual builder...
Picking a solid "do x" or "do y" makes for much simpler pull requests.
I currently work on a project with some GraphQL, some basic json and some json-api. Some of the app is BFF and other bits are not client to the micro.
Using our judgement is working out a treat.
Let'em come and tell me I need Kubernetes everywhere! To many people think it's black/white and that there is no grey in between. And more often I realize that the grey is even larger than any of the (hard) sides.
I can only speak for the places that I have worked, but I think that what he calls an "App" is what would likely get called a "Microservice" in a lot of places.
A poorly designed monolith is pretty much always going to be superior to a poorly designed microservices architecture. The floor protecting you from bad design is significantly higher.
The biggest mistake is that people tend to create services that are far too small, and too many of them. Then it gets chatty and you develop a distributed spaghetti code. Or failing to divide them into coherent domains. Or overindexing on scaling individual functions/cost rather than logical boundaries
You should almost always err towards services that are too large rather than too small. You can always subdivide further later on.
I interview tons of senior engineers and managers from reputable companies that describe designs that would produce immense technical debt
My take on it is that we should have called is "distributed services" because micro implies that they have to be small.
Usually there are clear borders in your app of stuff that can me compartementalized, but they usually should be called medium services instead.
Also - your database is a microservice - design your data acces layer accordingly, you dont need an api endpoint, where a stored procedure (or the like) will do.
> Usually there are clear borders in your app of stuff that can me compartmentalized, but they usually should be called medium services instead.
These are excellent points and probably loosely coincide with how many teams you might have working on different parts of the greater system as well.
> ...you dont need an api endpoint, where a stored procedure (or the like) will do.
I'm not sure about this. Outside of niche cases and domains (like batch processes), stored procedures are harder to develop, harder to test, harder to debug, harder to audit (or at least less convenient to, when you want to ship your logs with something like GELF) and also have worse tooling for working with them, ensuring code style rules and so on.
Just look at the reality in many places: https://www.jetbrains.com/lp/devecosystem-2021/databases/#Da...
Do you debug stored procedures?
47% Never
44% Rarely
9% Frequently
Do you have tests in your database?
14% Yes
70% No
15% I don't know
Do you keep your database scripts in a version control system?
54% Yes
37% No
9% I don't know
Do you write comments for the database objects?
49% No
27% Yes, for many types of objects
24% Yes, only for tables
While database design should be a major consideration that begs attention, I personally maintain that the majority of the functionality should go into the app, regardless of whether you use an ORM or not. An exception to this would be using plenty of database views to make querying easier and putting consideration towards how one could handle enums and such, so you don't end up with a database that's not possible to understand without reading the app source code in another window.
If you have someone in your org specialized on SQL debugging stored procedures is their job. By now there are also many sufficient tools, that will help you actually debug the code and it has been a first class citizen in visual studio for years now.
Also keep in mind, that most APIs are crud, not complex businness logic and that there is a middle ground: build your own libraries instead of microservices and share those.
This of course all is heavily debateable and only based on my personal experience, but your statistics are missing the value for the amount of people who are actually "good at sql".
Also a lot of people dont understand the value of relational data at all. So yeah, if you never learned to code you cant program. Sql is not just a second class citizien of your stack that "stores stuff" and of course you have to optimize your use cases, but again this is exactly what i mean with my top comment: you sir are dogmatic too :D
SQL is AMAZING and can be directly conected to your frontend. That does not mean its a smart thing to do.
> This of course all is heavily debatable and only based on my personal experience, but your statistics are missing the value for the amount of people who are actually "good at sql".
That said, one cannot just optimize for people who are good at a particular technology and instead one needs to look at the market averages, for the locale that they're in (unless they have that many choices of what culture to work in in the first place). In my case, where I am, we are still mostly on board with ACID and RDBMSes, however DBAs are, while not exactly a fleeting career, on the downturn (probably about 20 years away from being as obsolete as "GlassFish Server Manager", much like the DevOps trend digs into Ops engineer positions).
Sure, most people are okay with running EXPLAIN PLAN occasionally, but putting significant amounts of logic in the DB is usually a pain to deal with. I've seen first hand how much of a dumpster fire a system that has like 90% of the logic in the DB eventually gets: stored procedures for rendering forms, stored procedures for data validations, stored procedures for persisting data, with absolutely ALL of the issues that I mentioned before. Making changes in it was a mess, figuring out which of the hundred or so packages even needed the changes was a pain, the naming was nonsensical (due to length limitations for DB objects), it's like someone tried coding in a language that wasn't meant for writing proper code.
Of course, I've also seen some bad projects in the more traditionally tiered architectures, but none were as bad as the DB-centric design. That's my anecdotal experience, on which I'll base most of my future tech stack decisions. Of course, sometimes in-database processing is just the right way to go about things, especially in something like PostGIS where you don't have many options app-side for handling certain kinds of operations on the data, or need to do them without N+1 issues or too much network I/O due to latency.
Generate a Data Access Layer( https://en.wikipedia.org/wiki/Data_access_layer ). That does not mean that you should be DB-centric, but DATA-Centric. If your stack is written in one language, create a lib, that will be the key in going forward.
Application logic as a whole can be distributed throughout thousands of services but your DAL should be a common base. - at least for things that are in one database -
Here you can use procs, views or queries depending on what you like, but once i hit a couple of million guid entries in a db, i can in fact sometimes strip seconds of queries by saving on the round trip. Especiall in ERP Apps that "need to have all the data allways". Dont convert everything to stored procs of course because then you are just as unflexible as having a microservice per entity.
All in all, I don't believe databases are ever gonna go away and that dbas are still underrated, even though data is gold. As we have seen with machine learning, well defined data sets are the key to better models, meaning faster and quicker results.
It seems to me that after ~10 years, half the cargo-culted ideas that young developers go all in on fail. Perhaps that's just long enough for them to get older and gain experience.
Perhaps the real issue here is the "thought leaders" who should know better but are perfectly happy to sell their ideas as risk free, when in fact these things are incredibly context sensitive to the team, problem and company shape.
Nobody ever seems to say "Do you believe what you're saying, or are you simply collecting a speakers fee?".
This is a great question to ask, though I think one should also apply that same way of thinking to frameworks, languages and a lot of the other software out there (like OS distros or databases).
Probably while looking at the Gartner hype cycles: https://en.wikipedia.org/wiki/Gartner_hype_cycle
Sometimes there are good ideas that turn out not to be feasible, whereas the same can happen with the technologies. Whereas if a technology has been around for a large number of years and hasn't died yet, then that's a good predictor for its continued existence: like Linux, or PostgreSQL.
> It seems to me that after ~10 years, half the cargo-culted ideas that young developers go all in on fail. Perhaps that's just long enough for them to get older and gain experience.
Another thing I've noticed is that sometimes the "spirit" of the idea remains, but the technologies to achieve it are way different. For example, we recognized that reproducible deployments are important and configuration management matters a lot, however over time many have gone from using Ansible and systemd services to shipping OCI containers.
These technologies bring their own cruft and idiosyncrasies, of course, but a lot of the time allow achieving similar outcomes (e.g. managing your configuration in an Ansible playbook, vs passing it in to a 12 Factor App through environment variables and config maps).
Of course, sometimes they bring an entirely new set of concepts (and capabilities/risks) that you need to think about, which may or may not be intended or even positive, depending on what you care about.
A bunch of best practices around data handling, security, and general application architecture for specific use cases is useful.
Do my great new idea now or lose out is really awful and I've witnessed that kind of belief do irreparable harm to companies.
There's some parallel here to draw between the "new tech hotness" that appears every few years and the dancing fever plagues.
There is definitely still a lot of opportunity to breach the boundary between development and production to understand distributed systems
Could this have been working around with https://www.jaegertracing.io/? That was one of the requirements at the organization I worked out.
We didn't go as far as to block traffic if the request didn't have a parent span (which included the service name calling it) but if it became a problem I'm sure we could have/would have done that (like 200+ microservices for fintec/banking across like 10-15 teams) and figured out pretty quickly in CAT/UAT/QA environment what wasn't passing a span.
But the invasion hasn't stopped. Most in the Ukraine are surely utterly exhausted from hearing decades of their infrastructure, cities, housing and culture being destroyed on a daily basis. Exhaused from the daily disruption, broken families and the mental and physical impact of having to actively protect themselves or fight against it. But they have to keep living it and fighting against it, daily, right outside their own doors. They don't have a choice.
I am sure most in Russia are equally exhausted by the daily impact on their lives of sanctions, fear of reprucussions for an opinion let alone any attempt at change, by being implicated in actions and conflict they don't agree with or by being effectively forced to personally and physically go to the front lines.
It is exhausting, it is uncomfortable, and that is OK. Globally we need to keep rallying against this fight against Ukraine. But maybe even more importantly, especially if you want to be selfish about it, we need to remember and be uncomfortable so that we're part of avoiding the next conflcit from starting.
For your personal mental wellbeing, if you're overwhelmed which I'm sure you are given the comment, my best piece of personal advice is to try and take a solid break from the news for a while. At a minimum try to avoid the hourly and daily scrolling on social media that exposes you to so much negativity and minimise the time on that to a smaller part of each day or week. It's really hard to do that, but it's the best I've got.
I agree with some of your points, but not this. Personally I don't have any need to have a background level of discomfort in order to stop the next conflict. This one was started by a deluded megalomaniac who did not consult me (or my country for that matter) before he invaded a neighbor and placed us all at risk of WW3. I'm sure the next one will likewise start without my input and against my wishes.
Shup the fuck up
It is not for the lack of solutions in hindsight, just the constant feature creep and microservice development blind sighted everyone to large organization problems.
Start as microservices and you get a bigger and dirtier ball of mud. Guaranteed. Because you have no notion about boundaries at start. Monolith at least have luxury of automated refactorings.
1) We need to split this thing out because we need to isolate it
Problematic. Generally the issue is political--not technical. The service is necessary but not getting enough support--isolate it to force fixes. Some service not being responsive enough to internal customers--isolate so tickets now can be assigned and blamed on them. etc.
2) We need to split this thing out because it's a perfromance bottleneck and we need to scale it
A reasonable choice. The people scaling something need to be able to bound and limit the scope or they'll never make any progress.
The issue is that #2 is VASTLY rarer than #1.
But if you have a problematic part of your codebase that regularly shits the bed, keeping it in your monolith will take it all down. Splitting it out is the smart thing.
“If this services fails we can carry on working with a degraded experience”, notifications on Twitter et al are probably one example of this
Yes, design your applications in a modular way such that it becomes readily possible to stub parts of it out when the time comes. Don't start with a microservice focus or you're just begging someone else to beat you to market.
From his tweet thread. I think it's like a separate product except it's under the same company or domain name. Many companies do this. A product could be just a backend server providing API with its own database. All that is scoped by "value" .. or a target market.
This could imply that when micro services start to introduce to a system, we should question the value proposition of the whose system. We could mix up the engineering and business aspect.
Apart from that THANK YOU Jason. Finally someone with some common sense!
And for all you tech recruiters hiring with microservices and cloud native , get a fucking grip already.
Our backend is microservice based because it's a thin layer over storage. There are functions that do processing work, but those exist so the thin layer works.
The big problem with microservices is handling failure. If you are chaining microservices together you're screwed. Thats why stuff like graphql exists.
That said, the reason microservices exist is because bloat and dependencies eventually choke monoliths. Also monoliths are difficult to scale in a cost-effective way.
Of course an internal website with probably at most 1000 users should never have been written using microservices in the first place.
http://resources.1060research.com/docs/IntroductionToResourc...
> Monolith > apps > services > microservices
While I can imagine the difference between a service and a microservice, what's the difference between apps and services?