Hacker News Reader

> This is not because microservices-first is inherently flawed. This is purely a skill set issue. Microservices demand a greater skill set. A 'monolithic' approach puts much less of a demand on the engineer's skill set.

> The truth is that effective use of microservices (knowing how to build architecturally sound software) is way beyond the reach of MOST smart, senior level engineers that I've met in my career. If this sounds arrogant, it's not; there's a definitive architectural skill set that CAN BE LEARNED but that is missing from most software practitioners.

If this skill set exists, and almost nobody can or does possess it, then your argument is purely semantic.

Who cares if there's an ivory tower of geniuses out there spinning out perfectly formed microservice architectures in novel domains? We're talking about rules of thumb for practitioners here, not laws of physics.

I had the same thought, but the argument has relevance if you accept that said skill set can be acquired (which, I do).

I never said genius. And "perfectly formed" is not the objective. Tractability is. Go visit companies with medium-sized codebases and find me one that isn't lamenting how difficult it is to grow and maintain it.

I wouldn't consider myself that smart. But I have spent 20 years in this industry with what some might call a sick obsession for understanding how to compose and evolve systems and I've acquired some skills that I believe (1) very much matter to our productivity; and (2) can be mentored and taught, but (3) are not being taught.

My guess is that academy can't teach it because they don't have enough real-world experience behind their pedagogy. And industry can't teach it because it takes quite a while to learn and master -- and what business can afford to do that with their juniors who are just going to leave them in a few years anyway :)

Start teaching it then?

I've certainly never worked for a company that did not have a laundry list of complaints about its software, all reasonably successful ones too.

And it usually only takes a couple of beers to draw the same laments from people working in the sort of companies a lot of developers aspire to.

If you think you've acquired a teachable, repeatable set of skills that aren't being taught then it sounds like you've got a gold mine on your hands :) As a relative industry junior (about 10 years) I haven't been introduced to a codebase that wasn't lousy. Certainly all were lousy to one degree or another when I left them, but they served customers all while drawing down the ire of developers (including myself).

> I've certainly never worked for a company that did not have a laundry list of complaints about its software, all reasonably successful ones too.

Either this is a necessary problem (nothing to be done) or this is because of a lack of tools and skills (i.e., our industry is still immature). I believe very strongly in the latter.

> If you think you've acquired a teachable, repeatable set of skills that aren't being taught then it sounds like you've got a gold mine on your hands.

I might some day figure out how to teach this in a larger (non mentoring) scale.

What the skill set looks like and how to acquire it is actually the lesser problem. The harder part is proving to people, quantitatively, the value of the skill set. I know and I think we all know (on some level) the extreme cost of architecturally-unsound software. But it takes a lot of work to learn & master the skill set, so most people bail on doing the work it takes to learn it. (The industry does not reward the growth or reward the mastery enough.)

Only when industry can connect the value of the skill set (and also be able to hire to the skill set) will more people put in the work to acquire it. (The value to industry is enormous but I think it would take some work, research, creative thinking to make the value explicit & tangible and give industry a way to test for the skill set.)

If someone builds me a house I can walk around and test the beams and come up w/ a general idea how well they did and how well the house will withstand the weather. In today's software world, two people with drastically different skill sets can produce a code base -- and it's really hard for the people with the money to see what they've been built and how sound it is.

In my experience, the reason all companies have shitty codebases is because of risk-compensation. When a company does not have a shitty code base, it quickly throws all of its resources into improving the product and user experience for its customers, which increases its market share, revenue, and moat against competitors. This continues until all the new complexity makes the code shitty again, where they stage a cleanup & architectural fixit to be able to make forward progress. If the codebase is not shitty, it means that they're leaving money on the table, and will continue to get shittier until code quality starts costing them money again.

Companies like Google, Microsoft, and Facebook actually do adopt (and often discover) many of the industry best practices. Their codebase is still shitty; the reason is that any cleanliness and rationality in their architecture is quickly eaten up by new features and products that drive the industry forward. As consumers, we see the benefits; as employees, we just muddle along as best as we can.

The only way to break this cycle is for engineers to pay companies instead of getting paid by them. I don't think this is what you had in mind. (Although, some engineers do make this bargain. This is why Haskell/Erlang/Ocaml/Lisp salaries are often lower than they would be for equivalently-skilled Java and C++ engineers: you take a pay cut to work in a language that's actually enjoyable to program in.)

Barring that, the way out is to learn some negotiating skills and at least get paid for putting up with a sucky codebase. If you frame it as "employing my skills will let your other employees implement the features you've been asking for for a year, which will make you $XM in revenue", you can make quite a pretty penny. I know some senior engineers that regularly get paid a couple million in restricted stock for a couple years worth of work.

I'd love to hear your take on what these skills are. We've all struggled with code bases and some are definitely easier than others. And some people tend to write code that easier to evolve than others.

A startup in general cannot afford to get the best people in the field, unless it's some super sexy bleeding edge tech (which is very rare). So, you are lucky for your first 10 engineers to have one great one, and hopefully no awful ones. And you cobble a codebase together, and the overriding goal is that it isn't totally broken all the time.

Growth first, maintainability last. That's the incentive in a startup.

Unfortunately, in larger companies most managers don't reward improving the codebase unless you are on the platform side, just new shiny new features. Nobody will love you for saying this quarter I removed 2000 lines of shitty code.

Microservices aren't particularly hard to build, but they are brittle as hell.

The serialization/deserialization layer adds potential for bugs. The massively increased number of services all need constant monitoring. The massively multiplied error conditions (what happens when service 37 talks to 39 and 39 times out?) need to be accounted for and those edge cases must be ironed out.

There would be nothing particularly wrong with doing all of this if you gained something great by doing it, but you don't. You get fuck all except migraines caused by the brittleness of your app.

I feel like if you're building them brittle, you're building them wrong.

I had the pleasure of watching Sam Newman's tutorial on microservices. He's uploaded the slides here: http://t.co/2nhDBJK1oB

Building microservices requires a few changes in culture. If you are moving from a monolith to microservices, your entire culture must change how you look at problems must change and if you fail to do those things, of course your transition will change. It is not a silver bullet. It is a paradigm shift.

The more people use the phrases 'paradigm shift', 'culture change' and 'I feel like" and the less people give actual hard-nosed technical justification for this new architectural mold, the more convinced I become that it is a transient fashion that needs to die already.

> technical justification for this new architectural mold

The technical justification is easier than the cultural or managerial justification: you can make more granular technical decisions by splitting and merging code across services and servers. This is very powerful.

Microservices aren't going away—they are the core of a successful, scalable distributed system. The hype surrounding the term is definitely loaded with cultural and managerial implications that are rarely addressed, however, and I hope that changes soon.

>you can make more granular technical decisions by splitting and merging code across services

"granular technical decisions" -- can you please give an example of a 'granular' technical decision?

>Microservices aren't going away—they are the core of a successful, scalable distributed system.

It would be more accurate to say that microservices is what you call a system that you made distributed even though you didn't actually have to make it distributed.

> can you please give an example of a 'granular' technical decision?

Whether or not to upgrade an underlying library for a service. How many nodes to give a cluster, what features to prioritize on that node. You can do this with a monolithic codebase but you move slowwww if upgrading one unit implies upgrading others. With microservices, you could even write every service in an entirely different language.

> It would be more accurate to say that microservices is what you call a system that you made distributed even though you didn't actually have to make it distributed.

You don't have to make anything distributed. I think it's more apt to draw an analogy to the Actor paradigm—microserves are discrete actors in a distributed system and can be analyzed independently from the other actors.

All the microservices movement has really brought to the table is "no, you don't need to distribute your work to benefit from the looser coupling". If you're seeing anything more, I suspect there would be significant disagreement in the movement about it.

>Whether or not to upgrade an underlying library for a service. How many nodes to give a cluster, what features to prioritize on that node. You can do this with a monolithic codebase

Right. And it's no harder, either.

>you move slowwww if upgrading one unit implies upgrading others.

Absolutely not. You don't have to have microservices to have a set of loosely coupled libraries.

A reliable automated test suite is what is going to make library upgrades less painful for you, not 1,000 new, brittle serialization/deserialization layers.

>With microservices, you could even write every service in an entirely different language.

Really, really bad idea. Debugging a call stack is an order of magnitude easier than tracing a failure across a set of multiple services which is an order of magnitude easier than tracing a failure across a set of services written in different languages.

>You don't have to make anything distributed.

Some systems do have to be distributed for the ability to call on more resources than one machine can realistically provide, to deal with the inherent unreliability of individual servers, or (sometimes) for legal reasons. There are a great number of problems that you create for yourself if you do this, though, which is why avoiding it where possible and minimizing the pain of doing it is the architecturally sane thing to do.

"Microservices" is the 'paradigm' that you should create this headaches for yourself because Martin Fowler Told You To and because the idea that separation of concerns and loosely coupled libraries won't happen unless you do the technical equivalent of self-flagellation. The unfortunate reality is that it is just as easy to write tightly coupled code with microservices, it just creates a bigger headache for you when you do.

>All the microservices movement has really brought to the table is "no, you don't need to distribute your work to benefit from the looser coupling".

It seems to me that the microservices movement implies the direct opposite of this.

Okay then. Let's play the "hard nosed technical justification" for this new architectural mold. Specifically pointed at your issues.

> The serialization/deserialization layer adds potential for bugs.

Adding an agreed upon layer that aligns with clean architecture[0) will actually produce less bugs both upstream and downstream. (This is why contract apis are important) and are actually more valuable than monolithic systems where there is a sincere lack of data and state guarantees. (clean understanding vs black box mentality)

> The massively increased number of services all need constant monitoring.

It's the same amount of monitoring. It's just horizontally scaled and is actually easier to pinpoint issues. This also allows you the ability to change or create circuit breakers to fail over when issues targeting specific are found. (see slides 134-150 of Sam Newman's principles of microservices [1])

> The massively multiplied error conditions (what happens when service 37 talks to 39 and 39 times out?) need to be accounted for and those edge cases must be ironed out.

They should have been "ironed out" well before this point to begin with. If the application fails due to infrastructure issues, it is actually an application design failure that should have been identified and resolved previously via TDD and failure state testing. Graceful failure is always desirable, and having a legacy monolithic application does not prevent infrastructure/connectivity failures. (for reference, check out Martin Fowler's "The 12 Factor App" [2])

> You get fuck all except migraines caused by the brittleness of your app.

What it sounds like you're saying is:

"Building a monolithic app allows you to crutch bad behavior and poor design without having to understand the realities of software. You can ignore scalability, clean architecture, concurrency, data modeling and graceful failures."

I can understand how moving to a microservice could seem very painful and have a brittle result without the depth of understanding these realities. This is specifically why I noted it is a required paradigm shift for many developers.

For more inspired reading, you may want to checkout Gilt's discussions on moving to microservices.[3]

[0] http://blog.8thlight.com/uncle-bob/2012/08/13/the-clean-arch...

[1] http://www.slideshare.net/spnewman/principles-of-microservic...

[2] http://12factor.net/

[3] http://tech.gilt.com/post/102628539834/making-architecture-w...

"Building a monolithic app allows you to crutch bad behavior and poor design without having to understand the realities of software. You can ignore scalability, clean architecture, concurrency, data modeling and graceful failures."

I 100% agree with you. I have found that monoliths, no matter how militant the architect is, will eventually have bad behavior slip in. It is the path of least resistance/junior programmer where someone will eventually shortcut the interface.

In micro services there simply are fewer ways to cheat. The only access a service has to another service is through the defined interface. The service providing that interface must also now support it as is going forward - the opposite of brittle. As you mentioned above, when dealing with another micro service the programmer must think about failure all the time, which is what they should have been doing anyways!

Thinking in micro services from the start is not that hard, but untangling a monolith can be very hard. Micro services came out of people realizing that huge monoliths are impossible to change because over time all the bad practices that crept in make them impossible to reason about.

There's value in enforcing separation of interface and implementation. But tooling can do that without needing the hair-shirt of putting a network connection between them.

>Adding an agreed upon layer that aligns with clean architecture[0)

Creating new boilerplate never aligns with clean architecture. Clean architecture means solving your problem in the least amount of code possible, not the the most.

>monolithic systems where there is a sincere lack of data and state guarantees.

If anything you get fewer guarantees about your library if you expose it over a REST api instead of putting it directly on your call stack. Your library can actually go down and you have to prepare yourself for this eventuality and write a ton of code to deal with that.

>It's the same amount of monitoring.

25 services require 25 checks. 4 services require 4 checks, with all of the attendant graphs, notification systems, etc. There is simply no way to wriggle out of that one, no matter how great you think this architectural model is.

> This also allows you the ability to change or create circuit breakers to fail over when issues targeting specific are found. (see slides 134-150 of Sam Newman's principles of microservices [1])

This is a fix for a self-created problem. If I create a library to add two numbers together and create an API endpoint for it, I need a circuit breaker for it. If I instead just call it directly, I don't.

>They should have been "ironed out"

Of course they should be "ironed out" if you follow a microservices architecture! You have to do all of this self created work or your app becomes massively unreliable. The point is that you can avoid these problems altogether if you KISS and don't follow a microservices architecture.

>What it sounds like you're saying is:

>"Building a monolithic app

Firstly, there is nothing intrinsically monolithic about running two libraries on the same computer, in the same service. Loose coupling has nothing to do with whether you are running two different bits of software in the same service. It simply means that the amount of work you have to do to swap out a component is low.

I currently work on a "microservices" system and it is tightly coupled as hell and a monolith.

I've built non-microservices apps that are extremely loosely coupled.

This intentional conflation of the word 'monolith' and "non-microservices" is disingenuous as hell.

>allows you to crutch bad behavior and poor design without having to understand the realities of software.

If "not doing microservices is a crutch", strong typing is also a crutch.

As it happens, I don't need the "crutch" of creating rest API endpoints between my libraries to make them loosely coupled. I do it anyhow. If I were to work with somebody who doesn't write loosely coupled code, though, I certainly wouldn't want to magnify the explosions caused by their bad code by making it tightly coupled over a series of interconnecting, networked services.

>You can ignore scalability, clean architecture, concurrency, data modeling and graceful failures."

I prefer to make scalability, clean architecture, concurrency, data modeling and graceful failure easier to handle, not harder. I can do that by using a (where possible) stateless, non-microservices architecture.

This comment is identical to "I feel like it" except it's essentially just "I don't like micro-services." Your earlier comment did not provide any "hard-nosed technical justification" for why you don't like them, it was just "they're brittle" which the GP is arguing that if they are they're not implemented correctly.

What is the hard-nosed technical justification for either microservices being brittle by default or for them not being a realistic way to build a system?

>Your earlier comment did not provide any "hard-nosed technical justification" for why you don't like them, it was just "they're brittle"

I gave three reasons. Did you not read them?

> This is not because microservices-first is inherently flawed. This is purely a skill set issue. Microservices demand a greater skill set. A 'monolithic' approach puts much less of a demand on the engineer's skill set.

Its not purely a skillset issue. Its a domain discovery issue, which Fowler mentioned as boundary discovery. Most startups are inventing their own domain in whole or in part; as the business changes, either the "domain" changes with it, they segment and focus their efforts on part of the domain, or they invent a new set of domain concepts. Evolving an application is often difficult when everything is Just Right. When its not, and you throw into the mix artificial boundaries that have arisen from incomplete, incorrect, or evolving analysis, the job is going to become much, much harder.

You are saying that you cannot have an architecturally sound software system when you do not fully know your domain. This is a fallacy. In fact, architectural soundness makes it MUCH easier (tractable) to grow your system in the face of changing your domain.

That's not what I'm saying at all. You seem to be jumping to an assumption that all architectures are created equally at all points along the knowledge graph, or alternatively that a microservice architecture is fundamentally better than a monolithic architecture, and neither interpretation of that is true.

How is it easier to change a service interface than an API within a single app? The former requires a specification and tooling around serialization/deserialization to have any meaningful guarantee of correctness, it requires error handling to reap the benefits of decoupling, the latter can be covered with a unit test.

I'm sorry but this is pure hype to say that microservices make refactoring easier: that is true if and only if your boundaries stay static. It is a huge assumption that anyone can draw those boundaries correctly for a large microservices architecture in its infancy.

> This is pure survivor bias. The majority -- by far -- of 'failure stories' I know of in the software industry are ones in which a company tried to take their monolithic code base and modularize or microservice-ize it.

In my experience, this happens when a dev team has built a product but customers aren't flowing in. If they had customers, they wouldn't have time to fiddle with their architecture unless it was a growth-limiting problem. But the real problem ultimately is customer acquisition, not that they had to rearchitect the app.

> This is not because microservices-first is inherently flawed. This is purely a skill set issue. Microservices demand a greater skill set. A 'monolithic' approach puts much less of a demand on the engineer's skill set.

I disagree. I don't think "microservices-first" is inherently flawed, I think it's just really hard to know how you need to break apart your services in a way that seems logical to customers. It's less about good/bad architectural design skills (though I agree, that is a problem) and more about engineers being really awful at predicting demand.

But microservices are not a panacea that fixes everything on their own. You have to think about the entire deployment stack, from the hardware up. This means things like VM orchestration, network/load balancers/firewalls, configuration management, etc. that you don't have to worry about as much or as often with a monolithic application. Microservices optimize flexibility because they can be deployed in the widest variety of ways, but they do induce management overhead that is unnecessary in a lot of cases.

Besides, splitting out an app to a microservice architecture usually isn't that hard. I've architected this 3 or 4 times in the last year alone -- you use a proxy to break the traffic off to a different deployment endpoint and split the code up later. This assumes your monolithic app is stateless and using a separate persistence layer, but that's just basic software design.

> This is not because microservices-first is inherently flawed. This is purely a skill set issue. Microservices demand a greater skill set.

Actually, from the perspective of someone building them for the past nine months, the actual service is the easiest part to deal with because the loose coupling drops the requirements to service constraints. The hardest part, by far, has been coordinating common changes, tracking deployments, and running no-downtime migrations.

The hardest part of micro-services, BY FAR, is the management, communication, and coordination changes that MUST accompany a successful move from a monolithic codebase to many smaller codebases. Some of these are easier to address than others, but the barrier is NOT technical so much as it is social.

> This is pure survivor bias. The majority -- by far -- of 'failure stories' I know of in the software industry are ones in which a company tried to take their monolithic code base and modularize or microservice-ize it.

Interesting. I thought that the #1 reason why startups fail is because they run out of business, not because they couldn't modularize.

I believe the OP's talking about high-level software project architecture failure, not commercial failure.

Thank you, exactly. But the fact that a lot of companies attempt to decopule their monoliths should make it very clear how much unsound architectures hurt us. Business make these grand attempts because of the intense amount of pain their monoliths cause them. (They thought they had a software asset; when in reality it's becoem a liability.)

In my experience, they've done it because they've been sold a microservices dream that has nothing to do with commercial success.

And I've seen a lot of failure stories where the big monolith was tolerated until it became so painful that people were threatening to quit (or already had).

When that much pain is going on people try a lot of things. That doesn't mean that the thing they tried was good or bad necessarily.

All it really means is that deathbed conversions don't usually work. Don't wait until your problem is literally killing you before starting to think about how to address it.

I think where Devs tend to fall down is that we don't know how to think about a problem without immediately jumping to executing on a plan to fix it. Contingency plans are pretty cheap to create, if you know how, and you can jump on them as soon as the problem proves itself.

If such a rare skillset is required, doesn't that kind of imply that the risk of failure is greater when choosing Microservices?

Yes.

> This is purely a skill set issue. Microservices demand a greater skill set.

Right and sometimes the simpler solution is the correct one.

How is a monolithic architecture the same as programming against a global mutable state?

Programming with a global mutable state seems to have more to do with the programming paradigm and little to do with the architecture.

> there's a definitive architectural skill set that CAN BE LEARNED

Would you care to share what that skill set is, for the sake of improvement of a mid-level engineer like myself?

One of my pet peeves is that developers rarely understand the costs of traversing boundaries well enough to even think about them.

You won't do distributed services well if you don't understand that each step removed from in-process, in-cache carries with it a severe cost-multiplier for communicating state.

E.g. most developers (rightly) assume that function/method calls are usually quite cheap, but we don't tend to be good at thinking about cache effects. So we lose the first big factor of performance by e.g. chasing pointers around.

Then people tend to (wrongly) assume that if function calls are cheap, this extends to system calls. Another multiplier, as people e.g. does tons of small read()'s instead of larger reads into in-process user-space buffers.

But I see people go further, and assume that database queries or http requests are cheap, often helped by having them hidden behind ORMs and various APIs that obscures how often they're making these queries even if they're logically aware they're calling them.

The end result is that often these applications get written with the assumption that a function call is a function call is a function call, without thought to whether or not that function call is just a handful of machine instructions, or will trash the cache, or traverses a process boundary of hundreds of cycles before anything gets executed, or executes a remote API that may take millions of cycles to return.

And then something is "slow" and the developer in question often have only a very vague idea what is likely to be slow, and stumbles around trying to measure various parts of it without thinking about the bigger picture of what information traverses what boundaries how frequently.

The same concern applies to reliability tradeoffs as well as performance (e.g. lack of understanding of implications for high-availability for example).

I think a huge proportion of problems making effective use of distributed systems boils down to this. There are hard challenges, sure, but if you are aware that each step across a boundary carries a certain penalty, you're already well on the way to actually thinking about why you want to move something across a boundary, and think about whether or not you understand the tradeoff well enough, and to structure things to e.g. minimize the number of boundary traversals (once you think of system calls as "slow" you are much more likely to entertain the idea of the slight extra complexity of ensuring you read/write as much as possible in one go, for example).

Even just teaching younger developers to be aware of this, and how to actually measure and profile more, would help immensely, I think (one of my favourite things to show people is just very basic crude use of "strace"; suddenly it becomes very obvious to a lot of people that some of the things they do are immensely inefficient). Not because it would teach them to do distributed systems right, but because it would give them the tools to recognise much earlier when they're headed into territory they don't know well enough yet, instead of finding out after they've already painted themselves into a corner.