Slow deployment causes meetings (2015)

I tend to agree. Whenever I've removed artificial technical friction, or made a fundamental change to an approach, the processes that grew around them tend to evaporate, and not be replaced. I think many of these processes are a rational albeit non-technical response to making the best of a bad situation in the absence of a more fundamental solution.

But that doesn't mean they are entirely harmless. I've come across some scenarios where the people driving decisions continued to reach for human processes as the solution rather than a workaround, for both new projects and projects designated specifically to remove existing inefficiencies. They either lacked the technical imagination, or were too stuck in the existing framing of the problem, and this is where people who do have that imagination need to speak up and point out that human processes need to be minimised with technical changes where possible. Not all human processes can be obviated through technical changes, but we don't want to spread ourselves thin on unnecessary ones.

> The reason those meetings exist is because of risk! The more changes in a deployment, the higher the risk that one of them is going to introduce a bug or operational issue.

Having worked on projects that were perfectly full CD and also projects that had biweekly releases with meetings with release engineers, I can state with full confidence that risk management is correlated but an indirect and secondary factor.

The main factor is quite clearly how much time and resources an organization invests in automated testing. If an organization has the misfortune of having test engineers who lack the technical background to do automation, they risk never breaking free of these meetings.

The reason why organizations need release meetings is that they lack the infrastructure to test deployments before and after rollouts, and they lack the infrastructure to roll back changes that fail once deployed. So they make up this lack of investment by adding all these ad-hoc manual checks to compensate for lack of automated checks. If QA teams lack any technical skills, they will push for manual processes as self-preservation.

To make matters worse, there is also the propensity to pretend that having to go through these meetings is a sign of excellence and best practices, because if you're paid to mitigate a problem obviously you have absolutely no incentive to fix it. If a bug leaks into production, that's a problem introduced by the developer that wasn't caught by QAs because reasons. If the organization has automated tests, it's even hard to not catch it at the PR level.

Meetings exist not because of risk, but because organizations employ a subset of roles that require risk to justify their existence and lack skills to mitigate it. If a team organizes it's efforts to add the bare minimum checks to verify a change runs and works once deployed, and can automatically roll back if it doesn't, you do not need meetings anymore.

I am really interested in organizations capacity of soaking the changes.

I live in B2B SaaS space and as much as development goes we could release daily. But on the receiving side we get pushback. Of course there can be feature flags but then it would cause “not enabled feature backlog”.

In the end features are mostly consumed by people and people need training on the changes.

> By deploying small changes often, you get deliver value much sooner and fail smaller.

Which increases the number of changes per deployment, feeding the overhead cycle.

He is describing an emergent pattern here, not something that requires intentional culture change (like writing smaller changes). You’re not disagreeing but paraphrasing the article’s conclusion:

> or the harder way, by increasing the number of changes per deployment (better tests, better monitoring, better isolation between elements, better social relationships on the team)

I agree entirely - I use the same references, I just think it's bordering on sacrilege what you did to Mr. Goldratt. He has been writing about flow and translating the Toyota Production System principles and applying physics to business processes way before someone decided to write The Phoenix Project.

I loved the Phoenix Project don't get me wrong, but compared to The Goal it's a like a cheaply produced adaptation of a "real" book so that people in the IT industry don't get scared when they read about production lines and run away saying "but I'm a PrOgrAmmEr, and creATIVE woRK can't be OPtiMizEd like a FactOry".

So The Phoenix Project if anything is the spiritual successor to The Goal, not the other way around.

So this seems quantifiable as well - there must be a number of processes / components that a business is made up of, and those presumably are also weighted (payment processing has weight 100, HR holiday requests weight 5 etc).

I would conjecture that changing more than 2% of processes in any given period is “too much” - but one can certainly adjust that.

And I suspect that this modifies based on area (ie the payment processing code has a different team than the HR code) - so it would be sensible to rotate releases (or possibly teams) - this period this team is working on the hard stuff, but once that goes live the team is rotated back out to tackle easier stuff - either payment processing or HR

The same principle applies to attacking a trench, moving battalions forward and combined arms operations.

Now that is of course a “management” problem - but one can easily see how to automate a lot of it - and how other “sensory” inputs are useful (ie which teams have committed code to these sensitive modules recently

One last point is it makes nonsense of “sprints” in Agile/Scrum - we know you cannot sprint a whole marathon, so how do you prepare the sprints for rotation?

this isn't even a software things. Its any production process. The greater amount of work in progress items, the longer the work in progress items, the greater risk, the greater amount of work. Shrink the batch, shorten the release window window.

It infuriates me that software engineering has had to rediscover these facts when the Toyota production system was developed between 1948-1975 and knew all these things 50 years ago.

You had me at "whole org test" harness. This is a very, very interesting idea. Especially in conjunction with the concept of corporation as "slow AI" that I don't hear referenced often enough.

I don't see why you call it "literacy," though. I think Maturana & Varela's term "autopoiesis" more closely orbits the kernel, and I'll bet Stafford Beer's Autopoietic Systems would contribute to a good intellectual foundation.

At a certain point, though, I wonder if a purely software "business" doesn't just look like... SaaS?

I've called that "Organization as Code" some years back :-)

You sound very defeatist about fighting bureaucracy. If you work at an org with too much management, you can slowly push to move it in the direction you hope for or leave. If you keep ending up at places that seem impossible to change, perhaps you should ask more questions about this during the interview. I've worked at many small companies where there wasn't crazy bureaucracy because that's definitely what I preferred. I also currently work at a megacorp and yes there is difficulty, but being consistent and persuasive has lead to many things slowly heading in the right direction. Things take time. You have to realize why people have made things some way and then find convincing arguments to make things better. Sometimes places do just suck so don't stick around. But being hopeless doesn't seem helpful.

> Jenkins shouldn’t be near production

> we can’t possibly put things live without QA being involved

> we need this time to make sure the quality of the software is high enough

I've only developed software professionally since 2012, but in that time not only have I never encountered such sentiments, but (and, perhaps, because) it has always been a top priority of leadership to emphatically insist on the very opposite: day one of any initiative is Jenkins to production—often directly via trunk-based development—and quality is every developer's responsibility.

At the IC level, there was no "fighting bureaucracy," although I don't doubt leadership debated these things vigorously, from time to time, especially as external partners and stakeholders were often intimately involved.

> I would sack everyone but programmers and maybe two designers and let everyone fight it out

That works for me! But it doesn't scale. We definitely have to keep at least one product "owner" or "expert" or "manager" to enqueue stakeholder priorities and, while this can be a "hat" that devs and designers trade off, it's also a skill at which some individuals uniquely excel.

All that being said, I don't want to come across as pearl-clutching, shocked Pikachu face about this. I understand that many organizations don't operate this way. The way I've helped firms make this change is via the introduction of a single, experimental team of volunteers dedicated to these practices—one protected (but not dictated to) by a mandate from on high.

But, then again, this is California.

This is more or less Musk’s approach at Twitter - and ignoring the enormous baggage any discussion with Musk brings (if possible) - I would love to see a real academic case study on the effects of that to Twitter - there will be a lot to unpick but my bias is on your side here.

> Jenkins shouldn’t be near production

All of which sounds completely reasonable to me, in many situations.

Jenkins is the Wordpress of software development. It's gigantic state loop that runs plugins with no privilege separation. Giving your jenkins instance administrative credentials in production might very well be equivalent to giving root keys to that lone guy in Romania who authored that plugin you never audited. I can understand perfectly why that might not be desirable to everyone.

.. which neatly leads on to

> we can’t possibly put things live without QA being involved

If you deploy stuff in production that never passes QA, why do you even have QA? To fix stuff later?

If they are not empowered they will never have the chance to do a good job or have any pride in their work.

I’ve often said that it is the speed of deployment that matters. If it takes you 50 minutes to deploy, it takes you 50 minutes to fix a problem. If it takes you 50 seconds to deploy, it takes you 50 seconds to fix a problem.

Of course all kinds of things are rolled up in that speed to deploy, but almost all of them are good.

I figure it's because AWS can get away with it.

The reason by boss tends to give is that it’s made by AWS, so it cannot possibly be bad. Also, it’s free. Which is never given as anything more than a tangentially related reason, but…

This is just anecdotal but I have found anytime a network interface is involved, it can slow down the deployment. I had a case where I was deleting lambdas in a VPC, and connected to EFS, that the deployment was rather quick but it took ~20 minutes for cloudformation to cleanup and finish.

> A bit tangential but why is CloudFormation so slowww?

It's not that CloudFormation is slow. It's that the whole concept of infrastructure-as code-as-codd is slow by nature.

Each time you deploy a change to a state as a transaction, you need to assert preconditions and post-conditions at each step. If you have to roll out a set of changes that have any semblance of interdependence, you have no option other than to deploy each change as sequential steps. Each step requires many network calls to apply changes, go through auth, poll state, each one taking somewhere between 50-200ms. That quickly adds up.

If you deploy the same app on a different cloud provider with Terraform or Ansible, you get the same result. If you deploy the same changes manually you turn a few minutes into a day-long ordeal.

The biggest problem with IaC is that it is so high-level and does so much under the hood that some people have no idea what changes they are actually applying or what they are doing. Then they complain it takes so long.

Not a silver bullet; you increase api versioning overhead between services for example.

I think this was the meme before moduliths[1][2] where people conflated the operational and code change aspects of microservices. But it's just additional incidental complexity that you should resist.

IOW you can do as many deploys without microservices if you organize your monolithic app as independent modules, while keeping out the main disadvantages of the microservice (infra/cicd/etc complexity, and turning your app's function calls into a unreliable distributed system communication problem).

[1] https://www.fearofoblivion.com/build-a-modular-monolith-firs...

[2] https://ardalis.com/introducing-modular-monoliths-goldilocks...

It’s a monkey’s paw solution, now you have 15 kinda slow pipelines instead of 3 slow deployment pipelines. And you get to have the fun new problem of deployment planning and synchronizing feature deployments.

As long as every team managing the different APIs/services don’t have to be consulted for others to get access. You then get both the problems of distributed data and even more levels of complexity (more meetings than with a monolith)

You can do this with a monolith architecture as others point out. It always comes down to governance. With monoliths you risk slowing yourself down in a huge mess of SOLID, DRY and other “clean code” nonsense which means nobody can change anything without it breaking something. Not because any of the OOP principles are wrong on face value, but because they are so extremely vague that nobody ever gets them right. It’s always hilarious to watch Uncle Bob dismiss any criticism with a “they misunderstood the principles” because he’s always completely right. Maybe the principles are just bad when so many people get them wrong? Anyway, microservices don’t protect you from poor governance it just shows up as different problems. I would argue that it’s both extremely easy and common to build a bunch of micro services where nobody knows what effect a change has on others. It comes down to team management, and this is where our industry sucks the most in my experience. It’ll be better once the newer generations of “Team Topologies” enter, but it’ll be a struggle for decades to come if it’ll ever really end. Often it’s completely out of the hands of whatever digitalisation department you have because the organisation views any “IT” as a cost center and never requests things in a way that can be incorporated in any sort of SWE best practice process.

One of the reasons I like Go as a general purpose language is that it often leads to code bases which are easy to change by its simplicity by design. I’ve seen an online bank and a couple of landlord systems (sorry I can’t find the English word for asset and tenant management in a single platform) explode in growth. Largely because switching to Go has made it possible for them to actually deliver what the business needs. Mean while their competition remains stuck with unruly Java or C# code bases where they may be capable of rolling out buggy additions every half year if their organisation is lucky. Which has nothing to do with Go, Java or C# by the way, it has to do with old fashioned OOP architecture and design being way too easy to fuck up. In one shop I worked they had over a thousand C# interfaces which were never consumed by more than one class… Every single one of their tens of thousands of interfaces was in the same folder and namespace… good luck finding the one you need. You could do that with Go, or any language, but chances are you won’t do it if you’re not rolling with one of those older OOP clean code languages. Not doing it with especially C# is harder because abstraction by default is such an ingrained part of the culture around it.

Personally I have a secret affection for Python shops because they are always fast to deliver and terrible in the code. Love it!

Maybe the opposite, slow rollbacks cause escalating incidents.

In my experience, there's very little correlation. I've been on projects with 1 deployment every six weeks, and there were just as many production incidents as projects with daily deployments.

I was on a team that went from every 3 weeks to multiple times per day. The number of incidents in production dropped drastically.

But much more important than that drop, was that when things went wrong is was MUCH MUCH faster to find the problem. It was also much safer and easier to roll back, since there were so few changes that would be rolled back. No one wants to back off 3 weeks of work. That's chaos.

That is the opposite of my experience. Slow deploys mean bigger deploys mean more complexity going live mean more nervousness and more testing mean more hesitation mean more chance that something unforeseen mean errors that no one understands mean war rooms.

Yeah, and slow ones as well.

You don't need microservices to do this. It's actually easier deploying a monolith with internal dependencies than deploying microservices that depend on each other.