Continuous Delivery Sounds Great, but Will It Work Here?
queue.acm.org
queue.acm.org
This has many implications. It needs to be OK to push back and ask engineers to split large code commits into smaller units. Engineers need a strong understanding of how to sequence changes: "I need to do the ORM changes, then the API endpoint, then the controller change, and then I can do the frontend." This means that senior engineers invest time in helping junior engineers understand why this has to be done and why this particular sequence is safe. Methods need to be devised to avoid holding up future pushes - for instance, a culture of rolling back to safety at the first sign of trouble instead of debugging in production.
Anecdotally, the best developers seem to always be pushing small changes, or always have a few commits in the daily cut. Interpreting this is a bit of a chicken and egg problem - is throughput higher because the changes are smaller, or does high implementation volume just imply more changes? Or maybe it's just a free form of advertisement: "I always see that X is pushing, so they must be good"
More formally, this is true, too. At least according to Donald G. Reinertsen, the author of this book:
https://www.goodreads.com/book/show/6278270-the-principles-o...
Batches and queues are primary themes of the book and Reinertsen waxes prosaic on the extensive benefits of making batches smaller all around during product development.
I've noticed the benefits mainly with user story size and pull request size (which are generally correlated). Large batch sizes lead to longer delays which lead to cut corners which lead to more defects which lead to further delays. There are other underestimated penalties that Reinertsen discusses as well.
Of course, too small and you're incurring disproportionate overhead with each batch. The key, I suppose, is figuring out mama-bear batch sizes for your team or organization and effectively paring down to them.
"I have made this longer than usual because I have not had time to make it shorter."
Small changes mean front-loading a lot of the thought into breaking down the problem. Large stories/changesets is often a sign of a vague requirement or design up-front where the developer mostly figures out how to solve the problem while writing the code. It's not really the size that's the problem, it's the uncertainty. But the size is much easier to observe and use as a forcing function for increased planning (i.e. large stories need to be broken into smaller stories before a developer can start work.)
When you talk about the best developers being the ones that push small changes, what I believe you're really saying is that the best developers are the ones that are able to be given a large problem and break it down into its constituent parts most easily and the ones that have the discipline to do that in planning rather than while coding. Which, to me, is a lot less controversial statement.
This is a really interesting question.
I agree that better developers have a natural inclination to break down large changes into small, self-supporting chunks.
But I do worry that this approach can be doomed to stick at a local maximum.
> It is often assumed that deploying software more frequently means accepting lower levels of stability and reliability in systems
...
> While continuous deployment, the practice of continuously releasing every good build of your software, is mainly limited to cloud- or datacenter-hosted services, continuous delivery—the set of practices described here that enables continuous deployment—can be applied in any domain.
...
> In continuous delivery, automated deployments to production-like environments are performed frequently throughout the deployment pipeline,
As for the third quote, notice it says production-like, not production itself. The idea is that you test in an environment that is as close to production as possible. Nobody will die if you deploy a bad build to a real pacemaker - as long as it's not inside an actual person :)
With these constraints, even if you want to do CD, your CD will end up looking exactly like a waterfall development model. It makes no financial sense otherwise.
I work on predictive systems at a hospital, and this ends up being the case for almost every deployment. The costs of making a change and verifying it or adding features is so damn high that you need to think in 3-6 month deployment cycles about everything.
This assumes that this verification / testing has to be done by a person, and cannot or should not rather be a person producing automation that can be run repeatedly, as much as is possible.
Challenging this assumption is one of the points of Continuous Delivery.
yes, it's more expensive, slower and more painstaking when there are lives on the line. But this applies regardless of it being automated or not. And automation is more repeatable and scalable than human verification for most of the tests that you want to make.
I'm sure you can imagine the legal reasons.
A lot of the practical reasons stem from fears that some unknown unknown might trigger a failure that won't be caught by the quality metrics and upstream tests. Anyone in software knows this happens a lot more often than we'd like to admit. Given people will die if the software doesn't work and people will be sued if people die, it would be both irresponsible and risky to depend entirely on an automated test suite for the pre-deployment delivery. Then add a requirement to have a third-party audit the code for HIPAA compliance and security testing, and you can kiss your CD good bye.
Even if you want to act like you don't, you still have a milestone, a release date, the evil poison that brings agile thinking to its knees.
Our velocity increased, time-to-market dropped, operational workload fell. It was great.
Sadly, after a few years of the joy of full-CD, more conservative leadership took over our org. Now, all deployments of software must be accompanied by paperwork (digital), step-by-step plans of how you're going to deploy this software, approvals from senior managers, just general micromanagement. CD isn't possible in such environments.
These people need to create a veil of importance. They will create multi-level approval process. they will add QA team in India and customer signoff. When something happens blame is still on developers.
I find often when developers think something that is process driven like this isn't necessary it's because they're not thinking outside their own direct and immediate concerns.
I often think that management was transplanted into the foreign ground of software development and have no clue. They will then come with platinum quarterly releases were testing cost more than development.
The way I see it this is a communication feedback loop much like the one between tester and developer or product manager and developer and leaving that gap open means you're one step further away continuous delivery.
I don't think it's unreasonable for customers to be somewhat distrustful of us, their black box that spits out code.
Of course if your management is actively against CD that could be hard but generally approvals are not a big problem.
I noticed a process similar to this where bugs from production would appear to 'migrate' from tested scenarios to untested scenarios.
We'd start off with UI tests on a browser, for example, and then the bugs would be clamped down on there but would seem now to crop up in API interactions or untested browsers. Then we started accommodating the tests to handle different browsers and mocking API interactions and the bugs would start cropping up somewhere else.
What was counterintuitive was that, even though code coverage metrics were really quite low, most bugs cropped up in areas of the code with automated test coverage. "Environmental" coverage seemed to matter much, much more.
https://en.wikipedia.org/wiki/Sorcerer%27s_Apprentice_Syndro...
The middle ground I've found is to have fully automatic deploys in lower environments and automate the entire process for production. But rather than have releases flow there automatically, production is handled by pressing a single "Deploy whatever is in UAT to prod" button (i.e. the "Big Red Easy Button"). You then give access to that button to somebody else in the org (ex: a UAT team, a release team, your manager) and everybody's happy. It also conveniently gives you an excuse to never fill out production release paperwork as "It's not my job, I'm not allowed to release to production!"
People everywhere like feelings of control. And the more layers of management a company has, the more executive feelings come to predominate over on-the-ground business facts. But in practice, bureaucratic change-control processes don't actually produce better code or safer systems. In my experience, they make things worse, because large batches and slow feedback loops make bugs easy to add and hard to find.
One good next step after the middle ground you describe is continuous deployment of code, but using feature flags and A/B testing tools to control visibility and impact of features. As you say, if you give people some big, obvious controls, they feel in control. Decoupling code release with business impact lets me get on with what I care about. And it lets them learn to be bolder, more engaged in the process of delivering value to users.
Automating a deploy is relatively straightforward but doesn’t provide much value if that’s all you do. Local reproduction of build steps, metrics, historical data, building out test infrastructure, all of this is important for scaling up a team and keeping regressions low.
To do this the process needs to be self service. You can find mid level people pretty easily. Anyone who wants a promotion or become an SME should have tools to do a lot of that work themselves instead of having to extract it from the leads or follow the School of Hard Knocks.
Good automation supports this, bad automation makes this harder.
The solution, of course, is more bureaucracy!
And it worked, sort of. Everyone spends all the time writing documents asking for permission to deploy software changes rather than coding, so there's less changes overall, leading to less incidents. Never mind that velocity is a fraction of what it was or might have been.
In short, management wants to solve the problem in a way they understand rather than a way that works. And they love a way that has them micromanage everything and nit-pick when you didn't fill out the paperwork correctly.
Now, I should caveat all this- I'm a terrible pessimist sometimes and also I haven't had any caffeine today which can only make things worse.
What I used to do for demos was to automate the manual processes of ITIL style change review including diffs of the code, test plans executed, test coverage, etc. and the dependency graph of CI impacted by a change (because CMDBs are broken in 99.9% of orgs, you need tools like CloudFormation, Terraform, etc. to give the granularity necessary for a thorough review even in bureaucratic hell). All the other parts of dealing with bureaucracy is pretty much filling out other fields like a list of approvers, signed commits, org, billing codes, etc. are all a matter of trivia rather than smarts.
Your changesets need to become higher grained enough to warrant all the human eyes involved. But going backwards like that in processes is usually a sign that a place that is in decline though so that alone should be setting off warnings.
I'm also a bit disappointed at how generic a lot of the wording is. It seems like you could do a search-and-replace of "continuous delivery" with almost any other process improvement term (e.g. agile, kaizen, Six Sigma) and have people nodding right along for 90% of it. Is there a concise explanation somewhere that really makes clear how this is something more fundamental than devops-flavored kaizen? Or am I expected to read the entire "canon" before I understand it enough to evaluate it?
For example, I work in a company on a product that reads data from many products of other companies. How do we tell our customers that we certified the product with the certain version of the products that we read data from? If vendors put change into the production continuously, how can we react to those changes before the customer gets them?