Fulfilling the Promise of CI/CD
stackoverflow.blog
stackoverflow.blog
We absolutely do this and I don’t think we are THAT unusual.
Also, as you hit larger scale it’s questionable whether this is still a good idea. We are about 50 engineers and our CD means we release to production about 10 times a day, on average. At that pace, some tricky things start to happen: a metric starts to move sideways and it’s hard to actually correlate it to a release - you may have to roll back across multiple releases, which can create a fairly confusing situation for the team.
It's especially helpful for getting the data and forming the arguments you need to persuade leadership that CI/CD is a good thing.
[0] https://smile.amazon.com/Accelerate-Software-Performing-Tech...
Sure, our deployment process was 10-15 minutes in the best of cases (because of serializing deployments for different branches), but where does the author's impression that the collective "we" are not doing CD come from?
In an enterprise setting, due to segregation of duties, it's often true that CD is impossible due to restrictive change management processes that result in unpredictable and typically week-long cycles.
That's not automated. CD (however you (re)define it) is about automation and removing people from the loop.
As long as you have manual processes, it's not continuous.
IMO Continuous Delivery is kind of a nothingburger vague process term that lots of people can (and do) talk themselves into saying/thinking they are doing. That's kind of the theme of this blog post BTW. Continuous Deployment is the actual indisputable thing (you either automatically deploy new commits in mainline or you don't) that gives you the real benefits but which lots of orgs aren't yet doing.
Even their process is very much like true CI/CD but packages only get into "proposed" archive (still accessible to the public) without further human vetting (I mean, a code review is human vetting too).
I personally want my desktop/server apps to automatically update according to traditional LTS rules (no breaking changes, just bugfixes).
Agreed. I'm all-in for Continuous Deployment. Continuous Delivery is a half-measure at best.
Don’t do CD as a tech/ego benchmark thing, however (or Facebook/Google envy). Do it if you can agree that this brings some benefits to the business — eg reduced risk, higher throughput, etc. Also delivering faster really forces you to examine your process for bottlenecks and strip away overhead. I really like this line from the blog:
> The teams who have achieved CI/CD have not done so because they are better engineers than the rest of us. I promise you. They are teams that pay more attention to process than the rest of us.
This is a really important point. Rapid releases are not always inherently better. I used to work in a company that dealt with heavy regulatory bureaucracy (in the sense that putting out wrong/misleading information could result in class action lawsuits), and because of that, workflows were highly optimized for waterfall delivery. This meant aggressive reliance on classical project management tools and techniques, and hyper-specialization (as opposed to the sort of full stack engineer approach taken by many bay area tech companies) - and this has worked well for that company for decades.
You can also do hybrid approaches, e.g. continuous deployment to a QA environment, and have dedicated QA teams, while delivering to client on a waterfall schedule.
I feel like this approach lets developers still spot check and make sure their code looks good when in QA while still giving the more anxious folks who need production to be holy to have their manual button press to get things live. Ideally, your pipeline is also at that point a simple button press to go from QA out to production. It also hopefully isolates any production issues to environmental differences as you will be in the "bundled changes" territory referenced in the article.
From what I've experienced the FUD around letting things get into production rapidly really freaks out some organizations and having it hybridized is better than nothing.
TL;DR ... everyone has Jenkins but they can't just let it deploy stuff because there is tons of "professionalism theatre" bureaucracy baked into the SDLC implemented as crappy homegrown scripts that check you filled in fields in JIRA. On top of that the org likely has incorrect thinking that the way to reduce bad outcomes from software is to change it less often.
This. So much this. The way you turn a bug into a "bad outcome" is to make it slower to change. If you can deploy quickly and easily, you can fix any software issue before it becomes a big problem.
I agree that we’ve been lucky though, to have great tech leadership on this. And also a very supportive business.
The regulators have not had any problem with CD or high cadence at all — because we can also show that our systems actually got more stable as we adopted CD “in spirit” and increased release cadence. Also, because we no longer do manual promotions, we’ve improved our Cyber-sec position by a lot — just one of the many advantages of automation.
The most common seems to be auto-deployed to staging or a dev environment, with some sort of daily or weekly process for promoting to production. One company built a Slack-based approval process using +1 or -1 reactions, and another has a zoom meeting where every author has to attend and is walked through a checklist before the release is approved.
My team also had a manual approval step to production, which theoretically meant the dev would check logs, dashboards, and alerts before approving, but in practice both with us and teams I talked to, that is followed about 50%-80% of the time.
What we built into our product, Sleuth [1], is a way to automatically promote staging releases when the staging release was determined to be healthy and soaked for a minimum amount of time. This allows the 80% case to simply flow through to prod without developer babysitting, whereas we can easily interrupt the process with a -1 reaction in Slack if it needs more manual testing. I think this is the ideal - the common case is the code flows but you still have an easy way to interrupt the process when the change needs it.
Yes, there is value on CI, and there is plenty of value on very low effort deployment. But, except from a large downwards risk, what does automatic deployment bring in exactly?
Automatic deployment takes it to the next level where it becomes so easy to deploy that the dev starts fixing things they would have previously ignored because it was too painful, annoying, or time consuming to deploy before. A typo here, a refactoring there, and now they can fix, push, and go back to what they were doing within minutes, but do so with the confidence they won't break things. It is kinda like the difference from when CI takes 3 minutes to 10 minutes. It isn't much but that longer time means you alt-tab over to reddit or whatever, forget what you are doing, and now that task takes orders of magnitude more of wall clock time.
What benefit does have a computer deciding that a version of the software is ready to deploy instead of the developer pressing a button when he thinks the software is ready to deploy?
By making it automated, the dev can immediately start on a new task and the deployment can go out quicker, meaning the next dev can start on their deployment. Therefore, I see it as helping preventing costly context switching and therefore, improving efficiency.
Yet, it goes completely against my experience, so you've left me wondering what creates that difference and suspect it comes from organizational differences.
You certainly considered no post-deploy activity. That should have been obvious from the beginning, but I just didn't think of it. I suspect the post-deploy activities I'm used to go to the role of a product manager, and you are assuming a more jira-oriented environment than I was.
You are probably assuming larger software blocks than I am. I should have imagined that too, because I have a tendency of breaking software in many more independent pieces than it's usual, and of imposing that on my environment everywhere I go. I am used to people having something to do on a different project than the one they started a CI, so they can start right away, and come back after the CI is complete, so there is no wait. And there is no "next dev" waiting, because he is working on a different project too.
Also, having a different project to work on during CI means a major context switch if people need to fix anything on the previous change they made.
So, with that said, I can now imagine what kind of environment CD makes sense. It's somewhere with a small number of projects and a large team of developers (or, any place with a high developers/projects ratio). It's the kind of place where would be hard to coordinate a release if it was manual too.
CD is avoided because most software products should not be delivered continuously, as it is inversely correlated to software stability for customers. It may create value internally, but if it doesn't create value externally and is cheap to roll out then it's probably not worth it.
Slightly related, I think CI/CD is today where VCS was before git. It all kinda sucks, and no startup or company is going to fix it (nor can they, it must be free as in beer and speech). We need something that sucks less and becomes as standard a tool as git or make.
IMO, the idea of CD only works for companies that control the production environment and basically offer a service, not a product. If that's you, you may get a great deal out of CD.
But if you have a product and if that product manages your customer's data, is integrated with your customer's scripts, and runs on an environment that you don't completely control, you better think twice before permanently updating the system. You can still do it, but it will be expensive to get it right.
This has been shown to be flat out wrong. Stability has been shown to be highly correlated to the capacity to deploy quickly and often.
crickets
One (QA) can always dream!
However, I'm sure some people would argue that some of these teams are not CD enough due to items like the following:
- One team had very long (4+ hour) test suites. End result, devs hit merge in the morning to see their changes that day, and didn't if a test failed (and of course there were flaky tests). Eventually there was a project to pare down unneeded tests and address flakiness, but by the time I left it was still a 2 hour (though more reliable) time from merge to deploy. No human intervention needed to.
- Release windows. The release pipeline on one team only ran 9-5 monday to thursday. If you merged after 5 it kicked off the following morning. If you merged friday, it kicked off monday morning. I'd accept a claim that the team was only doing CD 4 days of the week, but not doing CD at all is overstating it.
- Code reviews as requirements of merge. I still think this doesn't disqualify it, because the developer still has the option of when to hit merge after review and go to production without anyone else's intervention. On 2/3 teams the CI pipeline ran on branches also was sufficiently reliable that if your branch build passed, you were pretty sure the release build also would.
I think, especially in the cloud, I'd would fear updating a database automatically and lose data. Some of the resources in AWS/CloudFormation are "replaced" and not "updated", which gives me a bit of paranoia.
On the AWS/CloudFormation side, "Cattle, not pets" is the motto[0]. Terraform specifically is really good at capturing the desired vs current state of your infra, and showing you what those changes will be before you apply them. Point being - as long as your automation is captured in Infra-as-code, you shouldn't care too much if something gets destroyed.
[0]- http://cloudscaling.com/blog/cloud-computing/the-history-of-...
The tradeoff is that if you're actually relying on the structure of that data, you're introducing risk since its changes aren't being intentionally managed at the DB level.
While the latter may sound easier (and often is easier when you're starting out and iterating quickly), you will soon start to feel the pain of managing dozens of different versions of your "schema" at the app layer. I've always found it easier to only have to deal with a single schema at a time in any situation where you can't just regenerate all records on the fly, like a cache, and for which you don't care about what the specific schema was at a point in time.
If you actually do care about what the schema was at any point in time at the app layer, your app WILL need to account for any version of the data—but if you need this you're accepting the tradeoffs of the additional complexity.
Definitely more nuanced than "no migrations = easier" though.
Beyond that however it’s about taking a look at your processes and your particular challenges.
Say if you have a old-school DB with lots of stored procs, no or not enough tests, and that people tremble to update, that DB is a business risk.
You should have backups and contingency plans (eg a hot standby?) to ensure updates are resilient even if something fails. And of course the “good hygiene” of adding columns, using feature flags etc as the other commenter wrote.
Once you’re less worried about the database, you can start refactoring it if you wish.
Refactoring a large, entrenched database is a large topic but re publicly available case studies, have a look at Netflix’s case study on how they migrated their billing system away from Oracle[1].
[1] http://techblog.netflix.com/2016/06/netflix-billing-migratio...
Currently, I'm mostly interested in greenfield projects and how to do things the right way from start.
At Release[0] we discussed this in our Build vs Buy[1] page, specifically in the section "Is building a PaaS your core competency?" Companies should be focused on building their product and delivering the value of that product to their customers, not spending years building out a platform to achieve CD.
My hope is that people's plan for achieving CI/CD in 2021 includes looking at all the companies working in this space and give them a chance rather than trying to spin something up on their own.
* If it wasn't clear, I work at Release, so my I'm letting my bias be known *
[0] https://releaseapp.io [1] https://releaseapp.io/build-vs-buy
Software not in the customer's hands hasn't been deployed OR delivered.
Alternatively: Software undeployed is software undelivered.
ITIL’s latest iteration, ITIL v4, has a track for “high velocity IT” which incorporates rapid release cadence, CD, etc and is described by them as suitable for “digital” organisations or organisations going through digital transformation.
I laughed a bit at this because this is just ITIL playing catch-up, but still, it’s a useful data point to get some ‘stuck in the past’ people to see that the “old Enterprise IT ways” are no longer un-challengable.
As a result I looked into building a CI/CD pipeline for the first time and practically every article I came across talking about “CI/CD” really just talked about CI.
Even today I have no idea how to easily automate deployments. The only service I know of that does this is Heroku. Ideally I should be able to push any changes to a master branch on Github and have those changes automatically deployed to my server. How this can be done is poorly documented from my experience and certainly nowhere near discussed as much as CI solutions.
EDIT: I subscribe what jasonpeacock is commenting above, I think it's better expressed in those words.
Some platforms provide this as part of their feature set (like AWS).
You talk about your side-projects - how do you deploy them today? Write a script for that and think about how many unique-to-yourself edge cases you are handling. If you believe your solution is generic and re-usable, then build that into a tool/platform for everyone and profit!
Most tools are Kubernetes-centric (i.e. FluxCD[0] or Jenkins X[1]), but there are some simpler Docker-only options as well such as Dokku[2].
[0]: https://fluxcd.io/
Right now, it is triggered by a push to "master" branch, the pipeline builds the app, runs tests, creates artifacts(files to be used for release/deployment). Then, the release pipeline is triggered, this builds a "release", and deploys the application to "staging" environment used for user testing. Once, the application is approved, I have to go to Azure DevOps and confirm deployment to "Production". There are many different approaches, triggers and settings.
Check out Azure DevOps they have a lot of good information on CI/CD pipelines.
You don't mention what technology you are using, but maybe you didn't find the correct resources?
https://codefresh.io/docs/docs/yaml-examples/examples/#deplo...
F.D. I work for Codefresh
I've never seen a team that operates with every developer committing every day. Small commits merged into a stable trunk as often as possible, yes. But every developer merging code every single day is unrealistic, counter-productive, and incompatible with code review practices.
That's not a hard requirement, it's more like a principle.
In my experience, it's easier to review smaller things than gigantic PRs. I don't take this line item to mean literally committing every chronological day, but in the sense of not withdrawing from the world and building out entire systems in a cave, so to speak. It's a more granular version of "prefer agile over waterfall".
Another subtle aspect of the commit-often mantra - particularly when it comes to a workflow with tests running in CI - is that you are encouraged to build software in a bottom up fashion (simply because if you try to "tack on" an incomplete feature to an existing live system, it'll obviously not work).
My impression is that they claim that "true" CD means each deployment is of a single author's changes, and that batching defeats the spirit/purpose of CD.
But many orgs commit to master faster than deploys can go out (eg 15 minutes) especially if there is caution taken to ensure the change did not cause problems (automatically or manually detected).
I only skimmed parts; was there a solution to this problem mentioned? Or did I misunderstand?
Our problem is that we need to manually cut a release branch with the right version number manually then go through logistical steps before we can merge.
Docker alone takes about 30 minutes to build.
Also a pull based approach allows you to integrate multiple repos. I have never seen anyone from the CD camp talk about how a company should manage CD with n interdependent git repositories with the usual push based approach.
Generally you would only need the manual approvals for prod. Dev, qa, staging, etc., can typically still be released completely automatically, so you just create a CI/CD infrastructure that can be used for both.
The tendency to bend words until they lose their meaning just to tick them off is a bit strange. No one can possibly use every trendy idea simultaneously. No trend is a great fit for every situation.