CI/CD Is Not a Progression
hackernoon.com
hackernoon.com
I see this line of reasoning a lot in the industry and I think it misses a key piece: monitoring/dev-ops.
Unlike the author, I think there is a clear progression from CI to CD. Where most teams stumble, is they are not prepared for the shift towards deploying new code regularly: measuring outcomes in production, running experiments, and validating assumptions against real traffic.
Developers tend to think their job is done once their PR merges, and therefore teams struggle to reap the benefits of continuous deployment.
I supposed you could just deploy stuff to prod when the build finishes without being in a state where you can safely deploy. Well now that I think of it I have seen places that do that, but it never turns out well.
It's similar to how continuous integration means your master is always stable (all builds work and all tests pass.)
If you can't do a 1% (or 10%) canaried deploy, you're not yet ready for continuous deployment.
The most important are reducing the blast radius, early detection, and reducing the mean time to repair.
In GitLab we offer incremental rollouts, auto monitoring, and soon auto rollback to help with these.
If you deploy each change you can't afford to handhold it.
1. What does bulletproof mean anyway? Total coverage? Well tested tests?
2. "Good enough" is deliberately vague and obviously will depend on whether you're writing a game or a HFT application.
I have still to understand how 100% coverage ensures that the product looks good, isn't confusing, does not contain grammar errors and actually implement what the customer wanted in the first place.
Maybe this works well in a company where everyone is great UI artists and who sits with the customers from day 1 until deploy.
Where I work we often think we have a good idea om what we want to implement but after seeing it run we realize we need to tweak some things to make it really good.
If you do deploy often, the code deltas between two deployments are smaller, and so it's often much easier to pin down the source of bugs.
This is the real win for continuous deployment. Smaller and more frequent release deltas are the real source of systemic improvement; CD is just a methodology for facilitating this.
There's many ways to do this, of course, but something I've found works well is just making it frictionless to deploy feature branches to a "dev" environment. You can test changes (especially those that integrate with other systems/services) but still keep everything isolated.
This is also great because when applicable you can take screenshots/gifs of the new feature actually running (not just on your machine) for the pull request.
Want to change the layout of a page? Normally this would require at least a little coordination with, say, customer support or marketing teams. We don’t want to hang around waiting for schedules to line up, so it’s often easier to simply deploy the feature behind a flag - it means it can, for example, be rolled out to some customer cohorts before others (beta testers?), easily rolled back, made live by the product team rather than engineering - that sort of thing.
Though of course a dev environment is _also_ incredibly important!
All my work lately has been done in branches, which then merge to master, and master gets deployed to production as needed: sometimes immediately with a single bug/feature, sometimes several bundled together after a week or two. Depends on severity, other work occupying the team (especially QA), and which software/system(s) are affected (eg. core software, infrastructure, or internal tooling). I am pretty sure we've only used feature flags in the core software, and only a couple times. Of course, that software is hybrid cloud/on premise, and as such we have strong version numbering for every release, so features/changes naturally tie to the customer-facing release numbers.
The amount of tests you need to automatically release "bulletproof" code is staggering. In one successful example, I noticed there were about 10k tests that took about 30 minutes to run (before the code could be checked in). I would think you need to use TDD to feel confident that your tests are good enough.
The other aspect of CI/CD that bugs me is the dramatic increase in complexity. I thought KISS and Occam's Razor would kill the CI/CD efforts. Moving to weekly release seems a lot simpler and I doubt most teams need to release changes much faster than that. With CI/CD, if you have several components and lots of shared code then managing which component to update given dozens of changes affecting different components is a nightmare. Unfortunately, once you start using CI/CD, there is no going back (often for political reasons).
I’m honestly really surprised to hear that view. If anything, I’ve found it much simpler. For some insight as to how my current team works:
- feature development happens on git branches - all pushes to github are automatically tested on CI, built, and produce a deployable artifact - any artifact can be deployed to a fully-featured staging environment - Builds from the master branch are automatically deployed to production - the product owner is responsible for acceptance testing the feature in the staging environment - a PR into the master branch is raised for code review - when this branch is never, after product owner approval, the feature is live.
It’s delightfully simple and entails no release schedule. Large features can be placed behind feature flags and allow other teams to activate feature for user groups as required.
Honestly I can’t imagine going back to releases ever again.
Occam's razor https://simple.wikipedia.org/wiki/Occam%27s_razor Suppose there exist two explanations for an occurrence. In this case the simpler one is usually better. (I was trying to say "If there are two ways to release a product, the simpler way should be better".)
With a reasonably well-defined, decoupled system, you should be free to run a targeted test suite without much concern for affecting consumers (assuming you're not breaking contracts).
Moving to CD(eployment) significantly reduced release complexity and increased quality of several of the system I owned in the past (which handled thousands of TPS and were affected millions of customers and dollars, globally, per day).
In fact, I preferred starting with a CD pipeline to kick off new services from the start so from the very first check-ins (even before any tests were written), the deployment automation was in place and all changes were going to the latest stage we felt comfortable with at the time (often production, since initially there would be no consumers). That sets an expectation from the start that when developers peer review and merge, their code is going to production.
I don't think it's about integrated patches into one piece of software, but rather about integrating the patched software into the entire software system in which it lives. So if you have three pieces of software (A, B & C) which are deployed together, they'll all live in the same repo (because multiple repos are almost always a mistake); each will have its own unit tests (because each piece of software is a unit); and then you'll have integration tests which test how well they play with one another, particularly when other folks are changing different parts of A, B & C at the same time.
Fowler's article[0] reads like it could go both ways. I guess if you see any system of multiple processes as a single piece of software then it's much the same thing.
[0] https://martinfowler.com/articles/continuousIntegration.html
1. We always commit to the master branch where automated unit tests use to run. We threw out all of our unit tests because of some design mistakes that I made as the first time Dev lead. We were too dependent on brittle Mock based tests when we should have used a more functional (no dependency/ no side effect business classes), based approached we started using.
2. We have an Integration environment setup where everything is deployed to whose purpose was to do automated integration tests but haven't gotten around to it yet.
3. Since every push goes to Dev but we don't want every push to go to QA. The devs have to do a post deployment approval before it gets to the QA queue.
4. Qa has to do a predeploymebt approval when they are ready to test a build. After they test, they do a post deployment approval.
5. Since we have a mandate thT each developer is responsible for getting their own code to production. They have to work with the product owner and coordinate when it should be released to UAT. Then the developer does a preapprovsl for it to go to UAT.
6. Once it is approved by the product owner and we get change control approved. I as the dev lead and my manager both have to do a post approval.
7. Then the developer does a preapprovsl and it goes to production.
Yes it sounds convoluted but people above my pay grade make those decisions (once the avalanche has started the pebbles don't get to vote). But, appprovals are done on a simple Visual Studio Team Services Web page accessible from anywhere and once done, deployments are automatic.
We also don't explicitly version code. We tag each built executable with the Git commit hash and that hash is logged as a property with each log statement. We use Serilog for structured logging.
We can rollback just by redeploying a previous release. If we need to make a change to a previous "version". We just branch based on the logged git hash.
Can you elaborate a bit further? I'm trying to make my design/test suite more robust at the moment and it would be good to hear from others who have been through the same process.
Of course all of the parts of the main process that had external dependencies like api calls and database calls were injected. While other people were writing the code, I was constantly changing the dependency chain on the core routines they were dependent on, I was changing the wheels on a moving train.
This wasn't an issue with the actual program because we were using a DI Framework, all of the dependencies were being wired up automatically at the composition root (http://blog.ploeh.dk/2011/07/28/CompositionRoot/).
But Mocking out everything was painful and if the dependencies changed anywhere within the framework, the tests broke - they wouldn't compile. On top of that, while you can add optional arguments to a method and it won't break preexisting code, it will break your Mock setup (long story).
Just like it is a well known anti pattern to use DI as a service locator (http://blog.ploeh.dk/2010/02/03/ServiceLocatorisanAnti-Patte...), so is the overuse of Mocks.
Moving to a more functional model means we have a service class where you still inject your dependencies but our classes that have business logic are completely functional - they take in data and return data. The service class is responsible for the orchestration of getting the data, processing the data, and sending the data.
The mapping classes also expose events that the service class can subscribe to if it needs to do something during the processing of each transaction like logging.
Now the only classes we need to unit test, have no dependencies and no mocking. (https://medium.com/javascript-scene/mocking-is-a-code-smell-...)
But we also found that our biggest issues aren't having regressions around our individual programs, but around integrations - thus the need to build up our automated integration testing environment.
My own model for software is tooling that enjoys infrequent highly stable releases, priorities on backward compatibility and iron tight accurate documentation. Where exactly is rapid incremental code delivery at such a premium? Why are these articles not contextualized to domains?
I'm in the same boat though, I don't get the point of CD for most applications other than to prove that you can do it (but other than impressing other engineers with your teams apparent discipline: who cares?)
Ability to get user or system feedback on code today, rather than in three months, is often quite helpful. That is, if your company depends on listening to customers or studying customer behavior. (I know of no company that isn't, though, except monopolies like Comcast.)
Sure... which is why your QA department should test things early. (You're not making your users into QA... are you?)
> Ability to get user or system feedback on code today, rather than in three months, is often quite helpful.
Users-as-beta-testers can work in some contexts, but there's a lot of of fields where doing that is a very bad thing. Experimenting on your users is, at best, a pretty nuanced business decision. And: again you can get feedback internally, or by recruiting testers, or by having a QA department. I do game development. We do play tests when we're trying to figure out if something is a good idea and we have QA to track the bugs. CI is very useful for coordinating with QA on things, but god help us if we started shipping everything checked in. Not every feature is a winner.
Not saying CD can't be useful in some contexts, but I remain unconvinced it's anything but a niche need. CI is pretty universally useful, but CD? Really depends on what you're shipping and who you're shipping it to.
That's going from one extreme to the other.
Just because you don't release every build that passes automated testing doesn't mean you wait six months between prod deployments.
In all fast-changing environments, as well as in early-stage projects that don't a very firm direction yet.