Testing with Jenkins, Ansible and Docker
blog.mist.io
blog.mist.io
Instead of doing:
RUN echo bar >> foo
RUN echo baz >> foo
RUN echo bar >> foo \
echo baz >> fooAgreed, I thought the same, but then I hit the 127 (I think) limit.
Genuine question, not intended as any sort of troll: what benefits would people with this philosophy say their organisation gains from routinely deploying multiple times per day?
I have nothing against better testing tools or more efficient development processes, of course, and if you have a serious bug then being able to fix it as quickly as possible is obviously beneficial. I just don't understand where this recent emphasis on always trying to move fast has come from, or what kind of management strategy someone might use to take advantage of such agility.
Over what? Are you seeking a comparison with releasing one a day, a week, a month, a year?
One of the responses to your question contrasts multiple-times-daily releases with once-every-two-months releases. There are certainly clear benefits there. Personally, i am less convinced of the net benefits of multiple-times-daily releases over a once-a-day or even once-a-week releases.
I work on a few apps which have fast release cycles, where we often release several times a week, and occasionally release several times a day.
One big benefit is the ability to fix bugs fast. When a customer service rep comes over and tells us about a mistake in the data we are showing our users, or a sysadmin comes up and tells us about some alerts we're causing on his dashboard, we can quite often fix the bug and release the fix the same day. That really helps keep customers and sysadmins happy.
Another benefit is the ability to get metrics into production fast. If we're making plans, sometimes we realise that we really need more information about the behaviour of the system or users to make a good decision. If that's information that can be captured in metrics or log events, we can add the metric release, and then gather information quickly enough not to completely derail our planning process.
The ability to add metrics fast also helps with tracking down bugs, because we can essentially instrument the production system on demand. For example, at one point, we had a suspicion that some service that took two overlapping date ranges as parameters was being called with date ranges that did not overlap. So we added a check for that which logged an event if they didn't, and pushed it to production. The next morning, we knew exactly how much this was happening, and which other application the was making the calls.
The flip side of this is that our frequently-released apps get essentially no meaningful manual QA, and precious little automatic testing. We have huge suites of tests which we run in CI, of course, but we don't have time to put every release through soak or performance tests.
If you wanted to make a general handwavy statement, i'd say that having really fast release cycles helps break down the wall between development and production. Which you might or might not find was a good thing.
If the code for a new feature is ready and all the tests pass, being able to deploy with the click of a button allows you to close the ticket right away and move on to the next task in line. Also, there's no need to coordinate with other developers that are about to finish their thing.
Overall, it's good for productivity and peace of mind, as streamlined deploys are less error prone.
It's specifically the motivation for continuous deployment that I'm curious about here. Fixing bugs quickly is obviously a plus, and I found the point 'twic mentioned about rapidly instrumenting a production system interesting. These are good arguments for being able to release an update on demand, and indeed for continuous integration as a technique to support that goal.
However, to me neither seems like a strong argument for routinely releasing multiple updates per day. I've always been a little surprised that so many web sites and SaaS applications seem to take pride in making such frequent changes, even though doing so surely incurs some risk of breaking things and typically also causes some degree of instability in the UI. I was just wondering whether anyone had experiences and/or actual data they could share about whether the practice made a measurable improvement to, for example, revenues or reported customer satisfaction -- something concrete that might outweigh the risks -- or whether it's more of a subjective preference for that way of working.
I don't understand this line of thinking. If there's a bug that you can fix in short order, why not fix it and deploy said fix? If you've set your CI and infrastructure up correctly, update rollouts aren't something the user even notices.
I've always been a big proponent of small, atomic commits. Being able to deploy just one thing at a time means we often know where breakage comes from (when it happens), and we can respond to feedback sometimes within minutes.
The customers love it, and we enjoy excellent stability and development velocity. These are the reasons we deploy many times a day. There's no reason for us not to deploy things once they are tested and ready.
If you've got an important bug to fix, sure, of course you want it dealt with as quickly as possible.
If you've set your CI and infrastructure up correctly, update rollouts aren't something the user even notices.
Not if it's for a bug fix or security patch, but presumably most of your development work is adding new features? In that case, isn't the whole point that it's something the user will notice?
I've always been a big proponent of small, atomic commits.
And again, no argument from me there. While I don't think every development project is suitable for that approach, I favour it most of the time myself.
Being able to deploy just one thing at a time means we often know where breakage comes from (when it happens)
This is where my experience and background seem to be very different to the deploy-hourly kind of philosophy. I've noticed that proponents of the latter often seem to assume that stuff is going to be breaking all the time, and therefore that being able to deploy bug fixes or diagnostics very quickly will be a significant advantage.
However, if you have so many significant bugs that you need to deploy multiple times per day just to keep up, to me that seems like a clear demonstration of insufficient QC/QA in the development process. And so I can't help wondering whether the pressure to release so often, and the consequent almost exclusive reliance on some sort of automated test suite for quality checking, is a causal factor in having so many bugs in the first place.
The customers love it, and we enjoy excellent stability and development velocity. These are the reasons we deploy many times a day. There's no reason for us not to deploy things once they are tested and ready.
I guess this is the paradoxical part that I don't understand. If your project enjoys excellent stability and your deployments are all soundly tested before they go live, then by definition you don't need the rapid deployments just to rush out bug fixes or diagnostics.
So that brings me back to where I came in: do you experience any concrete, measurable benefits from the "excellent development velocity" you mentioned, or is it simply your team's preferred development style?
Yes, see the following:
> The customers love it, and we enjoy excellent stability and development velocity. These are the reasons we deploy many times a day. There's no reason for us not to deploy things once they are tested and ready.
We're not just deploying bug fixes quickly (which you seem to be fixating on a bit), we are constantly making small iterations. If we're handling support and see something that could be improved, we improve it and roll it out quickly. If we have a moderate sized feature branch, it is merged, tested, and rolled out immediately as soon as it passes QA. No need to set a rigid "We only deploy on Wednesdays" schedule. Pass QA, deploy immediately.
> Not if it's for a bug fix or security patch, but presumably most of your development work is adding new features? In that case, isn't the whole point that it's something the user will notice?
No, a huge chunk of dev time is spent iterating on and improving existing features. There gets to be a point in a product's life cycle where you are more or less feature complete, where the emphasis shifts to refining and improving what you already have. Sure, there are new features, but if you never iterate, you've got older stuff accumulating dust and ageing badly.
But of course there's a limit; you don't want releases that are so quick that you don't collect enough data about the effects of each release.
That said, the biggest downside I've seen with Ansible is reusable components. They have something called Galaxy in beta [1], which should help, although it feels a bit rough yet...
It's nice being able to put together a plethora of servers within seconds, and start deploying them to DO, AWS, or a local box for playing around within a few minutes! See, for example: https://github.com/geerlingguy/ansible-vagrant-examples
This was a major, major issue with puppet, too - although somewhat worse, in that the manifests were more unreadable.
One issue that ansible still hasn't really solved is the horrible if/then/elses you have to put in to accommodate RHEL/ubuntu/centos/whatever else differences.
I like that galaxy at least lets you specify which environment the playbook will run on, though.
[1] http://www.appneta.com/blog/automated-testing-with-docker/