What is Continuous Deployment? | Hacker News Reader

What is Continuous Deployment?

sdarchitect.wordpress.com

70 pointsSanjeevSharma22 comments

> You may only continuously deploy to an environment before ‘Prod’ – UAT/Pre-prod…, but the environments you do deploy to should be ‘production-like’, so you know with very high confidence, that the final deploy to production will work without issues, when you do really need to deploy to it.

In my experience this is a pretty dangerous way to think about it. One of the best reasons for continuous deployment is to allow you to push small changes to production. This is desirable because small changes are easier to debug when things go wrong, and are less likely to make things go wrong overall. But the core assumption behind this is "the system is really complicated and has a ton of moving parts that are virtually impossible to think about at all times." Continuous delivery makes the assumption that a preprod environment is highly likely to be similar to a prod environment, and making that true is usually extremely hard at scale. Couple that with the fact that prod will (especially for large, consumer facing sites) often expose you to many more edge cases, and the fact that a developer who has pushed to preprod may not be there to help debug when his or her code finally gets to prod and breaks things, I see continuous delivery as a pretty suboptimal alternative to continuous deployment.

Coming from a banking and healthcare background it seems pretty suboptimal to build a process around waiting for stuff to break in production.

It also seems rare to find bugs in production that cannot be replicated on developer machines so I don't think getting an exact match on the preprod environment is a huge obstacle.

The important part is being able to push very small changes without it slowing you down (you need a quick and easy deploy process).

If you have a small change, it is easily testable, and the chance of it breaking something is small. The goal isn't to discover bugs on production, it's to make it extremely easy to fix bugs on production. Continuous delivery batches up many small changes and thus makes it more difficult to figure out what exactly the issue is when you discover one. I have yet to see a process that is able to release bug free code, so knowing that bugs do get to production as a matter of life, it is highly optimal to make your process make those bugs easier to fix. Ultimately you end up with fewer bugs, when you do have bugs they have smaller impact, and when you need to fix them they are easy to track down and fast to change.

Getting an exact match on preprod is extremely difficult when you have a relatively complex data environment (sharding etc.), a relatively complex build environment (assets built differently in production than dev) etc.

I still don't believe it's that hard to create a preprod that matches production. What kind of asset generation are you talking about that can only run on the production server?

Is continuous deployment just something for teams that don't know how to make a copy of their production server?

I don't really consider continuously pushing to staging or QA true continuous deployment. This still leaves you with the big push when you deploy features.

At Wealthfront, we prefer to push out everything and releasing features amounts to pushing out a single small change of whether a feature is hidden behind a user flag or experiment.

Yeah, I think the term for this workflow is Continuous Delivery, as in Jez Humble and David Farley's book: http://martinfowler.com/books/continuousDelivery.html

Continuous Deployment and Delivery both are there to remove the fear of change. Coupled with testing this is definitely a way to 10x your engineering productivity.

We've stopped counting how many times per day we push changes at Codeship as the number is meaningless in itself. If you feel you can't push regularly you giving up before starting the fight already.

It's just a very different and way more relaxed method of software development where you can actually focus on pushing the product further without getting stopped by infrastructure all the time.

Immutable Infrastructure will be the next big iteration in getting this 10x productivity I think.

First, Continuous Deployment is really awesome.

Second, you should completely ignore Continuous Deployment until you have sane systems.

I do contract systems work for a living and have seen hundreds of live production systems. Nearly every system I see is in some kind of serious peril:

  - no backups
  - if backups, no docs on how to restore
  - no monitoring (or very little)
  - production passwords in the wild (former employees, etc)
  - no configuration management
  - no path to scale
  - no path to replace defunkt servers
  - etc... (I could go on for hours)

Often when I'm talking with a potential client, they are SUPER interested in Continuous Deployment (or whatever the hotness is at the time), but they start to yawn when I start talking about getting their core systems in order.

Your systems are the foundation of your application and your business. You wouldn't build your office building on top of an active volcano or below sea level on the coast, yet many businesses happily do this for their systems.

You know all those constant news stories about massive outages, security breaches, crippled sites, etc? I'd bet 99% of them are due to them failing at the fundamentals of sane systems. Having seen behind the curtains of so many companies, I'm honestly surprised there isn't more massive systems failure in the news.

This is a critical problem in the tech world. Does it affect you? A few of you will have amazing solid secure systems, but the vast majority of folks reading this won't.

If you're using a PaaS like Heroku or Parse, then you have most of the systems worries taken care of for you. You can breath a bit easier.

But if you are managing physical or virtual servers and aren't even using configuration management (puppet/chef/salt/ansible/etc), then your systems are probably in a very precarious situation. You might not think so, since everything just happens to be working at the moment, but you are essentially coding without using version control.

You would probably think that a developer that used email for his code version control is an idiot. You would be right. But consider that emailing code around for version control is actually better than what most companies do for their systems. Most companies set up their servers manually. They often don't even have any docs or even shell scripts.

At one client, it took them 2 weeks (!!!) to bring up a new server. The engineers thought it would take only 4 hours. Those engineers nearly killed that company.

Do you want to be a 10X or 100X systems engineer? Then use configuration management! After we put that client's systems in Puppet, we could bring up a new server in under 5 minutes. Yep, that's 4000X faster.

If you aren't using configuration management and don't know where to start, just use Ansible. It's by far the simplest and easiest to get going. I've written a quick intro to it here: http://devopsu.com/blog/ansible-vs-shell-scripts/

If you're curious about how the configuration management tools compare, check out my book: http://devopsu.com/books/taste-test-puppet-chef-salt-stack-a...

Completely agree with Matt. You should get your house in order before aspiring to do automation or continuous deployment. I worked with one client who was trying to check off a list by trying to implement a continuous deployment on a system that he was planning to retire in a year. Bottom line is if you have system problems, fix them first, lay a good foundation and then move on to automation efforts and continuous deployment.

IMO, the example you're using in your blog should be simplified even more. I've never heard of Passenger before and it's not something I've needed.

But I have needed to ensure that a file exists in a particular folder on the server. An example of ensuring that would be more universal. Then you can explain that this vs a shell script is better because it's error prone/difficult to run the shell script twice, etc.

But I did get a lot of value out of this article because now I know that Ansible exists. Thanks

Ansible is pretty decent but new release seem to break backwards compatibility quite a lot so be prepared for playbooks not working on newer versions should you need to reuse them at a later stage. Of course virtualenv gets around this quite nicely.

> I could go on for hours

I, for one, would love to read more of these.

Likewise. I would also love to see some of those examples (wherever possible) being addressed using a configuration management solution like Ansible.

I really like the resources that ThoughtWorks lays out on their CI page -- down at the bottom they have some whitepapers on the topic that are certainly worth a peek --

http://showcase.thoughtworks.com/continuous-integration

We are language and environment restricted to waterfall at my job. The language does not have any form of unit testing and an extremely archaic version control. The language does not have a viable way to encapsulate small changes without coding the whole shebang. The environment itself follows some of the same pitfalls. CD sounds amazing in theory, but some languages simply do not fit into the paradigm.

If your language has a way to put an if condition around a block of code and change the value in the if condition at runtime, then you have enough to hide experimental changes behind a flag.

If your language has a way of calling an existing block of code from a new location and checking the results, you have enough to write unit tests.

I don't know any languages which are unable of those things.

I can't think of languages, but I can think of environments where it can be difficult. Maintaining software for embedded systems developed ages ago, for example. Unit testing of your new algorithm is, on the platform, not feasible. You may be able to put the code segment and mock up the rest of the software/hardware environment on your PC, but then you'll find out about undocumented features when you push it to the platform, or performance characteristics that make one algorithm less suitable than another (memory usage, time consumed). And you have to persuade those above you to give you the time to do this, when they don't see the value and want to get the code changed, reviewed and pushed to their normal testing procedures (which might be white-box testing, but is likely black-box testing at its finest).

And when your office has been built around this model for years, decades likely, introducing unit testing for new desktop apps (support apps like I make), is hard because they don't see the value of it. They have a process, I should just stick to it (that's their attitude, not mine).

Some business-critical DSLs are indeed this bad. Think "business process modeling" tools that are intended to be programmed through a GUI.

Can I ask what language doesn't have any form of unit testing?

Not sure about the OP but any procedural language would run the risk of being very difficult to write unit tests for, especially some of the ancient chron file based languages that a lot of U.S. manufacturers still rely on.

But hey, most programmers today (wow does that sound like an old fart) don't know there's a difference between event-driven programming and procedural programming, because all they've ever worked on is event-driven.

I wrote up a bunch of pros/cons on continuous deployment about 3 months ago

http://mayerdan.com/programming/2013/08/04/thoughts-on-conti...

I am pro continuous integration, continuous delivery to staging. I think it takes extra care to do continuous deployment to production. There needs to be a number of systems in place some of which @mattjaynes mentioned others really depend on having a very solid and accessible deployment pipeline.

For smaller product, continuous deployment can move at a really fast pace. For larger project, continuous deployment has to have a release and deployment cycle.

Google I heard does an amazing job at this. If I remember everything I heard from a Google talk, before peer review there is a pre-build/testing process. And while waiting to be reviewed, there should be a build triggered (or the build should sits in a CI queue waiting for testing).

Reply on news.ycombinator.com