Automated, large-scale deployments with Ansible's pull-mode | Hacker News Reader

Automated, large-scale deployments with Ansible's pull-mode

stavros.io

37 pointsStavrosK8 comments

This sounds like a recipe for breaking your servers without knowing about it. Plus, I hate VCS deployment strategies; I used to do them too but I've become wiser since.

I can see this being useful for a basic sort of "dependency resolution" on a larger project that requires deps sitting in github repos (the recent habit of treating github as a "global package repository").

While the pattern of using github to resolve dependencies isn't bad; I still feel like using it for your deployment strategy is naive. Resolve the deps locally first, package/containerize it, then push it to the servers.

I'm not quite sure how that's a condemnation of ansible, since the same can be said of any automatic pull-based system. In fact, the same can be said of any automation - it's quite easy to break every system in your environment if you make a mistake. I myself once knocked out our entire QA environment via puppet when I made a mistake with the firewall config assigned to the default node.

I'm not condemning ansible, or even ansible-pull for that matter. I love and use ansible in my own company.

"Plus, I hate VCS deployment strategies"

I love VCS deployment strategies! But I'll admit you have to be careful with them - exactly why you need a good test suite and a staging area to test your deployment "code".

Using the word "hate" was probably unfair; I can see it being okay for the small Heroku/WPEngine project deployment.

What I mean by that statement is that VCS was never designed to be a deployment solution. I won't argue with a seasoned programmer for using git to deploy their blog to WPEngine, it's one-dimensional and not mission critical (unless you make your living off of it).

Deployment is much more than just "getting your app on the server". Git happens to be in vogue (for deployment) because it is part of the developer's toolset and already handles transporting and merging diffs. This doesn't make it appropriate for deployment though, the argument is very long and I may just end up writing a blog post about it.

I don't understand what your objection is. Is it servers automatically updating, or using your repository to somehow store deployment scripts?

My objection is automatically updating your server using deployment scripts checked out from a git repository. Git wasn't and will never be designed to be a deployment solution, and automatically updating a cluster of servers using a cron is bad advice. You're a smart man and I enjoy your blog, so please don't take what I'm saying personally (I used language in my original comment that was too emotional).

I would only recommend this if you had a separate server that acted similarly to a continuous integration server: it listens for pushes to a repository, checks out the latest code, builds an image from the latest known stable and runs the new scripts/configuration/whatever in the image along with application tests to ensure nothing is broken.

If everything passes, then the CI server should bundle it up into a .deb and serve it on its own deb repository to the production servers. The production servers (if all running the same image version) should be safe to update using a cronjob; but even then you should make sure that their "image version" is next in the hierarchy. Deployment is then using the native deployment tools to the environment that were designed for it - this way you should be able to completely reverse your upgrade too if something DOES break!

Huh, that's starting to sound like image migration...

> You're a smart man and I enjoy your blog, so please don't take what I'm saying personally (I used language in my original comment that was too emotional).

Oh don't worry about that, I just wanted to hear your opinion.

> I would only recommend this if you had a separate server that acted similarly to a continuous integration server.

Yep, that's exactly the current setup. I should have included a disclaimer that said "only implement this if you're sure it won't break".

> If everything passes, then the CI server should bundle it up into a .deb and serve it on its own deb repository to the production servers.

This is how large-scale deployments should be done, I agree. My post (and the way I use this) is for single-server or two-server deployments for small projects, mainly in startups, where a full-blown provisioning server is overkill. It has worked very well there, but it does take some higher amount of responsibility.

For larger deployments, we're doing what you mentioned: Jenkins continually runs tests, packages the repo in a .deb if they pass, and triggers the deployment through a PPA to the corresponding servers.

I should update the post with rationale on which use cases this serves, I take that too much for granted sometimes. Thanks for the feedback!

Reply on news.ycombinator.com