Ansible 2.0 released | Hacker News Reader

Ansible 2.0 released

ansible.com

265 pointstachion82 comments

Ansible is probably the single most useful tool for doing working with servers today. It somehow manages to be simple and very featureful at the same time. Personally, a good chunk of my effectiveness in workings with servers and cloud provisioning can be directly attributed to it.

I've been using pre-releases of 2.0 for a few months now, mainly for its improved support for AWS services like dynamodb and IAM policies. I've had nothing but good experience with the new version, which cleans up the code and brings more consistency. Working with AWS, ansible shows its strengths by exposing to the developer a simplified API which gets you 95% percent of the way -- pretty much what you wish you'd get from Amazon. I feel that ansible does cloud provisioning not just easier but better than most other tools, including Amazon's Cloud Formation.

Ansible's playbooks (recipes for provisioning servers or infrastructure) read like pseudo-code, and although it's sometime not obvious how to write an idempotent playbook for a piece of software, it is always obvious what an ansible YAML file does (despite the obvious shortcomings of YAML for this job). This is important in the devops world: all knowledge about the infrastructure is codified in version-controlled ansible code and doesn't get lost when the job passes to a new hire.

This also means that you can often find a playbook for software you want to deploy already written on github (search with "language:yaml"). Often you won't be able to copy and paste it verbatim, but looking at how someone else has solved a problem (install java, configure apache, etc.), it will be obvious how to replicate it.

Every time I finish a piece ansible code, I smile -- about how many other tools can you say that? Congratulations to the team and all contributors for the new major release (and to the company for its well-deserved recent acquisition)!

Same here. Ansible has been added to my short list of life changing programs in my tool belt along with Mercurial and Emacs. These types of applications come along every 10+ years or so for me.

I fully agree. Ansible has a small learning curve.

There is little effort in getting started, since it runs over ssh without any daemons or agents required on the managed hosts.

For finding playbooks, you can also look at https://galaxy.ansible.com

Ansible and Salt were both huge steps forward for provisioning. However, after switching to immutable resources I've largely replaced the need for such services.

I don't mind that there's a panoply of bash scripts to configure docker builds. It's simple and it works and nearly every engineer worth anything knows how to read and write them.

A good bash script never goes out of style.

I might be the only one that does this, but I like using ansible with docker.

I use a Dockerfile for any long running tasks like installing dependencies so I can take advantage of docker's cache. I then use ansible for fast running configuration like using a template file for filling out user specific configuration.

The biggest disadvantage to this is that you have to add ansible as a dependency in all of your containers, which stinks. I feel like it produces easier-to-read configuration.

Don't get me wrong, bash scripting is a great tool. I just try to avoid it whenever I can because I personally don't like writing bash scripts.

I think in the future there might be a best of both worlds approach if docker adds a caching api to where ansible could interact with it directly. There is also a docker connection plugin added in this release that could add some possibilities.

I don't think you're the only one. I ran saltstack with my containers for awhile, but it just created too many dependencies (as you notice yourself) and made the build process difficult.

I find that if I keep things in bash the images become more "dumb" and therefore more portable.

In lieu of a caching API you might find a package cache (apt-cacher-ng, squid, etc) will help speed up installs a lot. I found one image that was pulling down almost 5 GB with every build!

Writing ansible is faster and less error prone than a bash script. It's additionally better organized and maintainable. I use it for both immutable and manual deployments.

True... if you suck at Bash.

Bash unequivocally sucks this is not to say it doesn't have its uses.

No, it doesn't suck more than YAML-based scripting. I write complex Bash scripts daily - if I have to write the same in, let's say, Python, it would take longer, will possibly run slower, and will definitely take longer to test. Again, Bash is a nice tool, but you need to learn it well.

how do you test bash scripts?

I've used bats[1] in the past. I also recommend the command line shellcheck[2] for linting.

[1] https://github.com/sstephenson/bats [2] https://github.com/koalaman/shellcheck

Sorry but the question is not if you suck or not at bash. The issue is that there's always someone else that DOES suck at it, in unimaginable ways, and the regular guys like you and me have to deal with it. And there, Ansible helps!

Writing bash scripts becomes a lot less error prone with shellcheck[1]. Syntastic for vim integrates well with it.

[1]: https://github.com/koalaman/shellcheck

In terms of user configuration in containers, Tiller (https://github.com/markround/tiller) is very much worth a look.

I've built a few hundred lines of code Bash library, and Ansible feels like an overengineered tool now. But it's true that immutable infrastructure changes the requirements as we use Packer and Docker these days. I personally think writing Bash equivalents in YAML is crazy. The abuse of YAML these days is unbelievable. It reminds me the days when people invented XML scripting, just because it was easy to parse XML. YAML-based scripting is no different. I'm sad that Chef doesn't get as much attention as it deserves as Chef Solo is sane.

I actually use Ansible to build Docker containers and it works really, really well. Much nicer than Bash scripts.

We use Ansible for basic installation of a new cloud server and Docker to deploy services on these machines. The two technologies together work just fine for us.

There are some good changes here:

https://raw.githubusercontent.com/ansible/ansible/stable-2.0...

- Given the dynamic includes, you can use variables in some places you couldn't before , which allows you to pass items in on an include.

- Error handling now uses a try/catch structure with block/rescue/always, which is much clearer than capturing a 'register' variable and handling it later

- It has a new 'free running' mode, where operations can be run on remote hosts as fast as they can be processed, rather than at the speed of the slowest host

- Lots more cloud-oriented modules

Love Ansible, but am I the only one who has found 2.0 to be unbearably slow? Some patterns I use may now be anti-patterns I should factor out, but even with some optimizations, I'm finding Ansible 2.0 to be 3-4x slower than 1.9. Maybe it has to do with # of hosts or variable field depth, but it's really really depressing.

I also use and really appreciate Ansible. I tried out 2.0 yesterday and found the sheer number of incompatiblities with my existing work very frustrating.

I wish there was a clear migration path of exactly what needs to be changed to port things from 1.x -> 2.0, rather than just many cycles of "try it and patch".

As another Ansible user contemplating the 1.9-->2.0 migration, I'd appreciate if you would submit PRs updating the porting guide with stuff that you hit that wasn't already mentioned:https://github.com/ansible/ansible/blob/devel/docsite/rst/po...

Thank you for pointing me to the porting guide doc! If only they'd linked to that in the article ;)

Do you have connection pipelining enabled?:

[ssh_connection] pipelining = True

Ansible 2 is extremely fast for me.

But then you have to disable requiretty in /etc/sudoers or some commands will fail. E.g. synchronize.

Yes, pipelining is on. I've been doing some further testing and I suspect some of the nested roles we're using are the source of the slowness, because for simple playbooks and simple tasks, the speed is comparable (Ansible 1.9 still comes in a little faster, unless I enable the `free` strategy in 2.0).

I am starting with Ansible and pretty much my first thought was to write my own "super" python scripts that called Ansibke instead of using playbook and their DSL

But, is this a good idea? As the first six comments here are starting that discussion, what are the pros and cons?

The API may be rich but it feels a second class citizen. But DSLs are really hard to get right and often lack ... Everything.

I would encourage you to start by learning Ansible's yaml syntax, which really goes a long way and which represents a lot of best practices in the devops world, such as immutability, configurability and config file templating. I have yet to run into a situation where I had to reach for python, calling the API directly or even extending ansible itself.

However, sometimes you may want to mix local actions (say, spin up ec2 instances) and remote actions (provision said ec2 instances). In these cases, a simple wrapper bash or python script to combine these has proven helpful.

This has been my use case. We spin up a lot of trial servers on aws for customers, each with some unique data. I've written Python to call Ansible (1.9) programmatically, which was a fun challenge, even if possibly more work than was needed: https://serversforhackers.com/running-ansible-programmatical...

Ansible 2 changes a lot of that, so a followup for how-to in Ansible 2 is in order eventually.

We wrote our own Api wrapper to run Ansible commands on our infrastructure. It aims to encapsulate all the complicated bits. Maybe this is of interest to you:

https://github.com/seantis/suitable

We use Puppet mainly, but depend on Ansible for updates, migrations and other things where we want to control the transition, not just the end state.

This is actually pretty neat. Thanks for posting.

Thanks, I'll definitely give that a look!

We use a super script that calls ansible. We have multiple AWS accounts and multiple environments within them and it's tricky to avoid repetition of configuration without an external tool. I see it as a good practice.

The different cascading layers of variables can be hard to reason about, but it does lead to a good amount of DRY.

Man, I am really happy that playbook parsing and error reporting has been improved.

Almost all the frustrations I have had with ansible haven't been from individual modules, but from the mini-language in playbooks defining what modules gets run when with what input variables. Almost pushed me to do away with playbooks entirely and just call ansible modules from a python script directly.

Ansible would be much better if they just used Python for the playbook API instead of YAML. Ansible would be a clear win over over-engineered competitors like Chef if they fixed this, but as-is I don't want to use it.

Using python as the playbook API sounds dangerous. I think the idea is to restrict playbooks so they can be sanitized.

To me, it still seems like a win compared to over-engineered solutions like chef.

I see absolutely no point in this. Code is data, and data is code. Why are Ansible users so eager to cripple themselves?

Do you know that some people use jinja2 templates to generate their YAML playbooks? Do you know what could have helped them? A real programming language.

Power comes at the price of complexity. If you want to write your playbooks in lisp, ansible is not the way to go.

This was posted on hacker news sometime ago,

https://news.ycombinator.com/item?id=10567408 -> http://lukeplant.me.uk/blog/posts/less-powerful-languages/

I think it is relevant to this discussion. Having used Puppet before ansible, I tend to agree with it.

The point is that it's not turing complete. Thus it's more readable, less susceptible to technical debt and less susceptible to bugs and always by definition, declarative.

Turing complete code turns into an unreadable mess far more easily.

But isn't in fact Turing complete?

It has variables, control structures, loops and exceptions.

That does not make it Turing complete.

"You need some form of dynamic allocation construct (malloc ornew or cons will do) and either recursive functions or some other way of writing an infinite loop. If you have those and can do anything at all interesting, you're almost certainly Turing-complete." -- http://stackoverflow.com/a/449170

There is a do-until loop: https://docs.ansible.com/ansible/playbooks_loops.html#id22

Which satisfies the infinite loop issue.

Have not looked carefully at the dynamic allocation, but I am guessing (hand waving here....) that it could be accomplished.

I see your viewpoint, but I personally feel that YAML is a much better solution. I don't want to run arbitrary code in my playbooks! I just want to be able to trigger ansible modules.

Besides, it's really easy to write your own Ansible module (in Python), which seems to solve the issue for any use case I can think of?

You've never wanted to do something as simple as map over a list in your configuration to handle repetition?

You're talking about eliminating duplicates in a list right?

    my_list | unique

I think this is what you're looking for. Any transformation of data that can't be done with the standard filters can be done with a simple filter plugin written in python.

>You're talking about eliminating duplicates in a list right?

No, I'm talking about any atbitrary list processing operation. Some combination of map, fold, and filter, for example.

So, in order to do that in an Ansible playbook, I need to write a Python function to do it and use it in the YAML file as a filter because the DSL isn't expressive enough to do it on its own. So... why wouldn't I want to just use Python again?

I feel as though I do process lists via ansible. You could be less abstract and provide code. Otherwise, it is hard to take this complaint seriously.

Here we install Jenkins jobs....

  - name: create the job config.xml files
    sudo: true
    template:
      src={{ item }}
      dest={{ conf_jenkins_home }}/jobs/{{ item | basename | replace(".xml.j2", "") }}/config.xml
      owner=jenkins
      group=jenkins
      mode=0600
    with_fileglob:
      - ../templates/jobs/*
    notify: restart Jenkins

And install SonarQube plugins...

  - name: install sonar plugins
    sudo: true
    get_url:
      url={{ item.value.url }}
      dest=/opt/sonarqube/extensions/plugins
      owner={{ ansible_ssh_user }}
      group={{ ansible_ssh_user }}
      mode=0755
    with_dict: "{{ sonarqube_plugins }}"

It's not the yaml part per se, which is easy to write, it's their weird mini-language they've encoded in yaml, with different vars coming from different places in different conditions.

Have to say that things are a lot better than when they weren't consistently using jinja2 to parse the strings.

Indeed I always liked Ansible, but the input format does occassionally get a little awkward. They've basically designed their own simple language semantically, but they constrained it to be expressed with the syntax of a data serialization language.

I think the choice of YAML is because Ansible is supposed to be declarative and a serialization format makes sense there, but some of the "metaprogramming" features they've layered on top like loops can feel a bit strange. Like I said I still like it, but I do feel occasionally that it might be nice to have another frontend that is more like a proper programming language.

in what way is Chef over-engineered?

Here's one select example: https://docs.chef.io/attributes.html#attribute-precedence

There's plenty of this stuff all over.

In what way is attribute precedence over-engineered? You can ignore most of it for most cases and just use default for everything, but when you need it, it's there for you. I don't see how it's more or less simple than what Ansible has defined: http://docs.ansible.com/ansible/playbooks_variables.html#var.... I count 16 precedence rules in Ansible 2.0.

Every time a new configuration management utility comes out, everyone loves it because of it's simplicity or some arbitrary measure of "lightweight," but once Systems Engineers need to solve real automation problems in the real world, they start adopting features to work around the illusion of being "lightweight."

I like Ansible for automating my home network where I have simple problems and no need for environment separation. For professional work, I stick with Chef or Puppet.

It's got a lot of complex rules to memorize, which aren't exactly intuitive either.

I don't think that's a very good example. Attribute precedence is easy to understand, and it's very necessary. After working with it for a bit it becomes very intuitive.

And all of the config management tools have a comparable concept. Ansible 2.0 has 16 levels of variable precedence, Puppet has hiera which gives you arbitrary levels of precedence, and Chef has attributes, of which there are 11-15, depending on how you count.

Allow me to refer you to my favorite bit of Chef documentation https://docs.chef.io/attributes.html#attribute-precedence

Also the Chef server is more complex and written in more languages than the stack of an entire early-stage startup https://docs.chef.io/chef_server.html

And the server uses its own versioning, because what I really need is something competing with my DVCS

Too many things to count, really

To make this a fair comparison, you should be comparing chef-solo (or chef-zero) deployments to Ansible since the closet thing Ansible has to a dynamic source of truth is dynamic inventories. But we'll table the discussion of whether or not I should be required to build my own CMDB from scratch or not for now...

The Chef Server stack is somewhat complicated, though with an omnibus install you will rarely have to interact with it at that level. These days, it scales well up to thousands of client nodes with no meaningful effort.

I'm not sure what you mean by the server using its own versioning. Cookbooks are versioned, as they are deployable artifacts. Environments then define which versions of all the cookbooks exist, so you have an easy promotion strategy. The challenge with Chef Server and DVCS is that you have to centralize the uploading of Chef artifacts (roles, environments, data bags, cookbooks) to Chef Server if you have a team with more than 1 person. That challenge is not unique to Chef, however. Hopefully, a team of Ansible users are not running Ansible playbooks directly from their workstations. If you try to upload a version of a cookbook that is older than the current version, it yells at you, so it makes certain bad behaviors more difficult.

There is plenty I don't like about Chef. And Puppet. And Ansible...I've worked with all of them, but I still haven't seen a valid argument to support the phrase "over-engineered" when used on Chef. Of all the configuration management products out there, Chef is still the most flexible and the most powerful.

>The Chef Server stack is somewhat complicated, though with an omnibus install you will rarely have to interact with it at that level.

Omnibus is the worst thing to happen to packaging. It bundles its own copy of everything.

Is anyone that's not an Opscode employee capable of building Chef Server from source? I tried and failed. Software that can't reasonably be built from source is proprietary in practice. I want to run Chef Server on Debian but I can't because its seemingly impossible to build, so I can only use a platform that Opscode provides a pre-built binary for. It's a terrible situation to be in.

Have you looked into SaltStack at all? The default is to write states in YAML+Jinja but you can change it to use pure python, or a PyObjects renderer that gives it a very pythonic API.

I think the reason they didn't use a procedural language is because they specifically wanted to keep the playbooks declarative.

Ideally, the playbook should describe the state you want the system to be in.

This is 'almost' the case for most modules and I think this was a very good design decision.

Having said that, the ghastly YAML format was almost a deal-breaker for me too.

Using a real programming language doesn't preclude a declarative programming style.

No, but using a 'real' programming language doesn't force you to write in a declarative style unless you use something like a pure functional language and before you know it people would start making a mess in their playbooks using the whole power of Python. Ansible is awesome for devops precisely because it restricts what you can do and forces you to try to do things in a consistent declarative (and hence idempotent) style, which is important to devops.

[edit: fix broken english]

This could have been solved by choosing a language like lua and restricting the runtime environment - which is what is done in game software. Lua also has a tight, efficient interpreter. Making a mini-programming language in YAML is _coding horror_.

> the playbook should describe the state you want the system to be in

This is the real reason.

The playbook is the final state. How it gets there is of no interest to you. All you need to be concerned with is the final result - the state described by the playbook. All the extra complexity is handled by the software.

But you can declaratively specify a full system configuration in a real programming language, too. This is precisely what the GNU Guix system configuration API does, for example. You write OS configs in functional, declarative Scheme code.

http://www.gnu.org/software/guix/manual/html_node/System-Con...

> ... keep playbooks declarative.

I guess I've never really considered playbooks declarative. They are written like and feel like a script I'd write in a language like, I don't know, Python. Now, Salt state files I would consider declarative. They are simply Jinja templated dictionaries (Ansible is YAML that tries to find values templated with Jinja; BIG difference as I've discovered using Ansible for the last 6 months) that actually describe a set of states the host should be in. Ansible's playbooks encode a set of function calls that should be applied in a particular order. Don't get me wrong, playbooks ARE powerful in their ability to describe processes that span multiple hosts and I recognize that, but I'd really like to use a real programming language rather than some half-baked one that is encoded in YAML.

I agree with the GP that I wish the Python API was a bit more robust. Currently we have several tools that generate playbooks on the fly, write them to temporary files and then run them with ansible-playbook to ensure we don't lose some functionality that makes playbooks as powerful as they are. Now, it appears the API has significantly changed in 2.0 and it looks like they expose (read: recommend using) more of it making it easier to script with Ansible.

Ansible's playbooks aren't declarative for me, I've came to the conclusion that they're a framework to write shell scripts.

Dependencies are expressed by the sequence of the code blocks in the playbooks and roles, and I see them as implicit. In contrast, Puppet makes the dependency tree explicit with the "requires" language construct.

The thing is, Ansible works well enough for 85% of the cases, where the additional complexity that Puppet brings is not needed.

You mighh be interested in looking at fabric in that case - similar idea of running commands over SSH via python. The downside is that it isn't as fully-featured because you have to write the primitives yourself, but it does allow you to write simple recipes with ease.

I hate ansible and swear at it regularly. That bastardized YAML syntax is the main reason. Unfortunately, it's still better than everything else I've tried.

Excited about the block functionality and the new execution strategies! I will definitely try them both out to see if I can optimize my Ansible code even further.

I'd love windows support on the control side.

I only skimmed over the release notes, but I hope they avoid adding too much complexity and keep it's prized simplicity.

Really looking forward to getting into this but it doesn't look like they've documented all the modules yet. Specifically I'm trying to find more information on the hashi_vault module

I just finished a deployment using Hashicorp's vault and ansible, but we did it without a module.

Do you have a link that references this?

Or, if you are interested in what we did I could probably put it into a gist

It's referenced in the changelog as a new module: https://raw.githubusercontent.com/ansible/ansible/stable-2.0...

After looking more closely it seems like it's just used to pull data from Vault. I'm currently using Ansible for Docker deployments and have been planning on using it to push data to Vault during deployments so this won't really suffice.

it would be great if rolling updates would work now..

What do you mean?

I was about to make some witty and intelligent comment about Ansible and its usefulness for cloud orchestration, and then I remembered that I've drank over half a bottle of wine and it's time to pass out, so g'night.

Wow, that was stupid of me.

Here, to make up for the above comment:

I've found that Ansible combined with Rundeck makes a powerful automation engine, and can replace more complex systems like Jenkins pretty easily.

Reply on news.ycombinator.com