Ansible vs. Chef (2015)
tjheeta.github.io
tjheeta.github.io
- Running a job against a single host will finish in 3 minutes... running that exact same job against thousands will take well over an hour and max out your machine.
- Running against more than around 3k hosts will somehow consume all 60GB of RAM and trigger the oom-killer
- CPU usage on the ansible runner is absurd for a large amount of hosts. We're currently using a c4.8xlarge (our biggest box) just to run deploy jobs and have them finish in a reasonable amount of time (10-15 minutes)
Slicing up our inventory into chunks and running them on different servers sucks big time and is pretty hacky. How do I combine the results? Can't do orchestration like "Run X on these roles first, then run Y on these roles when you're done".
Most likely what I'm going to do is have a single server execute ansible doing only the following in async (aka CPU friendly) mode:
- Upload a current copy of ansible to S3
- Upload the configs to the target machines with ONLY the secrets that role needs in plain text. (I'm not putting my vault secret on every box!)
- Have the servers pull it down and execute in --connection=local mode.
- Wait until each remote finishes
All that said, I LOVE LOVE writing stuff in Ansible. It is so easy to read, follow, and understand. I picked up most of it in a day or two just by reading their "Best Practices" page. Getting it to work at scale hurts though :(
Have you looked at Salt (salt stack)?
I never liked that the variable namespace is global, so there isn't anyway for a module to be self-contained. If you execute Module 1, and then Module 2, Module 1 can set a variable that inadvertently affects Module 2. The "recommended" way around this is to prefix all variable names, but this becomes unwieldy very quickly as your variable names grow in size. The "nicer" way to do it would be to set have a dict/hash of variables, but that makes top-level overrides difficult because there is no way to override "hash_name.variable_name" you basically have to override the whole "hash_name" variable or nothing at all.
I found it difficult sometimes to reason about how these variables would work, and if I needed to add something to defaults.yml or variables.yml in a module.
More than that, roles are meant to be distributable components on the Ansible Galaxy service they run. Galaxy gets almost no use because modularity and reusability is broken by having no idea how the role author namespaced their variables. Collisions happen all the time. Why we must manually manage scope with naming conventions when the computer can do it automatically with scope is beyond me.
This is Ansible's biggest downside in my opinion. I've talked with the core devs about it on IRC and they (bcoca) agreed but thought it too late to make such a pervasive change as introducing role-level scope.
As some of the other posts have mentioned, I still love Ansible despite this shortcoming.
It's the problem space, honestly. It all depends on the guardrails your workflow provides, and there's never enough in common.
Anyway, there's a reason everyone loves golden-images.
Have you used the variety of options to throttle this down? Running against 3,000 hosts at once is, in my opinion, absolutely crazy. And almost an contradiction to:
> Slicing up our inventory into chunks and running them on different servers sucks big time and is pretty hacky. How do I combine the results? ...
How do YOU combine them all? All results from all 3,000 hosts.? And it's not "hacky" at all, it's what you should be doing as a systems engineer/architect in the first place. No one network should be 3,000 hosts in size in a flat network structure; that should be partitioned up into logical arrangements for easier management (which you currently don't have, as per your own points), not to mention security.
As for combining the results. Have you considered writing a simple module for Ansible which sits locally and it called after every task is complete? It's easy to work with and you can push the results into ElasticSearch.
Ansible isn't a magic bullet, it's a tool on which to build, so build on it :-)
As for the deployment patterns. If you're in the cloud then you should be baking AMIs (or equivalent in your cloud provider) and shipping your configuration the same way you ship your application code, as native packages like .deb or .rpm. If you jumped on the docker bandwagon then your hosts are basically there to look pretty and host the containers which means you have some other way of getting configuration to your servers, i.e. etcd, consul, etc. so the problems brought up in this post don't exist in that setting. You are also probably using some kind of container orchestration system like kubernetes so again the problem of orchestration and deployment is offloaded to some other system. The only problem you have in that setting is doing a rolling deploy of containers and halting when things go wrong.
I think the only place any of these tools make sense now is some private on-premise cloud. Ever other place has already moved on.
I really want to see a clojure/clojurescript based config management system -- it would be so pleasant to write EDN/sexps for basic config, and yet have it be a real language when you need to do something hard.
Edit:
Forgot to mention, pallet [1] is something like that, but unfortunately it appears to be mostly dead.
Activity of the core project seems to be low, but that's true for many other mature Clojure projects. You can still use them successfully.
Yes, we learned to use less powerful languages where it was appropriate because they're more readable and less susceptible to technical debt.
This principle, in other words : https://en.wikipedia.org/wiki/Rule_of_least_power
Ant XML was as powerful as Java - it was turing complete and terribly designed to boot. That was its primary failing.
Likewise, using turing complete PHP to generate HTML was never as clean as using a less powerful templating language (like jinja2) to generate the HTML. Separation of concerns with a language barrier is a good thing.
If all of this means nothing to you, you've probably created some huge code messes in your time.
I strongly agree with your point, but Jinja2 is Turing Complete as well (it's still preferable to PHP though).
Metaprogramming / C++ style templating, for instance, goes above and beyond the power provided by regular turing complete programming constructs, and while that means that you can do cool stuff with them you couldn't easily do otherwise, they're a massive headache to reason about, debug, and keep free from technical debt.
Similarly, when you take blocks of code and "lower the power" to make it declarative instead of imperative (e.g. using list comprehensions instead of for loops) it almost inevitably ends up cleaner.
If anyone is frustrated by the speed of jclouds (which pallet uses internally), I recommend trying pallet's Amazon-specific driver; it's much faster.
(example: if you made some mistake while provisioning new hosts, you'll have one node data overriding another, without any notification)
Yeah, right... https://docs.saltstack.com/en/latest/ref/renderers/all/
I think the problem here is confusion, and I see it a lot in the Ansible community.
What you're doing, and I can completely understand why, is taking Ansible as being a programming language for defining state. That isn't what Ansible is designed to be.
That YAML you speak of is designed to allow you to define state in a format that is not only human readable, but also machine readable. All that YAML is meant to do is say: x=y. it's not trying to be a programming language or a scripting interface, it's simply a means of setting the state you want.
This is a common pitfall a lot of people fall into when they approach Ansible, and I'm always seeing questions on IRC like: "How do I code Ansible to pull down a file, read it, process it, and after doing some logic, do X?" The answer is: you're using Ansible wrong. Use it as a state management system and only a state management system, and you won't go wrong.
Use a scripting or programming language, write a script/program, and run that on the remote machine if you need highly complex logic to determine state.
On the previous iteration of CM, I noticed that people that had a prior background on development usually chose Chef, while the SysAdms chose Puppet.
Interestingly, I've found that a focus on solving the problem at hand (say, installing a package on a hundred systems quickly and predictably) rather than a focus on using the Absolute Best Tool is much more likely to lead to a pleasant work experience for everyone involved.
That, and "the Best" is frequently subjective and, even when accurate, tends to get replaced in a few weeks by something else.
If that something is templates that I can't reason about if the right context is not available then that's not solving any problem. First and foremost whatever I use I must be able to reason about and debug. Strings concatenated together with other strings based on some weird rules is anything but reasonable and debuggable.
Probably true.
> The only tool in this game that is not braindead is chef.
What's wrong with s-expressions? Code and data at the same time.
Sure, but you need something that will actually evaluate that sexp. May I suggest GNU Guix to fill that role?
Ansible is easy to reason about - it's never surprised me once in use. You have about an order of magnitude less to learn when compared with chef or bconfig.
Also, for setups with small target VM's, it's increadbly handy to not have to install a bunch of stuff on each server and make sure it doesn't conflict with anything else.
But mostly it's that Ansible can be understood enough without devoting a couple weeks of your life to it. And you can come back later and understand what you have written.
I could see this if you're working from one really powerful machine... no, that won't work, it's constrained by SSH, not hardware specs.
I could see this if you're calling Ansible on another host... no, then you have to copy everything out to the sub hosts, who have to copy everything out to their sub hosts... A scalability nightmare.
You can use redis as a distributed store of truth and... wait, what? Now there's a blog post worth reading. Show us how to scale Ansible with real-world examples using redis and autoscaling groups. Please.
Got it. I can see this work reasonably well if you're willing to wait 10 hours for a deploy to complete. Personally, I'm not.
> If you want to have 1000 forks, that will cost about 30 GB of memory
Ansible is not, nor has it ever been, limited by available memory. It's limited by the number of concurrent SSH sessions it can handle while copying every single module to be executed to that host.
There's plenty of reasons and ways to use Ansible for deploying code. Some of the post has accurate and reasonable information, but the scaling portion is pure fantasy right now.
Deploying Ansible: - SSH - SSH keys - Git - CI - ... fin?
Deploying Puppet: - Install the agent on every system (using Ansible, ironically) - Install a set of masters (need HA) - Install a message bus for mcollective (if this is still even a thing?) - Upgrade the agent on ALL of your hosts as and when there's an update or a security patch - Have an agent/process running on ALL of your hosts with root access to the box to execute any command a hacker managers to inject - ... oh you've installed Ansible? I'll stop now then.
I'm glad it's solving problems for you... that is ultimately all I care about at the end of the day.
How do you deal with dynamic resources, like automatic scaling, in Ansible? You don't, because it can't. With Puppet and Chef you can actually deal with that.
If you don't have a rigid routine for updating packages on your systems you shouldn't manage systems. Security will always be an issue, thinking that Ansible solves all issues is naive.
How do you initially bake the original image? What installs Puppet or Chef on that original image? Is it Puppet or Chef? Can it boot strap its self? The answer is no. Something has to create that image and "bake in" the ingredients. Some people use Ansible for this.
"How do you deal with dynamic resources, like automatic scaling, in Ansible? You don't, because it can't. With Puppet and Chef you can actually deal with that."
But it can. You've clearly never tried, but instead made an assumption. I've used both Puppet and Ansible, at large government projects, and the Puppet approach was a complete nightmare.
For your auto-scaling example, I would use Sensu to monitor load and fire off a CI job when that load met some criteria (say 90%+ CPU/network for ten minutes.) That CI job would clone our Ansible repository and call a relevant playbook for spawning more VMs, configuring them accordingly. How many it spawned would just be defined in a YAML file, updated via git if you so chose. It's beautifully simple.
I'll say it again: none of these CM solutions are all-in-one magic bullets - they're platforms or frameworks to build on top of. With Ansible, I would build on top of it with dynamic inventories, custom modules for monitoring and auditing the run time, etc. I do this from my local machine without some monolithic cluster to manage or an agent on all my boxes running as root (or with root privileges - same difference) 24/7.
"If you don't have a rigid routine for updating packages on your systems you shouldn't manage systems."
We create immutable instances which are updated as and when required, launch those new instances, and have our LBs use the new instances, keeping the old ones up whilst connections fade away, then they're destroyed. What's so difficult about that?
What's your point with this remark?
"Security will always be an issue, thinking that Ansible solves all issues is naive."
Er, I never said it did? ... what?
It's security 101, in case you didn't know, to run as minimal amounts of software as possible to reduce the potential number of attack vectors. I can't remember the statistic directly, but when I did a cyber security course online via MIT, there was a statistic along the lines of: for every 1,000 lines of code, there's one bug and potentially, an exploit. The take away from this is: run as few services and software packages as possible to help reduce the number of attack vectors.
On that note, are you arguing that running Puppet/Chef in a master/agent configuration does NOT introduce more services, and thus more code, to maintain, secure, patch, make resilient, keep up, etc? Are you also saying that Ansible's default mode of operation (push and run versus pull and run) over SSH is not more secure? If you believe the latter, than I believe, with all due respect, that it is you that should potentially not be managing systems (I didn't want to resort to what is clearly a personal attack, but hey, you made this a school playground to begin with ["If you don't have a rigid routine for updating packages on your systems you shouldn't manage systems."] - shrug)
-M
There are a number of tools to "bake it" into the original image. Most people don't really "bake it", they bootstrap it using things like cloud-init. And it can most definitively bootstrap itself into an environment.
While a novel approach to automatic scaling it seems rather hacky, instead of using the tools provided by whatever provider you're using, that are built for this exact thing. I can see a number of things that can, and probably will, go wrong with that setup. Sensu having a hiccup, f.ex.
And your assumptions about me not trying Ansible is incorrect. I've used it, and I like some of the concepts of it. But for anything other than one-off configurations or smaller environments, moving to Puppet was an easy choice. Speed being one of the biggest factor, and Puppet is slow as fuck compared to say, Chef. How do you do testing in Ansible?
The last remark wasn't an attempt at "school playground", just a factual statement to counter the argument the article raised about old versions of Chef. If you can't upgrade your packages, the why are you even looking at CM?
I never said that Ansible was insecure, or somehow more insecure than Puppet or Chef, I'm not sure where you even think I wrote that.
I said nothing of the sort. Please quote me.
"There are a number of tools to "bake it" into the original image. Most people don't really "bake it", they bootstrap it using things like cloud-init. And it can most definitively bootstrap itself into an environment."
I would be interested to see Puppet/Chef bootstrap its self into an environment. It could be useful for future use cases.
"While a novel approach to automatic scaling it seems rather hacky, instead of using the tools provided by whatever provider you're using, that are built for this exact thing. I can see a number of things that can, and probably will, go wrong with that setup. Sensu having a hiccup, f.ex."
Because using what your Cloud provider gives you is locking you into that vendor. Amazon and Google both have great offerings when it comes to autoscaling various bits of your estate, but the moment you start using features they offer you, features no one else offers (so virtually all of Amazon's catalog, then), it becomes incredibly difficult to move away from that provider.
"And your assumptions about me not trying Ansible is incorrect. I've used it, and I like some of the concepts of it. But for anything other than one-off configurations or smaller environments, moving to Puppet was an easy choice. Speed being one of the biggest factor, and Puppet is slow as fuck compared to say, Chef. How do you do testing in Ansible?"
Fair enough. Then I withdraw and apologise for my bad assumption.
I've never really found Anible to be slow, because I manage networks in chunks, correctly partitioned, and fenced from each other, which means an update to a product, service, or resource is generally done on only a few servers.
If I had an estate with, let's say, 500 web servers in it and those servers were on a flat network (the same VLAN), I wouldn't dream of rolling out a change to their configuration in one big swoop. That would be a mental thing to do. Instead I would do it with chunks of servers at a time, something Ansible can do for you. I would also utilise Ansible's failsafe features, so if there are any or enough errors during a run, it will back off and stop provisioning. I'm sure Puppet/Chef can do this to.
"The last remark wasn't an attempt at "school playground", just a factual statement to counter the argument the article raised about old versions of Chef. If you can't upgrade your packages, the why are you even looking at CM?"
This is true to a point, but if you're like me, you're probably bored of the OS by now. I am. I don't want to play around with kernel params, disk issues, packages, etc... I want that abstracted away and made easier.
Also, I've seen people, friends of mine, successfully take someone from zero to hero with regards to Linux, using Ansible as a learning tool: you do something in Ansible, you look at and understand the result. It complements learning nicely.
"I never said that Ansible was insecure, or somehow more insecure than Puppet or Chef, I'm not sure where you even think I wrote that."
You said: "Security will always be an issue, thinking that Ansible solves all issues is naive."
And my response is: security is a complex matter, but one way to reduce that complexity is through the use of as little software as possible. Ansible introduces virtually ZERO new pieces of software on your network (the actual boxes you're provisioning), Puppet and Chef introduce a lot, way too much, across your entire estate, and thus is less secure by definition.
I've been toying with the idea of making a trolling-but-no-really deployment framework called tarpipe, and all it does is take some files on your host, get em to a $place in one step, and run hook.sh. Oooooptionally, do some dir moves and symlnks to keep a prior state backed up, and service stop|start on either side of the mv/ln, to minimize downtime.
Usage could be `tarpipe ssh user@host` or `tarpipe <(echo "cd keke && bash -c")` just as easily.
It goes without saying that this simply wouldn't be comparable to Ansible, Chef, or other CM because it's too simple. It doesn't help manage state if it escapes $cwd. But if your application can curb its enthusiasm to a directory... boy is it simple if that's true.
I already do this on the daily to crashland my bashrc and dotfiles on any new remote host. Maybe this kind of explicitly zero-dep deploy would be useful for more situations.
Would anyone have a use for that?
---
EDIT: What the heck: I prototyped it: https://gist.github.com/heavenlyhash/b575092aa84ce9f3e1d2
What happens if you have binary files to move around that are big? How do you insure atomicity? What happens when a run dies midway through? Can you make sure operations are idempotent?
That's even before we get into variables, state, coordination/orchestration, and so on.
I spent a lot of time (a troubling amount, really) in the automation space, and while it's true that simple problems are easy to solve, simple problems quickly become hard problems. Then you're building state machines, binary distribution systems, and now all of a sudden you have an enterprise workflow/configuration management system.
I swear, half the products in this space started out exactly as yours, and the author just kept finding edge cases to fix until someone gave them $10m in series A funding and it was suddenly a business.
Sometimes I need kubernetes and orchestration and teleporting state snapshots and oh my.
Sometimes I just need three files on the dang VPS box.
(I see your point, of course. I wouldn't want to build orchestration in bash. Rather, I'd really like to see at least one tool that looks at all this, and goes "huh. We're gonna KISS. And no, it's never going to orchestrate 200 machines. That's okay.")
You can play the "python's everywhere" card now if you like, but I'll look at you funny.
How many megs, exactly, is the smallest thing you can have ansible push to an empty slave server with no prior contact?
If you're counting megabytes in a land where low-end mobile phones have 16 gigabytes of storage and 100Mbps wireless, then you probably have a boutique enough use case that you should roll your own.
I already use this like fabric in some places and have even built in some early conf mgmt with Go templates. Haven't really open sourced it yet except for the scaffolding at https://github.com/cretz/systrument
I ended up writing something _very_ like this at a previous job, where we were building out a lot of new systems but the existing Ops department were still living in the Solaris days and wouldn't let anything like Chef/Ansible into their world.
It actually worked really well, you got simple one-command deploys, with rollback and so on, which (seemed to) always work without a hitch.
The whole thing was maybe a few hundred lines of bash. That was a few years ago, and apparently that hacky little tool is still in use for all deployments.
- Super simple declarative yaml configs
- Agentless. You needed to have SSH working anyway so Ansible just uses that. With ssh pipelining it's so fast.
- The community support is huge and extensive.
- They have a module for everything and development is constant and active, much of which from the community.
- Hardware and networking equipment can be provisioned just the same as a VM or OS image.
The list goes on. Definitely give Ansible a try.
This is first time I am looking at Ansible and I am instantly taken aback with if conditions in yaml. Looks so strange.
Not to mention that Puppet totally freaks out if you have the exact same step configured the exact same way in two different modules and refuses to run ('just have package X installed, no further config'). Apparently you're supposed to write a whole new module at that point.
My guess is that Puppet would work better in large, complex environments as a sort of live-managed system, but down at my small scales (tens of servers), Ansible is far easier to develop and manage, I find.
If the application being containerized adheres to some principles like taking configuration options from environment variables or some other discovery method, then Dockerfiles are simple enough. But you if have to do anything more than a few steps, we've found it more manageable to encapsulate the knowledge in an Ansible role.
We also build our own base images from scratch using Packer (which we love) and Jenkins (not so much love there). That being said, having the required packages for running Ansible inside the image does bloat things a little bit.
If Dockerfiles had more powerful and elegant constructs we could stop using Ansible and remove a layer of abstraction. That would be great.
First is that management of the master is still a big, big pain point. Unless something's changed in the last few months, you can't meaningfully cluster a salt master. Minion keys must be accepted on each master in turn. Auto-accepting minion keys is risky as hell, too. Master signing keys sound like a great idea until you realize that it means you can't bootstrap a minion sui generis.
My other big problem with salt is that Salt formulas are really popular, but are _completely_ uncomposeable. Time and time again I found places where the action of a formula needed to depend on data that existed elsewhere in the pillar, but pillars can't be parameterized with pillar data (obviously). They're also not versioned at all, so occasionally breaking changes are merged in (I myself am personally guilty of doing this). The idea is a decent one ("separate the data from the code!"), but they end up being a real headache in a complex environment.
There are a few other, more minor, issues I had with Salt, but really my main beef is that Salt is an extraordinarily effective footgun. If you try to avoid writing "salt scripts" (think: Ansible playbooks), it's very, very easy to end up with a really messy dependency tree, crazy templates, etc. Yes, I know that discipline matters, but people aren't perfect.
I think salt helps to enforce that you can't arbitrarily compose disparate datasources with overlapping namespaces and have it magic it's way to a sane solution.
Chef tries to compose its data via a very sensible yet hard to get right ordering mechanism. Salt says "nope, you overlap, it breaks, and you get to fix it". I prefer salt's approach but what I like more is the temptation to write an external pillar provider that will do the right thing for me when I get to that point. It's imperfect but quite powerful.
Their bug tracker (which is 3500 open issues deep) has to be treated as documentation, and the real documentation is often a work of fiction.
I will note that the zmq layer very occasionally seems to loose connections and you need to ssh into a machine and restart the salt minion, that's the only issue I've seen.
It has a fairly nice configuration management language at its core, but it brings so much baggage with it that it's completely overwhelming. They can't figure out what they want to do with the project, so they do a little of everything.
The sheer number of concepts and awkward terminology you have to learn to use it are off-putting. I've had to drag a couple devs along for the ride and the learning curve is big impediment to further adoption in our org.
ChefDK is amazing. You get all the tools to do things right. Foodcritic will make you write good code, fast. It is hard wired for environments. It has a rigid precedence for anything, that is also well documented. I mean anything.
I work with Ansible, Puppet and Chef, and Ansible is cute, perfect for one-off configurations, but it doesn't do anything chef-solo couldn't do, if you wanted it agentless. Puppet and Chef does more or less the same things, I just find Puppet extremely slow to work with (heira, r10k, different environments, etc), hard to debug and in general just slow.
However if you have something complicated that needs templating, 1000s of servers, secrets/passwords to distribute, then something like Salt or Chef is the way to go.
Personally I want to see where NixOS/Disnix/Guix will end up. On paper those seem very powerful and I like the idea behind them.
Maintenance: Sure, chef has a server component. So does ansible, if you're using it the way he suggests (with a host periodically running ansible playbooks on all hosts). Ansible has no client component to upgrade, though, so that's a win, right? It totally is, until Ansible doesn't work on a host and you can't figure out why, and the error logs you get are useless because some of Ansible's many assumptions about what the host's initial state is are incorrect. Chef can be managed by a standard package manager, which costs nothing on the client side, and allows far, far better assumptions to be made.
For the record, I eventually gave up on the chef server, replicated the playbooks to each machine (using a cronjob and git), and chef-solo.
Speed: Ansible pipelining speeds it up, significantly. You can almost get one command a second! Chef runs on host. It is ruby, and goes slow, but I have programmed a lot of chef and run a lot of Ansible, and my average chef run was under 30 seconds, and I've yet to have Ansible run any playbook in under a minute. Some of this is from atrocious default behavior, like requiring all hosts to complete a step before moving on to the next step on any hosts, or the fact that it spends nearly 10 seconds of cpu time on each machine 'gathering facts' at the beginning of it's playbooks, even if none of those facts are ever used.
Fact caching: This is a solution to the aforementioned problems with Ansible. It may make sense in the chef-server context, but I don't have a whole lot of experience with it.
Tags: This is probably a matter of personal preference, but I prefer to give the set of things that need to be done, and have the tree descend downwards based on dependencies from there. The author clearly prefers to specify with tasks when they should happen, and for each host a set of initial circumstances. This one can be argued til you're blue in the face. I make the point that there's a clear tree that can be built of dependencies under my scheme.
Push vs Pull: There's no maintenance cost to upgrading? What is this dude on? When you change Ansible revisions, you have to do just as much work adapting as from chef revision to chef revision. Ansible has always been highly in flux, and not great about not changing default behavior.
Pulls still have to be triggered, but they can (and should) be triggered on-host, in a cronjob. Your monitoring system should alert you when the chef run is out of date, though, honestly, if it is failing on just some of your hosts you need to clean up and unify your infrastructure.
Raw numbers: Ansible scales one large machine. Chef costs you a tiny amount on each machine. One of these scales. One of these does not.
Search and inventory: Oh gods, if you're using Ansible for inventory managment, please don't. If you're using chef for inventory management, please don't. Neither are reasonable tools for the job.
Orchestration: Neither chef nor ansible are appropriate tools for dealing with your application's data model. Full stop. Actually, full stop. There's nothing else of value further down this article. Please don't take any of it's advice.
Great tutorial!
http://docs.ansible.com/ansible/galaxy.html#advanced-control...
Kubernetes, Mesos, Swarm, Fleet, Marathon belong in one category (orchestration tools).
Ansible, Chef, Puppet, Salt belong in another category (configuration management).And maybe it's because we don't have good terminology around this, but your 'orchestration tools' include both backend and frontend tools for cluster management- Marathon sits at the bottom of Mesos, etc- it just seems like a weird group.
Configuration management is a very different thing to what most of ansible/chef/puppet/salt are commonly used for - it's possible to do it but it's a bit of an uphill battle. The core problem of config management is not pushing out things, but rather how do you represent configs with their myriad of interacting multi-level exceptions while also allowing tens/hundreds/thousands of operational changes to be ongoing at the same time.
Ansible for example is good for about 3-4 levels of exceptions, which isn't that much considering region and environment (dev vs prod) are two of those.