Getting Started with Ansible
steampunk.si
steampunk.si
Seriously, it is beyond me how this glorified paralleized shell-loop spewing gzipped python can get away with something like this time and time again... Ansible really is a steaming pile of ill-thought-out ad-hoc decisions, endlessly retrofitted and then retrofitted again, in all tangible qualities and regards. If only we had something better with a big enough audience/"community" that served a similar purpose... sigh
I think in the long term, there may be a point where it will be more important to consider how the new collections are structured if you use ansible in a typical way, but right now at least, it seems like the new release plays nice with all my existing work.
Of course, years later when I finally dug into SQL and interacting directly with databases, I had a much better understanding of their mindset and wish I would have learned earlier.
Now your mindset feels very similar to me - "I'll write my own shell scripts, thanks." So, to avoid making the same mistake twice, do you have any advice for what to read / learn to better understand your position?
Note: I've written plenty of shell, but the shell I've writen is way to hacky to use as a replacement for ansible.
You obviously haven't looked at the code for some of the Ansible modules!
That's also why I chose Bash to teach config management and deployment[0]. Do it in Bash, understand it fully. Then go a pick up a tool of your liking. Or not, if it actually does it job.
Hopefully they were justified in their choice.
Most senior developers I've worked with who started with this mindset, be it about an ORM, a framework, or some library that they did not understand or wanted to spend some time to learn, ended up just proving that these abstractions were not completely useless after all. How? By recreating their own vastly inferior and headache inducing implementations.
I believe that there's an additional layer of skills between seniority and mastery. The former seems to be just about time and experience, whereas the latter adds a dimension of pragmatism and detachment. You become just a craftsman using tools with indifference to nimbly navigate complexity. You don't blame your tools, you just understand them, strengths and weaknesses. You know when to leverage them and when to ditch them.
There are many things I don't like about Ansible and I don't think it's particularly robust, but it is an improvement over manually written scripts, most of the time, because writing robust scripts is not easy.
What I think "senior developers" are usually disparaging is using tools like ORMs without appropriate understanding of how they work, and thus being unable to recognize when they don't suit the task at hand.
The ideal tool is good in the general case, and still reliable in the edge case.
That's exactly why it's good. It beat puppet, it beat chef, and saltstack fails to make inroads for the same reasons. They all make useless and difficult abstraction layers on top of what is essentially running shell scripts in loops over what's ultimately an ssh connection or a worse, proprietary version of one.
I won't defend some of the decisions Ansible has made over the years, but its simplicity is the key feature.
I started writing one about the time Ansible debuted because of some of its design decisions that I disliked. A bit later Docker dropped and I decided most of the whole category was approaching obsolescence already and stopped.
Given the above, and that a competitor would have to overcome Red Hat/investor money and mindshare, it doesn't seem there would be much of a demand for this. Ansible is arguably good enough for the niche it occupies.
for i in $(cat listofips);do ssh $i “some linuxy thing” ;done
Why we keep reinventing wheels in infrastructure management and control beats me
How are people properly hardening this account? I would love to hear how others are doing this because every time I think about our implementation, it just feels wrong.
When I first used this setup I tried to build a server configuration that itself was completely open source[0]. So there was no way to bring secrets (like a root or sudo password) to the machine as everything had to come from the source repository so I had to rely on asymmetric authentication.
If you feel manually filling in the sudo password every time is cumbersome, you could also integrate it with a yubikey or make it fetch your sudo password automatically via e.g. GNU pass.
What I do is SSH as root using SSH certificates. This allows me to use rsync commands in Ansible, because elevating to root after login does not.
What I want to do is generate short-lived SSH certificates for root login. Something like, “Run prod-ssh command, enter password, touch YubiKey, and I can SSH as root for the next four hours.” I know how to make this possible, it would just require a couple days of engineering to do it, and I have other things to do.
If I had more hosts to work with I would make one of them run HashiCorp Vault, and serve as an SSH certificate authority for the others.
If I were a team of people managing servers I would probably switch from Ansible to something else. I would want every configuration action to go through source control before being pushed to live servers. As it is, I’m one person, so I don’t need that.
Evaluate for yourself whether this meets your needs, of course. I want to find my own balance between security, safety (removing foot-guns), and convenience.
It'd need passwordless sudo so only a minor improvement, but avoiding root login allows you to limit which tasks run as privileged/non-privileged users, tick boxes when it comes to audits, and allows multiple users to run the same playbooks under their own account.
As you've just said, it might not fit everyone's needs, but thought it's worth putting out there for anybody who would prefer not to log in as root.
I strongly prefer NOT to have passwordless sudo. Disabling root login and then enabling passwordless sudo seems like a pointless exercise in ticking the boxes—any benefit from disabling root login is undone by enabling passwordless sudo.
* Prompt for username and password
* Create user
* Add it to the wheel group
* Copy over authorized keys
* Edit sshd_config with:
PermitRootLogin no
PasswordAuthentication no
ChallengeResponseAuthentication no
That user is only allowed to log in to those hosts a) with a couple of private keys (i.e., no password authentication allowed) and b) from a few specific (bastion) hosts (controlled via firewall and "Match" rules).
A dedicated private CA and short-lived certificates would be "better" but I haven't yet bothered.
Usually this is exclusively using SSH keys, password authentication is blocked. root login is blocked too, you can only login as a normal user and sudo.
It's quite easy to manage with ansible itself. Make a configuration file to store public keys and username and that's about it. That will work pretty well until the company has 500 employees and wants to invest into a SSO solution to manage authentication company wide (AD/kerberos/pam).
As it happens, a few months after I did that my hard drive failed on my Centos desktop. I was able to install a new drive, install base OS, and run my ansible role(s). Within ~45 mins I was back to my exact desktop with all local files restored from backup.
Since then I've kept up any small changes by keeping my roles in sync with my setup. The result is a self-documenting setup. I started this as an excuse to learn ansible but I now realize I've relieved lots of mental burdens from myself. Its so nice to go thru life now confident that I can go from complete bare-metal to "my setup" over lunch.
In your scenario, once those changes are captured, they can be written into `/etc/ansible/facts.d` and they'll show up in future playbook runs: https://docs.ansible.com/ansible/2.10/user_guide/playbooks_v...
Does ansible pull require a secret ssh key on the pulling machine? Seems like it would but I don't remember.
When I want to get something done quickly and easily though, it's the first thing I reach for.
You can unit test your configuration (which in ansible isn't really practical due to its dynamic nature). You can get completion in your editor...
In practice, though, you write a lot more code and have to comprehend more. Also have a look at community cookbooks, most stuff is very old.
I think chef have totally killed their community by doing that stupid EULA move. https://docs.chef.io/versions/
A lot of the functionality offered by community cookbooks made it way into the core DSL. Most community cookbooks for installing applications were abandoned for other approaches though.
>I think chef have totally killed their community by doing that stupid EULA move.
I agree. Seems like they tried to move to the "redhat model" without having a centos style OSS version. CINC has come along and filled that gap: https://cinc.sh/
This is also part of Ansible's philosophy: to be agentless and as thin as possible.
Puppet and chef are significantly more complex, and historically required remote agents on the hosts they managed.
On the threads about Progress buying chef and VMware buying Saltstack, it was pretty clear that the sentiment of the community was that things have moved passed chef and puppet to Ansible and Terraform.
One of the things i struggle with most in regards to Ansible is that things that are semi-interesting like loops, conditionals, data munging, etc are quite complicated and cumbersome. I agree that things like Puppet and Chef make these tasks _much_ easier and i end up writing more understandable code that works just like any other scripting language.
I also _very_ much appreciate the fact that i can unit test my code in Puppet and Chef. As part of a good DevOps practice, unit testing along with integration/acceptance testing is crucial!
On top of that i believe that the roles in Ansible Galaxy are very much sub-par when compared to modules in Puppet Forge. Most of the modules in Ansible Galaxy are simply "install package, start the service". In general, Puppet Forge modules are much more in-depth and allow for configuration and customization to many aspects of the application you're configuring.
Also, check out Bolt from Puppet: https://puppet.com/docs/bolt/latest/bolt.html
Bolt is an "ad-hoc task runner" very much like Ansible. One of the cool things you can do with Bolt though is apply Puppet code as part of the run, so you can reuse all of the Puppet Forge modules that already exist.
Ansible is imperative. You give it a list of things to do, and it will do each one, in order, whatever you say.
Puppet is declarative. It runs through your entire script to build a model of how things should be, compares the target system to see how things are different, then uses a combination of implicit and explicit dependencies to order changes and apply them to the machine. Because it is building a model, it is able to detect defects like two different modules trying to configure the same thing two different ways.
The declarative models make it so that doing the right thing is easier then doing the wrong thing (ie, making idempotent configuration is easier then exec'ing out). But the imperative model makes doing everything easier (ie, it doesn't guide me to make idempotent scripts, but makes it super easy to automate any old task list).
That is what makes Ansible easier to adopt, the very low level of mental overhead to understand what it is doing. Conversely, it probably also makes it harder to scale, both because performance is basically single-threaded and synchronous and harder to enforce good standards.
PS I work for Red Hat, but have used both Puppet and Ansible extensively.
Once I eventually figured out which of their indistinct products is actually the basic tool I then tried to follow their tutorials, which want me to install hundreds of megabytes of who knows what, set up heavyweight servers, watch video tutorials (!), and who knows what.
When I eventually slogged through all that to get to a minimally working setup trying to convert a few fairly trivial ansible plays for what I would have thought of as fairly standard stuff rapidly turned into "time to write some ruby"!
I gave up.
I wish there was a convenient way of using the modules (which are robust and powerful) without the yaml format and boilerplate.
That said, just to undermine my own point - it is trivial to create an ansible module and i have done that when the alternative YAML would have been horrible.
Another promising effort in this direction is Jetbrains Space, which features CI config in a Kotlin DSL.
But it seems like it covers a bit of a different space to Ansible... Pulumi will boot up your infrastructure but you still need something like Ansible to install packages on the nodes afterwards.
My impression of Ansible in the past was that it's more complicated than it was supposed to be.
I kind of hope Pulumi will extend in that direction over time, I would rather use one tool for everything and I don't see why it couldn't. The docs for writing your own providers are a bit weak at the moment though.
https://github.com/Fizzadar/pyinfra
HN discussion: https://news.ycombinator.com/item?id=23487178
I always describe it as "bash on steroid", it's super easy to write and to run remotely on other servers.
Automation is a preseed, a script to do user management, and a phone home package which sends information about the machine back to a central server.
My main bugbear about ansible is it leaves very little trace -- it runs by copying a file to the remote server in /tmp, executing the file (sometimes under sudo), and then deleting it. I get the notification on syslog that ansible has logged in and done something, but I don't know what, so I rely on the ansible
Compared to when someone manually goes in and does something like "sudo vi /etc/resolv.conf" - even if doing it en-mass, I get notification that joebloggs logged in at 14:53 on Wednesday and edited resolv.conf. If I see something broke about 3pm on Wednesday, that's a great start to working out why it's broken. Ansible means I need to look at the ansible logs, but even then I don't know exactly what was actually executed on the box.
Then of course there's "to foul thing up is ssh, to really foul things up requires ansible" angle too. Automation massively increases the amount of
What a shitty idea that turned out to be.
And it's not simple. It's just as complicated as its competitors. It's full of weird quirks and corner cases, and just like all the other tools in its class it feels like it was more congealed and cobbled together out of random bits than designed in a consistent, elegant way.
(If you dig into the details, you will find some "atrocious hacks" as the sibling comment puts it, but because of how it works, the hacks are almost completely encapsulated at build time and those builds are reproducible, so there's much less to worry about when you're deploying/provisioning.)
Systems administration and automation is complex.
That being said I've found Ansible to be the most useful tool for the job. It does the things I need it to and new modules are usually easy to incorporate into my work if I need specific tasks done. It has its shortcomings but it hasn't blown up into an unmaintainable mess like many of the other solutions I've poured hundreds of hours into.
YAML is not fun, but it's been good enough for the task of pushing parameters around.
The hardest thing for some people seems to be whitespace issues, but that can be resolved easily enough by integrating a linter into your environment, or minimally showing whitespace characters in your editor.
But really, that's not the problem. The problem is when you inherit someone else's playbook that is itself trying to do a hackjob around a community playbook.
I could be biased since I've worked mostly with Python, Ruby, PHP, and Java.
The other alternative is to just give up and rewrite the ansible script, which has been what I had to do on several occasions.
Even allowing a playbook to opt out of the default delimiters in the same way one can in a template file[1] would allow a transition out of that mess
#jinja2:variable_start_string:'<%', variable_end_string:'%>'
- debug:
msg: <% my_var %> needs no quoting!
Recently I had a use-case where I wanted to re-use some data from a defined variable (list of objects) in a second module, but the format had to be different. In python I would have done something like new_var=[{a:i['a'], b:i['b']} for i in old_var], but apparently that is impossible in ansible. The best solution I was able to find after lots of searching was to to pass new_var to a jinja2 template which would render new_var as text, then run that through a json interpreter and set that as new_var. Absolutely rediculous.
It astounds me that a tool BUILT on python lacks any semblance of python's biggest feature, list comprehension.
But "writing" Ansible playbooks is definitely not python. It's its own dsl bastardized between yaml, jinja2 and reserved/special keywords. This is the worst thing I can say about the whole toolkit.
Unfortunately, I'm really productive with it and when I return to work I've written 3+ years ago:
A) It still runs
2) I can read and understand the intent of my playbooks
However, I still find it very hard to manage updating roles. Git submodules is just terrible, and you can't have a different git repo for each role subfolder. So how do you manage updating third-party roles?
OTOH talking about Git: I usually recommend using subtrees in place of submodules as they allow you to actually merge-in the contents of other trees rather than holding pointers to them.
If your hosts have a long duty-cycle Chef may still work better as it handles upgrades and MIA hosts coming back a bit better. I believe someone else in the comments mentioned the OSS project for it https//cinc.sh if you go that route. I've generally found Chef to scale better beyond 1000 nodes, up to about 100k.
One of the useful parts of Ansible was its windows modules. I know that Microsoft has made DSC (Desired State Configuration) to handle config management in a declarative way, but haven't tried it out.
Docker is a fairly precise proxy for the inadequacies of Linux; all it's really doing is papering over the missing, badly needed parts of the OS.
Code review is the key in both cases.
mkdir -p /opt/app
rsync -a src /opt/appUsing containers + docker-compose in my own personal projects over the last few months has changed my life, so always interested in learning some more granular info re: best practices and potential pitfalls... Thanks!
You can absolutely use Ansible to deploy applications, but, if there is any essential complexity in your run-time environment or configuration needs, it seems that Ansible has a tendency to make that complexity compound multiplicatively instead of additively.
Disclaimer: I say this, not as an ops person, but as a developer who often uses containers and occasionally watches people wrestling with Ansible from the sidelines.