Which automation tools/scripts do you use to save time and feel like a boss?

It's a little embarrassing how much I do through emacs: manipulating the file system, url fetching, RPC... stuff you really shouldn't be doing with your text editor. But hey, the tool you know...

I'm not even sure if Emacs will hold up to tools like IntelliJ for Ruby/Rails/JavaScript development at this point, although in that space it's a lot closer.

Overall, however, I agree: Emacs is freakin' awesome. Heck, I booted into it as my shell for a few years. I keep meaning to get back into it but I rarely have the time, and I need to learn more Vim as well. Time is the enemy.

Is there any not confusing doc about saltstack available?

The pillar model and the patterns around it are great. Heira is a terrible substitute, despite kind of following a superficially similar structure.

I also really liked the idea of reactors, but that came to be right around the time I left that gig, so I didn't get to play much with it other than writing some toys.

In general, Salt just feels well architected. Puppet, in comparison, feels like a ramshackle collection of useful things that just don't mesh well. The language design makes me think of Perl without a competent language designer.

I will say that Salt has a higher upfront learning curve. I highly recommend giving yourself enough time to throw out your first attempt at designing anything more than a simple system setup.

In the long run, I think I prefer Ansible, if only because it limits how much control any one server has over your infrastructure. Ansible breaks down when you have a lot of servers (thousands), but IMO at that point you really should be working at a slightly higher level of abstraction with VM images and automated scaling groups.

1. Salt simply broadcasts commands out to the nodes connected to it. Because it has no concept of who should be connected to it, it cannot compute whether a node is missing, or if that missing node matters. It's test.ping command is just another command, too, so it suffers the same problems rendering it useless as a ping command

1. Salt's CLI will return success on failure.

1. It is not able to parallelize within a node.

It helps to know what a "state" looks like in Salt past here (this is YAML):

    a_name:  # the name of the state; the name/and mapping value here comprise the state
      pkg.installed:  # a state function; this installs a package
        - arg1: value  # args to the function. Yes, it's a list of single-element maps.
        - arg2: value
      user.present:  # another state function
        - …

Also, note that this YAML file is templated with Jinja2.

1. You can't create and depend on a "PHONY" target; this means if you want to depend on the result of several states, you're SoL. You can just inline the set of states that you need, everywhere you need them, which is bad, or you can use a temporary file on disk, which is a hack.

1. Attempting to write code (which, really, is what SLS is) in YAML is crazy. (You can actually write these in at least 4 different languages! All Salt needs is a JSON-like structure.)

1. It is not able to pass information between "states" (a "state" in salt is a task in the grand list of stuff it needs to execute).

1. The error reporting leaves much to be desired. If you make a mistake in a YAML file, you can easily receive back errors with wrong line numbers, wrong file names, quoting the wrong part of the file.

1. At least when we used it, there was no way to properly escape inputs to the YAML file. (This might have been fixed; I honestly can't remember. At some point, it was "fixed", but not really, because the fix didn't work for any practical purpose.)

1. The minions would just "crash", and by crash, I mean, just stop responding. Since its not an outright crash

1. Bug reports were generally met with "you're not using the latest version, please upgrade"; generally, I'm understanding of this attitude, as it takes some work to determine if a bug being experienced in an old version is still present in a new version. Salt's architecture made upgrading nigh-impossible: minions/servers of different versions weren't compatible; DNS couldn't be used to switch minions over in some of our hair-brained ideas to gradually upgrade as the minions would cache DNS results indefinitely.

1. Minions would reconnect automatically; for whatever reason, reconnecting took a considerable toll on the server. If you hit the point where there were too many minions talking to the server, the fleet as a whole would go into a death spiral: too many minions attempting to connect causes an already connected one to miss a keepalive ping, and reconnect. Now you have yet another minion attempting to reconnect, and the problem is worse. More and more minions start doing this, and you're hosed. We had to restart the entire fleet of minions (by hand, b/c our manner of running a distributed command on the fleet was salt) multiple times.

1. Salt in some places, such as templating a file on a minion, needs to reference a file on the server. A relative path would have sufficed wonderfully, but instead Salt uses "salt: URLs". a. they're not URLs, and b. they MUST be absolute. This means that if you move a subtree of stuff (such as for refactoring), you're forced to adjust these URLs. Thankfully, they're easily greppable.

Out of all the CM systems I've tried Ansible feels the most sane. It still has a some things that annoy me, but overall I've found it the easiest to deal with.

My current shop puts everything in Docker containers and runs them on DC/OS scheduled via Marathon. I don't touch the CM parts anymore, but I think our DC/OS team provisions the actual nodes via Puppet.