Hacker News Reader

Comment by erulabs | Hacker News Reader

So... systemd + Ansible?

I really disliked SystemD before I got my hands dirty with it. Assuming you're developing with any modern language, the language itself probably wraps OS differences anyways (Node / Golang / Rust / Ruby / Python / anything-libUV-based to name a few), not like you can convince developers to change their habits anyways.

People act like Docker makes builds reproducible by magic, then go on to not pin any Golang deps or curl internet resources or not generate lockfiles for NPM, ad infinitum.

There is and never will be a shield operations people can put around their developers. You just need good developers. If you want SoA scaling or automatic cloud configuration, why not just use the most established tools available? Linux's init system and kernel _already do_ resource management, even outside of LXC. People act like a "cloud scheduler" is the only thing that lets multiple processes work together.

This whole idea of isolation is also really inane. Each app will have different disk, io, network, cpu requirements. I also dream of the day Kubernetes can do this "sorting" better than an experienced operations team can, but that day is _far off_ (and I will die before I put redis/mysql/postgres/cassandra/any-other-mature-datastore behind docker and iptables)

View on HN

The idea with Kubernetes isn't necessarily that it can do resource allocation more carefully and more intelligently than a human ops person, but it can do so at a level of automation and speed that a human can't.

Pods die, Kubernetes spawns new ones. Server goes full, Kubernetes spawns new node. Server runs out of disk space, Kubernetes evicts nodes until everything is good again. Server goes down, Kubernetes shuffles all the pods over to a different node. And so on. It does this at a scale of hundreds or thousands of machines. The utility it brings also applies to small clusters. You don't remove ops, you just flatten it into Kubernetes' operational model.

Equally importantly, Kubernetes also brings a declarative deployment model. You have the same with Systemd, but at the node level. Kubernetes is a bit like Systemd at the cluster level; its "units" are pods declared in YAML/JSON files, but unlike Systemd they run on any node the cluster, and are subject to a resource allocation model that includes things like persistent storage (e.g. AWS EBS volumes). You don't get anything close to that with Systemd, which is much more analogous to Docker.

I haven't heard anyone ever claim that Docker brings reproducible builds -- why would anyone assume that? What Docker give you is reproducible deploys; build once, deploy many times from the exact same image. Maybe that's what you're thinking of?

I don't dislike Kubernetes, in fact im a (very minor) contributor and regular user and am a member of the federated kubernetes workgroup. A small group of microservices very happily lives on Kube in production.

But, for the sake of the argument:

> Pods die, Kubernetes spawns new ones.

AWS ASGs also solve for this, as does every other cloud provider at the infrastructure level. For applications, systemd and any init system can restart an application when it crashes...

> Server goes full, Kubernetes spawns new node.

I assuming you mean utilization - again, ASGs/CloudWatch solve for this as do every other cloud provider (if they didn't a few minutes worth of Sensu plugins and good-to-go).

> Server runs out of disk space, Kubernetes evicts nodes until everything is good again

Right but this can and has lead to disaster. I got paged for bad response time and found out the issue was that containers were being evicted and pods were going critical because _all_ hosts were filling up on diskspace and there were no healthy nodes on which to put pods. This due to one bad service writing out tons and tons of log data). Had that been dedicated, I would have just gotten paged for one full disk and fixed the problem when it showed up instead of waiting for a bigger issue. This is super fixable in Kube (and was - that service now has its own volume and still lives in kube) - but my point is that it's a problem for everyone.

> Server goes down, Kubernetes shuffles all the pods over to a different node.

This effects response time of user facing applications as new 1GiB+ containers are shuffled around (I know, thats an unfair example). ASGs and Cloudwatch solve for this in a saner / safer way.

As for the reproducible builds, it was in response to the article which says:

> The ‘repeatable builds’ argument is the most interesting to me, in part because this is the same promise that java made in the 90s (write once run anywhere).

You may be correct that Docker doesn't promise this explicitly, but it is a big misconception. Additionally, `tar` solves for "reproducible deploys"...

I'm not an AWS employee, just figured I'd use a common cloud as an example.

Your responses assume that I'm running in the cloud and AWS specifically. What if I need to run on-premise?

Kube is actually fantastic for on-prem/dedicated/colo. I still wind up needing to manage my _non_ Kube stuff - and fall back to Ansible + systemd for the data layer for example.

Additionally, one big complexity re: Kube on dedicated is dealing with custom hardware. Kube and Docker really require modern and specialized kernels - This actually applies to AWS as well; the current Kube AMI for AWS doesn't install the `linux-aws` kernel for example, so you have no access to the ENA driver or the ixgbefv driver, and the t* and i3 instance performance is awful (the most common and the most vital machines respectively). There is a PR for that open somewhere and I'm sure it will be fixed soon, but you see my point.

After the hardware/kernel build (which you'll need to manage outside of Kube, so you'll probably choose something like Ansible), you'll probably want auth management at the host level (assuming you run on-prem), which you'll most likely fall back to Ansible for again.

I should probably also admit somewhere in this thread I actually use and prefer Chef :)

tar does reproduce a deploy but onto c clean slate machine. docker is just a clean space where you can take an issue from there and easier reproduce elsewhere because there is no state left over from installations of previous packages and/or other software.

What makes containers in general, and Kubernetes specifically, compelling to me are the following:

- the deployment artifact is a "fresh" instance that's "booted" in a fraction of the time—as in a couple of seconds vs a few minutes—than a new EC2 instance would be

- since the deployment artifact is a fresh instance, there is no need to manipulate an existing instance already running your application as part of your deploy process (e.g. scp, ssh, unpack, run commands, restart/reload application)

- if you design your build pipeline appropriately, the actual data needed to be shipped is effectively just a diff from the previous image. In my org, that could mean the difference between shipping a 300MB vehicle to ~300 instances and shipping a 300KB vehicle to almost certainly less than 300 Kubernetes nodes

There are other things as well, but I wanted to stay roughly within the parameters of your examples.

If your organization doesn't already have a continuous integration story, I'd say it probably isn't ready for containers in production. If your cloud footprint isn't costing you let's say >= $50k/mo or so, I might argue that the engineering effort to Kubernetize your things is probably better spent in more direct ways on your product.

If you do have a CI/CD story, and you do have a very wide, expansive, and expensive cloud footprint, containers and Kubernetes are extremely compelling possible solutions.

What if you're in the ground-zero scenario: there is an application but practically no infrastructure?

I'm in that situation now (with a SaaS product), and my approach so far has been:

* set up tests

* set up Docker images

* set up docker-compose for easy local deployments

* do lots of work to make the app behave inside Docker

* take control of AWS and IAM using CloudFormation, making it easy to set up and roll back permissions

* bootstrap KOPS/K8s with CloudFormation

Future work:

* set up Jenkins CI on top of K8s

* K8s deployments of the app for CI

* staging K8s deployments

* K8s in production

This has been my first venture in operations and it's been very interesting and rewarding, but at times it gets very intimidating to "be" operations with no senior guidance. I enjoy the freedom and responsibility and I feel like I at least have a significantly better grasp on this than anyone else in the company, but I wish I knew of a way to feel like my work was good in an absolute sense, like "definitely good enough".

Has anyone here gone through a similar scenario, bootstrapping ops infrastructure in a small business? Would you have any remarks regarding architecture, or the politics required to make the importance of solid ops and the improvement of software reliability and security clear?

Generally it's fun resume builder but not actually time well spent on your business. One super server can go far enough and hopefully you can design it to be able to scale as needed later.

This is a risky assertion to make, in some cases for us.

All engineering advice comes with the implicit caveat "subject to your local concerns".

Yes, of course you may be working in a startup that requires 25GPUs to serve even one customer. No sarcasm. I can imagine some startups that might meet that requirement.

But there are an awful lot of startups that start massively overengineering their footprint early when all they need is a web server with a cold or hot spare (use a load balancer with automatic failover if you've got one in your cloud or something, if not I wouldn't stress, and automatic failover on prototype-level application code can cause its own issues) and a database server with a good backup story and some mechanism for quickly bringing up a new one if necessary. (This generally leads you to some sort of clustering thing or a hot replicated spare because it doesn't take long before your database requires hours to rebuild from a backup or something.)

You're often better served just giving occasional thought to how you might split things up in the future and using that to at least hint your design than actually splitting things up immediately.

> you can design it to be able to scale as needed later

What's different about what you're referring to above and what the commenter mentioned? It seems to me like they were designing the app to be able to scale.

Except they are also doing the dev-ops yucky work to do the scaling. It is a lot of extra work with no value until later.

Say they are making a crud app. Keep no state in session. Deploy as easily as possible. Boom, done. Later you can add a load balancer to scale to 100 servers. Right now you can run it on 1. It is 'designed to scale'. But actually running your single app in a 1 node k8s cluster? That is days of extra work for nothing right now.

It's tough to give a definitive answer because every company is different, but I work at a very small strange shop, 5 devs + 1 manager, maintaining 10-15 custom websites (5 of which are on a single VM) and I have been deploying our new apps in Docker (no K8s). I use each container as a "miniature VM" which runs the entire web app (except the database), blasphemous I know. Compared to putting multiple apps on one box, the Docker method adds some minor complexity, but keeps apps isolated. That was my biggest requirement, to prevent devs from cross-pollinating applications, which happened constantly when everything was on a single server. It was much simpler than setting up Puppet on a bunch of legacy machines. I also considered putting each new app on its own VM, but went with Docker because a lot of our apps hardly get any traffic and would have wasted quite a bit of resources to spin up a VM for each (all our servers are in house).

The pros to Docker so far: Dependencies: Dockerfile gives a list of explicit system dependencies for each app. This can be done in other ways with package files or config management but this was not being done before and this is an easy catch all to force it for any different type of environment. Logical Grouping: App environment (Dockerfile + docker-compose.yml) lives alongside codebase in a single git repo Deployment: Deploy to any box with `git clone myapp && docker-compose up` for testing/dev instances or migrations Development: We mount the codebase from a host directory into each container, with git hooks to update the codebase, which works well for us (we have no CI) Plus it's fun!

Cons: Operational Complexity: Devs/Ops teams probably won't want to learn a new tool. I setup a Rancher instance to provide a GUI which makes things a bit easier to swallow. It has things like a drop in shell, log viewer, performance metrics, etc. Network complexity: we never needed reverse proxies before, now we do. Clustering/Orchestration: We don't cluster our containers, but the more we add the more I think we might want to, which would add a whole new layer of complexity to the mix and seems unnecessary for such a small shop. Security?: lots of unknowns, lack of persistence can be bad for forensics, etc. Newness: Documentation isn't great, versions change fast, online resources may be outdated.

Like you, I'm sometimes unsure if this is the right choice. Maybe a monolithic server or traditional VMs + Puppet would be easier, simpler, better? In the end, I think Docker just fit with the way I conceptualized my problem so I went for it. You may never get that "definitely good enough" feeling, but if it fits your workflow and keeps your pipeline organized and manageable, then I say go for it.

That's interestig to me, the Rancher bit: I went the route of writing down all my routine docker-compose invocations in a Makefile and I gave that to the devs with builtin documentation (list of targets + examples of workflows), but I see how Rancher could standardize that.

Very interesting ! I am a solo guy but I sort of followed the same way you did. And when I had to go with the Kubernetes road because managing multiple Docker over multiple boxes became too complicated, I just went back to one website = one VM... Giving me time to learn all the k8s stuff, which will be probably be useful soon but just not right now.

Docker-compose is horribly broken, it manages deps in a much too coarse manner, effect lay becoming a broken package manager on top of whatever the container has.

I agree the code quality and "feature parity"/consistency with Docker is absolutely crap. I'm now hoping K8s will provide a better experience while also being more amenable to quick and scalable AWS deployments.

> What if you're in the ground-zero scenario: there is an application but practically no infrastructure? > I'm in that situation now (with a SaaS product), and my approach so far has been: > * set up tests > * set up Docker images > * set up docker-compose for easy local deployments

In my personal opinion, this is all good work to do. One minor nit: You probably want to get out of the habit of thinking about `docker-compose up` as a deployment, and just think of it as a much cleaner, much more "prod-like" local development environment.

> * do lots of work to make the app behave inside Docker

I'm curious about this. What was the overall nature of the work? Things you would have had to figure out in a distributed platform anyway, or container-specific issues?

> * take control of AWS and IAM using CloudFormation, making it easy to set up and roll back permissions

This is amazing, I applaud you for your foresight. I wish we had known about this pattern much earlier.

> * bootstrap KOPS/K8s with CloudFormation

This is where there's going to be another giant time sink, unless you're already pretty familiar with Kubernetes' resources—Deployments, Services, PersistentVolumeClaims, Secrets, etc—how they work together, what they enable, what their limitations are, and so on. Not to mention picking a network overlay plugin, deciding on a service discovery model, and understanding how Kubernetes works. BTW I am currently working on this very knowledge, and at least for me it's been pretty heady stuff at times.

>Future work: >* set up Jenkins CI on top of K8s >* K8s deployments of the app for CI >* staging K8s deployments >* K8s in production

Everything here seems relatively sane, in broad terms. I mainly caution against moving too quickly and also against spending engineering efforts on this if you are also an IC for the applications your teams are writing.

>This has been my first venture in operations and it's been very interesting and rewarding, but at times it gets very intimidating to "be" operations with no senior guidance.

Honestly, you seem to have the right mindset for this and so far you haven't mentioned anything that I see as a huge red flag.

>I enjoy the freedom and responsibility and I feel like I at least have a significantly better grasp on this than anyone else in the company, but I wish I knew of a way to feel like my work was good in an absolute sense, like "definitely good enough".

Welcome to ops! There will never not be an operational issue that either needs to be fixed or is silently waiting to strike at 3AM on Monday morning.

Tar definitely doesn't solve the issue of reproducible deploys, a tar deploy has a high chance of working differently or not working at all unless the target environment magically happens to be totally identical to yours.

We use docker to solve the distribution/deployment headaches where we don't know and definitely cannot influence the (different) environments where the package will be deployed. Docker made a headache go away for us - we can make a deployment available to others, they can run it, and it generally will just work properly the first time.

The earlier alternative for that style of "reproducible deploys" was full virtual machine images, not tar. The alternative in the years before shipping VMs was easy was configuring it manually on site or (if it made financial sense) shipping pre-configured hardware. Something like tar was never a reasonable alternative for deploys - even decades ago when big unix vendors controlled/owned/managed your whole system, them 'simply deploying' a tar package worked usually, but didn't work reliably.

Dunno if this has been mentioned, but on the reproducible builds front, you can use bazel (Google's build system) to produce docker images: https://bazel.build/blog/2015/07/28/docker_build.html . It's a little more difficult and explicit to introduce non-determinism into a bazel build.

Doesn't auto rolling evacuation like that give the ability for one malicious user that found a flaw in an app to DoS an entire kubernetes cluster by filling disk space?

As an Ops guy, I preach Ansible + systemd all day everyday, but so many of our Devs (and Ops) have drunk the containerization Kool-aid.

Unfortunately, the same culture which fucked modern web design (a new JS framework every 6 months) is starting to creep into Ops; some REALLY smart people make a tool which promises to be X but with X's main problem solved. They do the easy 80% and then abandon the project when the hard 20% needs to be fixed (and forget documentation).

The JavaScript situation is grossly exaggerated. It's doing very well. It's just that the people who aren't productive are also the ones shouting the loudest on HN. Meanwhile, a lot of people are being very quiet getting tons of stuff done really efficiently. (Source: Me, very happy React developer who is so glad we're finally out of the terrible "wilderness years" of server templates and jQuery.)

> terrible "wilderness years" of server templates

wat.

That's fine, it's:

> terrible "wilderness years" of server templates and jQuery

where things go off the rails.

I reckon

> terrible "wilderness years" of ... jQuery

is the part where things go off the rails (and literally off the Rails).

Didn't Rails originally ship with Prototype? I liked Prototype.

I will rather take any server templates and jQuery over flavour of the month JS framework, and will keep doing that until my boss or customer tells me otherwise.

No need for JS for what can be done in plain HTML/CSS.

Offloading view changes to clients can both give a better experience (changes are reflected ~instantly vs a refresh and a round-trip to the server) and can scale much better (e.g. Server only used to process CRUD operations, CDN or nginx / Apache used to distribute static assets like the JS app, index html, CSS + images etc).

It can also potentially help with things like caching, but more traditional server-side templated designs can also use things like Varnish with the ESI feature, utilizing caching mechanisms in the framework / app (e.g. Memcache or whatever), etc.

I used to think SPAs were dumb, but now I love Rails 5 API-only mode + EmberJS + EmberData. It also makes server-to-server a breeze, because you already have your API. Mix in JSONAPI, swagger, and oauth and it's a pretty happy environment to build software in.

> changes are reflected ~instantly vs a refresh and a round-trip to the server

You can achieve that with a little sprinkle from jQuery.

For UI/UX rich experiences, taking advantage of OS animation frameworks and widgets, I would anyway advise native applications.

> and can scale much better

Offloading everything like that to client is antisocial behaviour. Seriously. This goes beyond "bad engineering"; for any popular website, this literally means saving a bit on your servers while causing lots of electricity being wasted by your millions of visitors. This is the computing equivalent of selling extremely fuel-inefficient cars to regular people.

It can give benefits but all too often I'm left looking at a spinner wondering what the hell is going on before eventually being presented with a simple page that would have been trivial to render server side.

c2.com being a tragic example of this.

At first I said to myself, "C2, wtf is this guy talking about I could load that on a calculator".

I was wrong. Load time was an abysmal 14 second(!) wait. I normally don't make jokes on HN but today I'll be pouring out a sip of dark roast for my metaphorical 'dead homie'.

Changes on the client still usually roundtrip to the server to get the data (via a Rest call these days, then parse the JSON response, then render and apply the diffs to the DOM).

And I don't buy the argument that javascript-based client apps scale better than servers rendered partials where there's been over a decade worth of solutions for scaling out the server layer. Also, you'd be applying a lot of the same techniques for scaling a pure API as something spitting out partial web views.

I've been exploring Docker lately, but I've just launched a small project using Ansible and systemd, much much simpler to setup and deploy updates.

> They do the easy 80% and then abandon the project when the hard 20% needs to be fixed (and forget documentation).

That's a nice summary. The 2nd level ignorance of those people is astounding.

Oh, sure, write the 618th package manager, I'm sure you'll be the one who finally gets it right.

I suspect that systemd will end up in the same state...

Funny, i consider systemd very much a part of the containerization cool-aid...

Ansible + systemd + containers is a match made in heaven.

https://developers.redhat.com/blog/2017/05/04/breaking-up-th...

Systemd + ansible is very close to what we have. We are python shop so we use: - salstack to deploy and update virtualenvs and applications. - salstack to install and configure upstart jobs - logging goes to /var/log/upstart but this gets shipped automagically by salstack configured per app awslogs config. - All config is read from zookeeper. - The only thing that is initialised on the machine is a ZK host environment variable (by salt-cloud). Everything is ummutable. (Using Postgres RDS for the moment). A system dies we can just make another one with salt. We can run multiple services on one machine or one per machine or install them in lxc containers trivially.

Minor nitpick... I believe you mean "saltstack". Yes?

Sorry, yes saltstack. Typing on my phone while my kids jump on my head. saltstack is like a toolkit for all this sort of stuff.

Off topic, I just looked up the definition of nitpick. Apparently it refers to removing nits, the eggs of lice :O

https://en.wikipedia.org/wiki/Nitpicking

I did not mean to imply your comment has lice :)

I have kids in primary school, there are probably some lice in there somewhere.

Yeah. Would be really interesting to see a container technology that people were eager to use for DB management. Persistent storage is the thing that hasn't clicked yet.

My guess is this technology will be some kind of cluster management. If there were a 'platform' for horizontally scaling data that managed sharding, rebalancing & backups plus gave me user-space hooks to override policies, databases might slowly & cautiously build on top of it.

Most DBs have their own of these (pgpool, mongos, redis sentinel, all of cassandra & hadoop) but would be cool to centralize the common reqs and collaborate.

> would be cool to centralize the common reqs and collaborate

Oh baby the desire to pitch my WIP SaaS product. This is something I'm extremely keen on and I couldn't agree more!

The problem is that the constraints of wanting your data and CPU on the same node, needing data synced to more than one node, and avoiding being too much of a performance bottleneck, all combine to make it very hard to do better than something which has privileged knowledge of how the database works under the hood. Performant consistency is hard.

smartOS gives you persistent storage btw. And its lovely ZFS storage..

saying smartos gives you persistent storage is like saying docker gives you persistent storage because you can use

  docker run .... -v /storage:/storage myapp

That's not what people are talking about. If you run a container on smartos and the physical machine it is running on blows up, what happens to your persistent storage?

technically it still persists, just in the form of hawking radiation instead of bits & bytes. Some people think this is what makes amazon glacier so cheap.

I'm not too knowledgeable on the subject but I know there are a lot of groups working on persistent storage for Docker. I think most of the time this means a docker volume that lives on some network attached storage (this could be redundant across the compute cluster or on dedicated storage servers), which is then linked to a container via the network. If you remove that container or destroy the host, the volume persists and you can recreate the container on a new host and re-attach it to the associated volume (automatically).

As far as databases and horizontal scaling, this could probably be done currently within each Docker container at the database layer as you mentioned (CockroachDB also comes to mind). So let's say you add a new docker container with its own persistent+redundant storage volumes, and that container is running a cockroachDB node. You can add more containers as cockroachDB nodes, each with their own persistent+redundant storage, and you get all the DB sharding etc across nodes from cockroachDB while the redundancy+persistence are handled by Docker (plus storage solution) as far as restarting, migrating, reconnecting, etc. This probably has all sorts of problems I'm blind to but that's my rudimentary understanding.

Handling horizontal scaling, sharding etc at the storage layer directly through Docker sounds much much harder, and I don't even know what I've been talking about so far so I'll leave it at that

Here's a list of some people doing persistent Docker storage :

  Flocker
  Portworx https://portworx.com/
  StorageOS https://storageos.com/
  Rancher Longhorn https://github.com/rancher/longhorn
  Quobyte https://www.quobyte.com/containers
  Nexenta https://nexenta.com/solutions/containers
  Infinit https://infinit.sh/docker
  EMC REX-ray https://github.com/codedellemc/rexray
  Ceph
  Gluster
  Mesos https://mesosphere.github.io/marathon/docs/external-volumes.html
  Kubernetes https://kubernetes.io/docs/concepts/storage/persistent-volumes/
  vSAN
  Nutanix http://next.nutanix.com/t5/Nutanix-Connect-Blog/Containers-Enter-the-Acropolis-Data-Fabric/ba-p/11122
  Netapp (Docker volume plugins for SolidFire+ONTAP+E-series)
  AWS EBS/EFS

I'm biased but checkout Portworx. It is designed to be that and if you don't think it is, I'd be curious why is missing. https://portworx.com

https://github.com/CrunchyData/postgres-operator is the most promising thing I've seen lately.

I see a lot of "me too" responses to this, and for the life of me, I can't figure out what systemd brings to the table. I understand the Ansible part, but what does systemd do that's notable? My understanding is that it's an alternative to init, and I can't imagine calling out init as part of my deployment/config strategy. What am I missing?

I'm wondering the same thing, as someone who has only started using systemd. From what I can tell it handles:

- starting the various pieces of the app in the correct order - permissions (uid, cgroup, maybe even virtual file system) - environment variables, working directory - passing around sockets - restarting things if needed - monitoring resource usage - centralized logging

You've done nothing to justify your "systemd + sensible" in this comment. You've just ranted against other technologies. I don't see the point.

I suppose systemd could be swapped for "any modern linux init system" and Ansible could be swapped for "any modern cloud configuration tool". There are probably 3 or 4 items that come to mind for both categories.

The idea is that Linux already solves that vast majority of these problems, as do cloud providers. I then ranted on a couple of the "problems" that containers and container orchestration "solve".

You're right tho - a bit of a ramble. Been a long day / on-call week :P I will think about how I could have been more eloquent for next time.

> [...] Ansible could be swapped for "any modern cloud configuration tool".

Deployment tool, not configuration tool. Ansible is awful for keeping environment configured, it was from ground up designed as somebody's deployment script.

Amen.

Ansible is great for fire and forget machines you don't need or care about. You need real config management with an audit trail when you go enterprise.

What's an example of an configurable aspect of some environment that Ansible handles awfully but that another configuration tool handles well?

Generally, all these tools are doing the same thing, i.e. running a bunch of commands on a bunch of servers. I haven't discovered what specific details tho indicate Ansible being less suitable (or unsuitable) for configuration management.

I could easily imagine that the 'modules' in Ansible aren't as robust as those in, e.g. Chef, at least for some platforms. But my experience so far having used both Chef and Ansible with Windows VMs (on VMWare) is that Ansible is much better.

> What's an example of an configurable aspect of some environment that Ansible handles awfully but that another configuration tool handles well?

Hosts that were temporarily down when Ansible tried to ship configuration. CFEngine or Puppet just apply the changes some time later without operator's intervention.

> Generally, all these tools are doing the same thing, i.e. running a bunch of commands on a bunch of servers.

So does Ant or Gradle. Something is different about the operation, though, because nobody uses Gradle to manage server configuration.

> I haven't discovered what specific details tho indicate Ansible being less suitable (or unsuitable) for configuration management.

The architecture and the protocol.

With Ansible you get synchronous sequence of operations that breaks on first problem (e.g. host is down for maintenance). This is OK for deployment, but not for ordering a configuration change, for which you don't care when it will be applied, as long as it's soon enough. Note that deployment and managing configuration are similar, but different things, and it's not uncommon to need to do both (although not at the same time).

Then the protocol. Ansible uses brain-dead idea of using the debugging/interactive channel (SSH) to configure things, including possibly SSH itself. One small mistake can cut off your all remote access at once. It's negligible risk if you only configure SSH daemon when deploying the server, but it increases quickly in the log-term.

Not to mention that managing SSH access properly (e.g. not ignoring warnings about hosts' public keys, but not just that) is much more tedious than keeping a CA, so SSH is a bad choice for unattended operations.

We use ansible and systemd with around 100 services in 3 continents and works well.

Cool. If any of it is opensource, please post the links in the reply, I'm curious.

+1 for systemd + ansible. That is exactly my setup on ubuntu 16.04 deployments.

I really like the CoreOS approach of using systemd for container supervision.

Reply on news.ycombinator.com