Small-scale? Run a simple script. Large-scale? Use a network-hosted configuration (optimally read-only root and network booting so the entire system is known-good) to avoid the entire class of configuration drift / migration / state-accrual problems associated with the above.
I just see puppet as sort of trying to provide the latter and failing, resulting in a complex version of the former.
There are two things I want:
1. To describe the desired correct configuration as a directed, acyclic graph.
2. To have some automatic compare-and-repair mechanism regularly bring my systems to such a state.
If you don't have a good tool for those two requirements, you wind up reinventing it anyway.
Your small shell scripts start being littered with lots of checks for this and that file, if-thens and cases. Then they break when script 27 silently gets out of sync with script 42.
You realise one day that system configurations can be expressed as DAGs (possibly it's a new insight, perhaps you were reading ITIL documentation). And you begin to dream about a tool that can take a descriptive DAG and generate the correct shell scripts. You have now half reinvented puppet.
But the systems still get out of sync. So you start tinkering with a tool that periodically checks each system and reruns the correct script. Now you have to ensure that your scripts are all idempotent. All those if-thens and checks creep back in.
So now you begin to dream about a system that only generates the steps needed to close the gap between the DAG and the current state of the system.
Congratulations. You just reinvented the other half of puppet.
If you have already decided on your solution before you have analyzed the problem then there's nothing to discuss.
> 2. To have some automatic compare-and-repair mechanism regularly bring my systems to such a state.
Right. My fundamental point is that puppet-esque solutions only describe certain aspects of the system state, ultimately failing here and resulting in 'configuration drift'. Entire system images are far more elegant.
Most of the use cases I have seen for puppet-style stuff are legacy-situation based and only make sense within that context.
So what is the alternative to puppet-style solutions? Personally I use full system images on cloud and cluster solutions (corosync/pacemaker) to maintain server state.
You can see how I arrived at that solution.
> My fundamental point is that puppet-esque solutions only describe certain aspects of the system state, ultimately failing here and resulting in 'configuration drift'.
Particularly with runtime / process supervision. One of my pet peeves.
> Entire system images are far more elegant.
Yes and no. It really depends on what your cost/benefit tradeoffs are. I like system images for startup, but systems still drift from their initial configuration no matter how that configuration is established (DAG or blob).
Even with system images you'll need to detect divergence from ... what, exactly? Doing a byte-for-byte comparison is going to suck.
Do you just kill and relaunch periodically? I can see that being stochastically effective.
Personally no, but you easily could.
> It really depends on what your cost/benefit tradeoffs are.
Well, realistically, to facilitate automated deployment and testing of multi-machine systems you really do need to be operating at this level of abstraction. (ie. to declare your desired state). Once you reach this point, a configuration file for some version of a package within some node is really a distraction rather than a help; it should have been abstracted to some known-good state and taken out of the equation if you are to have any hope of staying sane. Many people use entire system images coupled with service monitoring as the de-facto segregation point for management purposes.
> Do you just kill and relaunch periodically? I can see that being stochastically effective.
While that's entirely possible and probably a reasonable approach in some cases, the corosync/pacemaker de-facto/spiritual approach is to detect issues with a given resource (roughly: 'service instance'), automatically destroy it and fail over to another instance thereof, and then potentially start another instance to replace it automatically on the same, or some other cluster node. To facilitate rapid failover, the master/slave paradigm allows you to have live backup nodes running and promote them as masters easily. Any type of hardware or software can be scripted as a managed resource. The type and frequency of resource health monitoring checks can be custom defined.
I still feel you're ultimately defining a DAG ("I want 20 web servers, 3 database servers and 2 load balancers") and relying on some compare-and-repair (health checks and replacement).
But you've moved your unit of management from packages etc to machines. I've previously argued that this is the key thing that will change web hosting economics. I was sorta wrong, but I can see now where you were coming from.
In an interview with MTV on his new formal collection, taglined 'S-eXpression', Nerd-DAGgy was asked what the secret of the hot new look was. In characteristic brevity, he replied "It's declarative."
-- This Season's Assembly, Fashion World, February 16, 2013.
Puppet-like systems have the same issue in that they don't attempt to specify the entire system, but at least with puppet if there is an issue due to system drift you can start with a clean base installation and re-run it.
You just uniquely name the environment, for example with a version number, and/or use a snapshot-capable datastore.
I guess you could just write a changelog, but that requires a large amount of self-discipline to ensure that the changelog exactly matches system state.
Definitely don't do this.
Let's say you discover some minor instability, and discover that it was introduced 8 months ago, but the changelog for that image was completely innocuous. Can you do anything other than throw out all 8 months' worth of images and try to work your way back to present via the changelogs?
If you write a test that can trigger the issue and replay the test against past images ("regression test") then you will identify exactly where the issue was introduced. The key thing is: don't have manually configured what-not in production. Keep it versioned, keep it solid, keep it known.
Puppet-like systems have the same issue...
Exactly. This is my point. They don't really deliver on the promise of decent automation, because they are inherently patchwork/partial-scope in approach.
The feeling that I want to write something quick or easy shouldn't depend on the size of my infrastructure, it should be easy either way -- 500 servers, 50, 1, doesn't matter.
That being said, our modules do need some patch-love for things like launchd and OS X groups, we've been a lot more Linux/Unix focused to date. (Patches are welcome of course!)
The resource model is still very puppet-like (state=absent/present, idempotence, etc), in that we have states and convergence, but I have tried in many places to not make this a programming language, and keep the amount of syntax down. A key focus was making the content readable, so you didn't have to think about dependency ordering and could tell by skimming the playbooks top to bottom exactly what they will do.
The other aspect is that many admins are ally do not like the bad habits associated with an image-level of abstraction, which leads to many hidden dependencies and configurations settings. Aa package-level manifest like Puppet, Chef, or Ansible enables much more of a complete explicit specification of the environment along with cross-machine dependencies. Such a description also allows easier reassembly of subsets of the system into different combinations.
The trend towards image-level abstractions does work for many as an alternative: Netflix for example avoids Puppet and just reassembles AMI images on each deploy, with auto discovery of cross machine dependencies at runtime via (eg) Zookeeper or other cluster facilities. But they have the discipline not to get into an image sprawl situation.
If they are often on the network, you can use occasional rsync, for example to the system partition either at boot or in the background. That way, boot is possible even if disconnected.
If they're not often on the network, and you are seriously considering doing this at all, I would reconsider which problem you are trying to solve.
Also: I don't know about anyone else here, but I would prefer not having a company manage my laptop. The use cases for such a scenario, in my view, are pretty hard to imagine.