Augeas – A Configuration API
augeas.net
augeas.net
Nowadays, though, I primarily use Ansible and its core "lineinfile" module gives me pretty much the same end result.
Then later when you want to automate generating bits of it, you can write simple Python (or whatever) scripts to transform the data. If you want the generated stuff to be less obvious or harder to change (just to stop people who can't help poking configs in production), write it in binary format.
(1) https://developers.google.com/protocol-buffers/docs/referenc... (2) http://stackoverflow.com/questions/18873924/what-does-the-pr...
But when developers go to write their own configuration format they often lean towards a custom syntax so that the end-user, who is expected to be a human being with a text editor, has a more comfortable working environment. And the resulting syntax is write-only, with comments, syntax sugars, etc. that would make it hard to impossible to automate data manipulations and also emit a document that resembled the original.
Tooling like this carves back a mix of both worlds - because it's just operating on the parse tree, it doesn't convey the final meaning of the data as it would be used by the thing being configured, just the "look" of it as a human would read it - but that's still enough to get an automation job done, for most kinds of job.
If you are a developer who wants to ease this job for your own configuration formats, of course, there are plenty of ways to go about this - provide more config formats, or your own extracted parse tree, a scripting API, or the ability to change parameters live. But for pretty much any solution, there is an engineering tax to be paid, and often a new surface area for security risk. So nobody goes about this stuff the same way, and many projects just never really solve it. Hence we end up with a third party like Augeas that comes in to fill the gaps.
My own policy tends to be to start with an XML format and then add a custom syntax to it later as necessary. The initial parsing job becomes straightforward with little chance of catastrophic error, and subsequent custom syntax can emit XML, so you get a twofer on the development time. The downside is basically that XML is a pretty big dependency and nobody is happy with it as a primary UI.
"Changing" goes beyond "parsing and generating"; e.g. ordering, comments and formatting are retained.
The alternative would be search-and-replace style edits (not robust), custom implementations for every format (hard work to get right, especially without Augeas' infrastructure in place), or being limited to templating entire files.
The list of supported file formats is quite extensive: http://augeas.net/stock_lenses.html (note on naming: lens is a well-known theoretical concept)
It's sad that Augeas is necessary though -- in an ideal world, developers wouldn't come up with custom ad-hoc file formats or abuse 'sh' and /etc would be empty by default.
If you can turn the files into parse trees, manipulate them semantically, and then turn them back into files, that rocks. For instance, right now I am working with a database that has to have a lot of things configured -- one of them is how big the buffers should be, which is a function of how much RAM you have, which in turn depends on the kind of AWS instance my product runs on.
With a smart configuration API I could take a configuration file supplied by the user and set the buffers to the right size really easily.
Which has nothing to do with the configuration format itself, it only needs some knowledge when building config files. It can be achieved as easily with a simple template (that doesn't understand the config it produces!), only supplied with information local to the system the config is going to work under. The easiest way is to build the config from a template on the system it's going to run, but masterful Puppet with its facts database can do it on the side of commanding server as well.
If the system does understand the configuration file, then you can get the full value out of having a "fact database" that describes the system.
Or because I put appropriate constraints on data filling the template. Integer fields are the easiest format to handle.
Generating configs with template systems is surprisingly strudy in real world scenarios, where at least half-competent programmer provides templates for at least half-competent sysadmin to fill.
> If the system does understand the configuration file, then you can get the full value out of having a "fact database" that describes the system.
The problem is that this "full value" is not that much bigger, but the effort in understanding config file formats is quite large.
Precisely. CFEngine (3.x, but also venerable 2.x) does it the former way, but there are templating engines (Template::Toolkit for Perl, on which I have built my cfgen, and, less convenient to control, Jinja2 for Python, which is the base for my current work with configs) that make it easy to do the latter way.
CFEngine 3.x adopted also templating at some point, I believe (I remember second-hand communication with CFE engineers that said our (my) templating gave them some ideas, but it was long ago and may be invalid anyway), but since I haven't used CFE 3.x, I can't tell much.
Apart from Augeas, I haven't seen the idea of working with configs that had such a solid foundation. It's such a shame the gains from having a semantically-valid configuration editing are so small against the cost of developing and understanding the grammar of config syntax compared to generating the said config from a template (at least that's what I think, that the gains are so small). I certainly would love to use Augeas-like tool in my daily work.
Using diff/patch and things like Ansible's lineinfile can apply the change, but the result is not guaranteed to be semantically or even syntactically valid.
With Augeas, it is going to be valid.
Also the big promise with a tool like this is that changes are bidirectional. See the Boomerang project for more on this. http://www.seas.upenn.edu/~harmony/
My own experience with Augeas has been mixed. It's quite prickly and uninformative in use and probably unintuitive to many. For example augtool by default handles the entire tree of config files. So if you just want to change maybe /etc/fstab you might encounter errors on files that you don't care about. Also I've often found that the lenses aren't quite complete, at least in standard distributions. For instance, the Centos6 Augeas's (version 1.0.0) xml lens didn't handle PCDATA or the DOCTYPE declaration.
Luckily its improving and more examples are becoming available.
So say you don't really care what's in /etc/ssh/sshd_config, but want to make sure it doesn't have passwords enabled. With puppet you could:
augeas { sshd_config: context => "/files/etc/ssh/sshd_config", changes => [ "set PasswordAuthentication no"], notify => Service[ssh] }
So basically augeas provides set, unset, match, remove, append, and related functionality for pretty much all configuration types. Then you just each whatever tool you are using (like puppet) about augeas and you suddenly can easily manage 100s of file types.