Puppet vs Chef, Fight
devopsanywhere.blogspot.com
devopsanywhere.blogspot.com
Having been round the houses with cfengine, Chef, Puppet and more now I think these tools are, overall, a poor use of time. In general they're 99% used as config blasters: Yet Another Way To Put Files Onto A Computer.
Turns out, packaging systems already did that and have better dependency analysis.
This also helps to match the lifecycle of configuration management objects with that of the components they configure. I've seen far too many sites that had one big hairball of a Chef/Puppet repository that tried to service the needs of multiple conflicting application releases.
The final nail in the Chef coffin is that it encourages parameterisation of config files rather than configuration by convention, which is simpler and less prone to production-environment gotchas.
Everything else they do can be replaced by a very small shell script.
The one thing that keeps me from wanting to switch entirely to apt for deployment is its insane insistence that multiple versions of the same package may not be installed at once. This is (usually) fine for user-level stuff, but there are lots of legitimate reasons to want to keep multiple versions installed in a deployment situation.
What tools do you end up using? Some sort of custom apt repository plus some shell scripts? What does it look like?
So we do per-commit integration testing with "unstable", promote into acceptance & functional & regression testing (and showcases etc) with "testing" and then promote into "stable" for production. The actual promotion is done with some very small shell scripts off a CI server; the installation is managed via rundeck.
The configuration packages are per-node role dependencies and we try to keep granularity large - e.g. there's a configuration package for a core Java webapp that includes the application's own resources & nginx & jetty configuration bits.
For per-environment stuff (like database passwords and external integration endpoints) we just inject a (quite small) yml file to each server for the config packages to find. But configuration by convention is preferred so we also manage the DNS carefully by role (again, from rundeck) so that services are at well-known unqualified label names.
Finally, the per-environment spec itself (i.e. that the rundeck scripts look at) is in a cheesy cvs tree. I keep meaning to move that to git.
The base images themselves are stock virtual machine templates (AMIs for EC2 testing/dev) and the config packages & rundeck take it from there. The first step is to install a basic platform package that has lots of dependencies for our common tools, libs and needs, the the role-specific config package does the rest.
What do you recommend for managing the running state of processes, etc?
It's a whole "my house, my rules" strategy. If you play nice with the underlying platform, it'll play nice with you. The downside is that you do have two packaging models to build for. I know that abstracting this is one of the motivations for Chef et al. I have yet to see a usage where this was worth the opportunity cost.
We have since switch to Fabric http://fabfile.org, and are much happier.
My current workflow for managing systems does use Fabric however. I maintain configuration using a custom framework and work under git/hg -- Fabric tarballs the tip, syncs the tarball to several dozen servers, and then runs and my configuration module on each server. I did use a bash script previously though.
We are in the process of moving away from Capistrano to a system where code is automatically deployed when a new release is tagged. The deployment system is automatically managed with Puppet and everything is monitored with Nagios.
Fabric does what we need it to do for now for provisioning & deployment. And frankly you can make it do whatever you want. "Programmatic use of SSH" isn't something to be scoffed at; it's simple yes but it also lets you do whatever the hell you want very simply.
More info after slide 42 here: http://www.slideshare.net/ffunction/fabric-cuisine-and-watch...
The former are pull management, eg nodes request configuration. The latter are push management, eg main server pushes to nodes.
My main concern is that you did not point out your requirements. So you leaped from one technology to a different one because puppet was 'maximum pain'.
Maybe this means I should just fork up the cash for hosted Chef, but I don't see it as a 100$ a month value (I currently have 10 nodes and treat them all as ephemeral).
I do wonder why this is never brought up. Is it not seen as a red flag to anyone else that the core of Chef is built on Merb?
As for the core being built on merb. I don't really notice it. You'll probably never have a reason to go into the chef-server internals to change something. Besides that, it's not like it has to be serving 30k requests per second or anything.
Otherwise, the Issues I've run into with chef-server: it was a little annoying to get everything installed and playing together. Couchdb + solr are very temperamental. After everything was running and stabilized(1.9.1 bad!) I've had no issues with it.
Perhaps your issue is not tied to your ruby version: http://tickets.opscode.com/browse/CHEF-920
I ended up literally printing out the wiki. (And the wiki seemed to be in a state of pretty extreme flux and/or disagreement with what blog posts suggested was best practice.)
The business model of the companies promoting Puppet and Chef seems to be to charge for support and/or hosted services. Which is fine. But is it leading to abysmal documentation?
I wonder if the author knows of resources other than that book and the wiki.
While not brilliant, the Puppet docs seemed better and when added to the Pro Puppet book ( http://www.amazon.com/Pro-Puppet-James-Turnbull/dp/143023057... ) are more than adequate, so I just picked Puppet on that basis alone, and have not had any issues.
Another useful tool , in conjunction with Puppet/Chef is Blueprint which lets you build configurations from existing servers , http://devstructure.com/ .
This is one of the reasons we're moving Puppet Labs from being a support company to a product company, and not for hosted services.
If your bread and butter comes in from support, you have no incentive to actually make your product easier to use. There are plenty of enterprise-y software companies who make lots of money operating like this, but that's not the sort of company I want to work for.
I'm quite proud of the rapid improvement we've made on the Puppet Docs over the last year since we hired NickF, our most excellent tech writer:
I particularly like the solution focused docs he's done, as opposed to the dry reference material that presupposes a lot of knowledge.
http://docs.puppetlabs.com/learning/
Anyway, just wanted to point out that that's not our business model.
(Product Manager at Puppet Labs)
What it needs above everything else is a table contents that stays with you as you read. The documentation you have now only lists a TOC for the section you're in. And the TOC is tucked away in the right-hand corner and stays there when you scroll. So whenever I dive into your docs, I never get a sense of "where" I am in the whole big ball of string, and I get frustrated.
The first page is weirdly arranged. For example, it's got a big headline "Learning Puppet", then "Learn to use Puppet! New users: start here." Then there is a link to "Introduction and Index". I interpret this as being the link to the "new users" guide, and that the next two sections ("Part one", "Part two") are not related. But if you click on "Introduction and Index" you come to a new page which lists the exact same TOC, and has no section named "Introducion". It's confusing.
The "reference manual" and "guides" sections are similarly confusing because of the lack of a unified table of contents. The reference is particularly bad because I want to look up something alphabetically ("I want to manage users, so I bet it's called 'user'"), but the reference is arranged by topic.
I would add though that once you've gone through the basics, you find yourself primarily living in the Type Reference, which is arranged alphabetically.
http://docs.puppetlabs.com/references/stable/type.html
Always happy to get bug reports, feature requests and feedback on the docs...
Yes, that's were I ended up. I'm way beyond that level now, but I sometimes have to look something up in the docs, and always have to hunt down the URL to that particularly page.
The one persistent thread I've seen with "support-based models", notably Red Hat, is that ... the tools leave you requiring support, and yet the support still sucks (in large part because the support engineers have to use the same crappy tools you do).
I'm not going to pretend I've got the Free Software business model sorted out. But I have noticed that project goals and incentives do line up with results. Debian has a social contract, a constitution, and policy, and the tools to support these. From an ease-of-use / ease-of-maintenance perspective, Debian (and its derivatives) blow RPM-based systems out of the water. And to be clear: the distinction is much less the packaging formats (though that's part of it) and far more the use and management of these tools.
I think there's someone who died recently who made the end-user experience key. While his organization struggled at times, it's emerged in recent years as an absolute behemoth which suggests ultimately that the alignment he chose is better than that of a principle rival.
I'm not going to pretend we've got the Free Software business model perfectly sorted either.
We're producing proprietary software as well as developing/curating open source software projects, and there is a balancing act involved there.
Not to sound too bandwagon-jumpy (I've found the recent hyperbolic public outpouring of grief somewhat distasteful) but I care about user experience and documentation, and I've always believed that focusing on the end user will lead you in the right direction.
http://puppetlabs.com/blog/what-is-user-experience-in-puppet...
We're looking for more UX staff, and have an open UX Designer role right now actually...
http://puppetlabs.jobscore.com/jobs/puppetlabs/ux-designer/b...
What kind of a comparison (Debian vs. RPM-based) is this? Does dpkg(1), as opposed to rpm(8), really make all the difference in "ease-of-use"?
Of course, I understand what you mean, however, don't you think that if Debian really blew RHEL and SLES out of the water, its adoption in the enterprise server space would be a bit higher? You are kidding yourself if you pretend that the commercial distros are only used because they come with support, and also need support, being intentionally crippled.
Obviously, a meaningful comparison of Debian, RHEL and SLES does not fit into a HN comment, but as someone who has used all three, I would always choose Debian last, and not only because of the lack of commercial support.
(By the way, the main point of the support you get for RHEL and SLES is not access to a guy on the phone. The real value is the L3 support, i.e. bugfixing. As a Debian user who runs into a bug that cripples your system, your options are:
a) fix it yourself
b) pray to $DEITY that Debian developers fix it soon
c) try to expedite b) by harassing Debian developers in bugzilla, which is going to be as inefficient as you'd expect.
As a paying customer of Red Hat or SuSE, you are entitled to and will receive a fix for the bug ASAP, and although ASAP often translates to "a couple of weeks", it is still a hell of a lot better than what Debian can offer.)
I prefer to work with Chef much more than Puppet (as a user/developer), but I wouldn't want to be responsible for running Chef cluster itself. There are too many different elements to learn and take care of. Unfortunately official hosted service is very expensive, so there's no good alternative once you need many nodes.
When comparing Chef to puppet then Chef comes out as the pragmatist. Once up and running most common every-day tasks are less painful in chef. Apart from that its main advantage over puppet (to me) is that it allows to semi-sanely manage transient hosts (cloud/EC2) that enter/exit a cluster ad hoc. Puppet can also do that in theory, in practice you'd rather want to fork your eyes out with a spoon.
However, the pragmatism comes at a price: The chef-implementation is an absolute and unmitigated disaster.
You'll spend quite a bit of quality time initially to get the six dozen components to play ball and to fix up basics that shouldn't need fixing (i.e. you'll want to ensure that everything is under version control and not just the parts that chef deems worth versioning).
Over the first couple months you'll also see the odd server-crash while you figure out the peculiarities of your particular installation. Chef is very heavy on dependencies and the exact software-versions depend on when and how you install (pkgs vs source-code).
However, once you're over that hump and if you're not too worried about standing on the shoulders of 'a one-eyed amongst the blind' then the whole cookbooks/roles/runlists arrangement is quite comfortable to work with.
Just don't expect features like dry-run, real idempotency or clean dependency tracking that some would consider "basic" for such a tool. Also don't expect a security-model at all; to my knowledge all hosts that are authorized to talk to a chef-server can see all cookbooks and databags on that server.
If you care a lot about those latter minor quibbles then perhaps Puppet might be more your thing.
Puppet is conceptually much cleaner (night/day difference), which sadly and ironically is also its biggest drawback; they took it too far. Puppet made a bad decision early on by inventing their own language. This decision will be your personal ball on a chain for the lifetime of your puppet deployment.
But, is it really that bad? Well. Yes.
After the initial (steep) learning curve there's only a small plateau of reward before you begin to run into the more subtle issues. The most commonly heard complaints about the language are the ass-backwards class/variable-inheritance and the blurry (and effectively undocumented) divide between what should go into the manifests (sorta like chefs "cookbooks") and what into a storage layer called "extdata" (sorta somewhat like chefs "databags"). But rest assured, there's plenty more, I don't want to spoil it all at once here.
So, yes, you will hate puppet every time you have to make a complex change.
Yet for some it might still be worth it, here's some of my reasons: Once you finally have something up and running puppet feels much more predictable and "solid" than chef. You can actually dry-run and test (most) changes before rolling them out. Puppet will provide meaningful error messages in most situations (unlike the esoteric chef stack-traces). The puppet daemon is just that; one daemon (unlike the conglomerate of moving parts that comprises a chef deployment). Generally speaking there is much less "magic" in puppet than in chef. You will almost always know precisely what went wrong - a pretty important attribute that chef unfortunately doesn't share.
Oh, and no least: the puppet documentation is heads and shoulders above chef (although the latter has improved recently).
So, if you're in the market, good luck making your choice. I'm not making a recommendation here because, quite frankly, I wouldn't recommend either to anyone other than my worst enemy. ;-)
It seems to me like people complaining about Make being a poor programming language.... IT'S NOT FOR PROGRAMMING! It's for build sequencing. Every single program on your computer is a command-line-invocation away. Let Make handle incrementally executing external scripts based on timestamps, and let real programming languages do what they do best.
Isn't the point of Puppet to have an idempotent declarative language. And can't it execute arbitrary programs?
Yes and no. The main argument against puppet's dedicated DSL is that it's dedicated. I.e. you'll spend a lot of time learning a complex language for the sole purpose of running puppet. You cannot transfer that knowledge to anything else and unless managing puppet is your only job you'll frequently have to re-read the documentation to recall the more exotic idiosyncrasies.
It's a love/hate-thing really. In puppet you can express many things much more cleanly and elegantly than in chef. Other things you can't express at all or require you to resort to terrible hacks.
In chef-cookbooks you rarely need to resort to hacks. But at the cost of the chef-infrastructure itself being a pretty terrible hack...
What are some examples of things are really elegant to express in Puppet? What are some examples of things that you had to resort to terrible hacks for?
Please use personal, concrete examples, if you don't mind.
I've only got a couple of boxes to manage, but reading through the docs for the big three configuration management systems, I really was confused about the need for most of the stuff in the tutorials. I'd also be interested in concrete examples of where shell-scripts fail so badly.
This will assemble a hosts-file that contains all hosts from $group, on the hosts that include the class.
It's awesome and terrible at the same time because it demonstrates an Exported Resource (@@host, Host <<| |>>, awesome) and at the same time it shows how we had to resort to extdata for grouping (extlookup = terrible) and how opaque the puppet DSL is to the uninitiated.
How is this better than something like this:
gen-hosts config-file > /etc/hosts
See this gist I just made: https://gist.github.com/1274379
That script is (1) more understandable to a programmer, (2) unambiguous with respect to output, (3) idempotent, (4) super extensible.
I suspect your Puppet script is doing some other stuff, but I can't begin to mentally parse it...
EDIT: Just wanted to point out that I didn't try to make that script either correct or identical to your script. Just wanted to give an impression of the flow of things.
What would I be reinventing?
Again, I'm genuinely ignorant in this department. Really curious what I'm not understanding, but I just get the feeling that "worse is better" here.
I also looked at cfengine3, and I liked the language, although I am not a fan of the "promise/promiser" terminology, the concept is consistently applied and as a result the language grammar is much simpler than puppet's. It basically came down to not being able to really choose between puppet and cfengine3 without actually trying them, and since I cared less about identifying the optimal tool than just getting stuff done I selected puppet arbitrarily.
I rejected bcfg2 mainly because it seemed very server-oriented and although there was documentation for setting up the system there wasn't a lot of documentation about how to actually configure systems. I've revisited bcfg2 lately and the situation doesn't seem to have improved much. (eg I can't find a decent reference for the configuration language anywhere online. There's no equivalent to that puppet page that lists all the data types.) I also think the choice for XML as the configuration language is suboptimal but I could probably work with it.
So we went with puppet, and although I left the position several months later (for unrelated reasons) we were already hitting some of those issues you mentioned.
Incidentally, you don't even need the daemon to run puppet, so long as you're willing apply your own security model and distribution method (rsync over ssh, etc/).
Could it be you made the wrong design decisions both times? Sometimes, people try to do complex things in puppet, when they should using puppet just to push out that cronjob or init script, or rpm/apt package to handle the complexity.
Can you give more details like how many serves you have? Do you have a complex combination of hardware(physical + virtual) and operating system platforms?
Do you have a complex combination of apps on these platforms? Did you use the "environments" feature of puppet/chef? (Environments are totally broken in puppet) Did you use Dashboard/Foreman or Grill(chef only) and did that make things easier?
Did you migrate from the "hairball nest of Bash and Perl that holds your data center together" or did you build your infra from scratch with puppet/chef?
Sorry, that part was irony. I would of course recommend CM. I was just trying to express that the current crop of tools is nowhere near a level of maturity where you could recommend one or the other as a no-brainer.
Could it be you made the wrong design decisions both times?
Surely. More so with puppet than with chef as the puppet language offers slightly more rope to hang yourself on the foot. However, I'm quite confident we have maxed out both tools to about the best of their abilities now (short of the more esoteric extensions such as mcollective). There is of course always a next step, but I can't think of something that could potentially change my perception of their respective problem areas (which, as I outlined in my parent post, are pretty fundamental for each tool).
Can you give more details like how many serves you have?
Largest puppet deployment is ~70 hosts, largest chef is ~40 hosts.
Yes, we've had to migrate the puppetmaster to a pretty beefy host. Yes, chef's claim to scalability is a joke. We experimented with couch replication but quickly went back to running separate (beefy) chef-servers and sync'ing them via git.
A tangential reason for running separate puppetmaster/chef instances is the lack of a trustworthy security model in both. Extdata/databags do occasionally contain sensitive information, neither puppet nor chef accounts for that (don't get me started on "encrypted databags").
Do you have a complex combination of hardware(physical + virtual) and operating system platforms?
We run (only) debian on physical and cloud servers. For cloud deployments chef works better for us. For physical servers we're rooted in puppet but slowly leaning towards chef for uniformity. I wouldn't say our deployments are especially complex. And even if they were then that'd be no excuse for the tools that were created to manage precisely that kind of complexity.
Did you migrate from the "hairball nest of Bash and Perl that holds your data center together" or did you build your infra from scratch with puppet/chef?
The bigger clusters were built from scratch but we have chef/puppet clusters in various states of finesse.
Oh, this reminds me of one of the less obvious issues that I forgot in my initial rant: Updates.
You absolutely, positively want to either update your entire cluster (all puppet/chef clients) in lockstep with the server or not at all. The "backwards compatibility"-claims didn't work out for us when we tried.
In all seriousness (kinda), many thanks for the great write up on the current state of provisioning tools.
Yes, there's a gap that the current crop of CM tools don't address (and why we see new versions of capistrano clones with a bit of system management sprinkled in) but the three major tools right now - Puppet, Chef and CFengine are all at a state where they address 95% of the use cases for system management and automation.
http://blog.lusis.org/blog/2011/08/22/the-configuration-mana...
It stimulated a very interesting discussion in comments in which some of the leading lights from both communities (and there is some overlap) took part.
The fact of the matter is that X vs. Y is ENTIRELY subjective. It isn't a sporting event where there's a clear cut way to call a winner.
Some people like Puppet. Some people like Chef. Anyone trying to decide between ANY software - whether two competing client libraries or something as critical as configuration management owes it to themselves and the company to try all the options. The one that's best is the one that encourages you to adopt configuration management.
Not only is Bcfg2's explicit configuration structure unpleasant to work with, the heart of it is in XML. I tried to use Bcfg2 and was easily put off by the required structure. The fact that Bcfg2 stems from the science/math community was actually the only thing that kept from overlooking it completely.
Client-server model
Insufficient online documentation for what seems to be a verbose and somewhat limited configuration language.
The XML is a minor turnoff but for me it's more a question of what the language is doing. Eg:
<Package name='openssh-client'/>
<Package name='openssh-server'/>
<Package name='emacs'/>
<Package name='curl'/>
<Package name='rsync'/>
CFEngine spawned Puppet spawned Chef (rougly speaking)
I never got involved with BCfg2 or lcfg so I don't remember the history behind.
cfengine2 is a simple and effective mechanism for managing systems. However, cfengine2 has some issues. It has limited facilities for abstraction, resulting in a lot of code duplication. It has no built-in template engine, meaning that if you need to assemble a file based on various conditions you have to cobble it together with pattern matching and line-insertion. It's pretty easy to screw this up and lose idempotency. There's a 30-package-install limit per cfagent run, which fails silently. This is very annoying and requires some hacks to circumvent.
Also, in cfengine2 you have a configuration file that's basically <classname> : <list of hosts...>. If you approach more in terms of <hostname>: <list of classes>, it's a minor inconvenience.
Cfengine3 addresses some of these issues, but I haven't used it much so can't really give it a fair review.
So while the tools are roughly equivalent in capability, if you're new to configuration management in general I think you'll have a lot better success getting started with Puppet.
I also hold live webinars to help people get started with CFEngine 3. If you need any help, I invite you to register for one (http://cfengine-quickstart.eventbrite.com/)
And I'll try to tone my next post better so it comes across as eager rather than shameless.
Yours truly, Aleksey
Despite some cool things I've been able to do in Chef, Puppet is probably going to end up the winner, if for no other reason than it having much much better support for RHEL and clones (ie Scientific Linux). Chef was extremely unimpressive there in many ways, and anything related to deployment where the wiki includes the line "RPM installation via ELFF has been deprecated" is going to be a non-starter in the HPC world. (Or doesn't, anymore... any mention of RPM seems to have been scrubbed now.)
We're probably only going to use it on infrastructure/head nodes though: our compute nodes are provisioned on ramdisk statelessly using Warewulf, and are easier to reconfigure with a reboot/redeploy than via CM.
(edit to clarify)
Without chef-server using so called "chef-solo" lacks a lot of good things:
+ (remote) bootstrapping + distribution (like "little-chef"; non-official)
+ data bags (third party cookbook only; non-official)
+ search (e.g. on a central file system based data-base which gets dumped into a json that will be uploaded on bootstrap/update time (push))
and my own report from Configuration Management Summit last year: http://www.usenix.org/publications/login/2010-10/openpdfs/Co...
I haven't tried out scalr my self but I hear it described as an alternative to RightScale with 90% of the features at 10% of the cost.
If anyone here has used scalr please enlighten us of on its current state.
Why? I prefer to simply write my own bash deployment scripts. Seems easier.
For keeping track of os-level packages and the like, I find Puppet much better than something I could build myself. It doesn't just take care of the initial setup, but also handles incremental changes. And it does so in a largely declarative language. It all guarantees that the entire installation is documented and replicable.
Guaranteeing the entire installation is "documented and replicable" is huge. Chef/Puppet can become a large part of you disaster recovery plan and increases the "bus number" in ops.
The value in this is immediately apparent after the first time you have had to rebuild a server that you inherited.
I used slack[1] for a short while, but then started my own configuration management framework for Perl[2]. I wanted to have templating and better reporting, and also had hopes of keeping a very slim rulesest -- forcing any extraneous functions to per-instance (bash|perl|python|.*) scripts. I need to get back to developing and testing it though.
The reason is that by encoding your configuration in a metalanguage such as Puppet's, you have described your environment independent of the actual box. If the box goes down and you need to swap in a new one, you just run Puppet again, and it will magically turn into the old box.
If you need to replace the box with faster hardware, for example, just run Puppet on the new box, and it will (if you have done your job properly) become identical to the new one. Suddenly need 10 boxes to handle a load spike? Just boot up 10 cloud VMs and point them at the Puppet server.
Since these systems are modular and lightweight, you can easily describe hundreds of vastly different boxes using a single configuration.
If you've done your job right with the shell script, the same will be true. The problem is that if you have more than N nodes, "doing the job right" becomes more difficult. However N is quite obviously greater than one.
From what I can tell both Puppet and Chef seem a little over-engineered, so I feel there's a niche for a simple (solo only) tool out there. By using something like Node it would be possible to a) run with a much smaller memory footprint b) have a more event driven architecture and c) run multiple downloads or other setup related tasks in parallel.
Additionally, javascript is NOT more widely accessible - if anything, learning non-DOM-manipulated javascript is INCREDIBLY painful for a beginner.
Remember, these tools are for system administration and configuration management, not for writing applications.
Your point about them being over engineered is much more interesting than you just thinking we need yet another system but in a different language. And javascript is certainly not the only way to achieve the goals you list. Just seems like an attempt to start a language war honestly.
I agree that JavaScript isn't the only way to achieve this. It's just that JavaScript is a language I like, so I thought I'd find out if there were others who shared my views. Perhaps I should have expanded more on things in my first comment. That being said, I'm pretty surprised by the negativity in the comments and downvoting.
That's a separate issue from what the system itself is written in...
The downvoting is probably over confusion, shared by myself, about what the language used to write the system itself has to do with the merits of the system.
(EDIT: perhaps having the ability to define recipes using a JSON only format would work as well)
It might start as a lightweight rival and yet it will continue to grow in the same trajectory as its rival eventually.
I would recommend more people to read, analyze, and understand the history and nature of software.