Using Puppet to Automatically Configure New EC2 Instances | Hacker News Reader

Using Puppet to Automatically Configure New EC2 Instances

codelord.net

51 pointsabyx17 comments

I've recently done a project with puppet that turned out horribly. I'll share what I discovered:

* Puppet's error messages are next to useless. Sometimes it can only tell you which file had an error, and not which line number. Sometimes you'll be left wondering where in the many files that make up a complex project there was an error.

* The only way to actually test the correctness of your puppet scripts is to deploy on real servers. Your staging environment has to be identical to production if you don't want any nasty surprises.

* Despite being written in ruby, the design doesn't follow ruby's philosophies or best practices. The config language is not dynamic or ruby-ish. Compared to ruby, you get code blowout when you can't use DRY and have to unroll loops to fit inside the bounds of what the config language can do.

* Simple things are hard. You have to write your own modules or use 3rd party modules for what I consider simple things, such as editing an existing system config file to add two lines in the middle if they don't already exist.

* There's a steep learning curve because a large project is like spaghetti code where you aren't quite sure which file to find what you're looking for. It is very easy to create dependency loops, and hard to find them, and even harder to resolve them. I ended up generating the dependency graph to a PNG file and eyeballing it, several times per day.

I got the impression that puppet is ideal for getting a "PHP+MYSQL" server up and running that requires almost no deviation from the operating system defaults. If your setup is a more complex one I'd stay well away.

I for one, appreciate a pretty succinct article on setting up Puppet. Having used Chef briefly in the past (and now getting ready to use Chef - or puppet - across a brand new infrastructure setup), I came across the lack of (or just bad) documentation for Chef many, many times.

I've not used chef, but I'm in the process of setting up a new infrastructure using puppet and I've found the documentation to be pretty good.

I've never used puppet before, but in an EC2 context, I'm puzzled about why one would want to use on-the-fly configuration instead of fully setting up AMIs to serve specific roles.

At first blush, future-proofing against lower-level AMI changes and maintaining instance role flexibility don't seem like a good tradeoff against increased startup time in most cases.

We're just starting off with Puppet and EC2, but in general my current feeling is that I'd rather postpone maintaining images as long as I can. This is because a) I don't trust myself to keep these up-to-date, meaning I'll need configuration once the instance is up b) The cost of making a change is tiny with puppet. If you've got images you have to save them again after making the changes. That extra step currently has no value for me, when setting up a machine takes 2 minutes (which is fast enough for us)

If anyone here could discuss the pros and cons of this compared to chef for server configuration i would be very interested.

I often spend way more time than i expect to when configuring a new server, and have recently been very interested in ways to speed that up.

Initially, it won't speed you up. Developing the right provisioning formula requires some investment of time but I think it's time you get back many times over because you don't need to follow a bunch of manual steps and making changes to the infrastructure is easier when it's all code. I prefer chef simply because it's easy to develop customizations; the DSL is just ruby. I have a pretty non-standard application stack but automating the provisioning is pretty straight forward. I can bring up the whole thing from bare machine images in minutes. Check out http://wiki.opscode.com/display/chef/Launch+Cloud+Instances+...

I don't know if there's a puppet equivalent to chef-solo but using it with vagrant and VirtualBox is my favorite way to develop chef recipes. I did short presentation about this a few months ago, see https://docs.google.com/present/edit?id=0AVIsoyl05MRmZGNwY2N...

BTW, the OP talks about needing a separate security group for the puppets but last I checked, the default security group on EC2 makes the machines you own visible to each other and nobody else's that you haven't explicitly permitted. So I think all of that was unnecessary.

The equivalent to chef-solo in puppet is "puppet apply". It's very handy for developing puppet recipes on an offline virtual machine.

A Puppet provisioner was recently committed to Vagrant.

The huge advantage to Chef is that it's just ruby, where as Puppet has it's own syntax for recipes.

Another light-weight alternative for configuring new servers is Babushka: http://babushka.me/

Puppet (as of 2.6) has a pure Ruby DSL. Puppet and Chef are rapidly converging on the same feature set, at this point its like arguing .rpm vs .deb. There are certainly differences and if you spend a ton of time with both tools you will have a favorite, but fundamentally they both serve the same purpose and will enable you to get the job done.

It's amusing how every article about Puppet gets trolled by the Chef group. Is there ever a useful technology that doesn't inspire holy wars?

No. I don't believe there ever can be. For a specific technology choice to become useful -- and more importantly to remain useful -- I think it needs strong competition.

I'm going to sound like a broken record but I don't understand why bcfg2 isn't more popular.

If anything, I'm going to 'port' this post to an bcfg2 example.

This is super useful thanks. I'm mystified why puppet lacks substantive ec2 documentation on their website. I would have thought this was a major use case.

I wrote a pretty detailed piece on doing exactly this last summer, hopefully soon iClassify will be replaced by Puppet Dashboard, but for now I run both.

http://onehub.com/blog/posts/coordinating-the-onehub-cloud/

I like his way of doing the auto signing. I wrote about the topic at http://www.ibm.com/developerworks/linux/library/l-migrate2cl... and just had the new instance run the manifest locally.

The other option is to have the instance sign its own key, much the same way I have the instance set itself up in Cacti.

Reply on news.ycombinator.com