Introducing Consul Template | Hacker News Reader

Introducing Consul Template

hashicorp.com

101 pointsPerceptes16 comments

confd author here.

I'm glad to see this idea validated, and I think consul-template looks pretty awesome and well integrated with consul. consul-template takes advantage of consul specific features such as the consul services API and datacenters, which is really great for consul users. These are things I did not want to add and maintain in confd, since I'm supporting consul, etcd, and environment variables as backends I felt those would be one off features, which really belong in a consul focused tool like this.

Some will ask why did Hashicorp not just contribute to confd? Well they did. They added initial support for consul to confd almost a year ago and have been very helpful with my questions around consul ever since. The contributions go both ways too. I've been a contributor to Hashicorp's Packer project -- I added Ansible and GCE support and various bug fixes.

This is just how opensource works. I'm actually pretty happy to see one of my projects get so much attention and now inspire another project. That's the whole point right?

It's neat! Where I work we are about to use Consul and this is great news for us as we were looking for something like Consul Template.

I really like that they are simple Go templates :)

This looks great and glad HashiCorp keeps releasing Opensource software!

Anyone have a good end to end example of Consul, Docker, Service Discovery? Nginx example is good to get started, but just need some of the basics to point in the right direction.

> Anyone have a good end to end example of Consul, Docker, Service Discovery?

There are a few floating around. Here's mine: https://github.com/kanzure/docker-basenode

Here's something useful for end-to-end: https://github.com/hashicorp/consul-haproxy

I am not so sure that every possible update can be performed with those go templates, so I wrote a python version of confd (pyconfd) that uses polling and python plugins that do whatever they please: https://github.com/kanzure/pyconfd

What would be really neat is some way of having consul initiate an event instead of having pyconfd (or confd for that matter) sleep and check every n seconds. This may already be implemented in consul upstream but I haven't been paying very close attention...

Edit: dry run is very neat :-)

Consul supports long polling. This is what Consul Template uses. With long polling, the changes are edge triggered so there is very little delay.

We also compile and read the template once to determine only the keys you care about from the template to determine what Consul services/keys to look for, so we're not requesting ANY changes.

Again not to take anything away from consul-template or pyconfd, but confd supports "real-time" updates from both consul and etcd, the watch support was added about 2 weeks ago on the master branch.

confd also has dry run support: https://github.com/kelseyhightower/confd/blob/master/docs/no... It was apart of the initial release of confd.

These distributed K/V stores are designed to offer a minimal feature set, and the watch support is a perfect way to deliver events to clients. While HTTP call backs would be nice, I think you have to draw the line somewhere. Since these types of systems, consul or etcd, sit at a critical point in the infrastructure lets keep them lean and focused. But that's just my opinion.

Hey that watch stuff is pretty neat, I might have to switch back in some of my containers :-)

https://github.com/kelseyhightower/confd/commit/16d6b347c3d3...

I also wrote something (before confd but I've yet to see anything that replicates it) for treating etcd like a filesystem and doing edge-triggered updates. You form directory structures, and those long keys get mapped to files on disk with their value as the file contents.

What really bugs me about these solutions -- confd, consul templates -- is that managing templated configuration files isn't as quick, but has been trivially possible with normal configuration management tools since their inception. When I think of managing HAProxy configurations, for example, I don't see Chef as a bottleneck. I can parallel SSH and run Chef on those hosts in 20 - 30 seconds. I don't care about the configuration being pristine, either.. because that's what backend health checks are for. That extends to a lot of their examples, but it's one of the ones that stuck out to me.

The things that I think are hard, or harder, are small bits of data, like database credentials, etc.... where people absolutely don't want to be making an HTTP call for every single thing, or don't want to define a template for hundreds of individual files that might be needed. That's a real bottleneck - you need something on top of these solutions to manage that. So for every file you care about being updated fast... you still need to manage a corresponding file.

I don't know necessarily how to solve that -- I treat K/V pairs as file name/contents pairs in my tool and don't think twice -- but it definitely feels a bit wrong.

Hashicorp continues to kickass.

However, I wonder why this isn't built into consul directly?

For scenarios where I felt etcd/gconf/consul is overkill I wrote goconfd (https://github.com/niko/goconfd). Features:

* reads config from a json file * GET requests with deep links into the json tree * POST request with template as body and deep links (using the go template engine, too) * long polling (response has to be triggered explicitly by a PUT request)

I also made a simple docker container running HAProxy - configured by goconfd - to orchestrate docker containers.

It seems like there are a lot of similar competing projects in the service config space now. I was interested in trying this out last year: https://github.com/coderanger/etcd-chef

I wrote a script to auto-reload HA_proxy if nodes change in consul, now I'm glad I can now retire it and switch to this!

Probably the biggest feature over Confd is the ability to run arbitrary commands when a template update completes.

Not to take anything away from consul-template, but that has been a feature of confd for a while:

https://github.com/kelseyhightower/confd/blob/master/docs/te...

My bad, I misspoke.

Reply on news.ycombinator.com