Comment by jbergknoff | Hacker News Reader

Comment by jbergknoff | Hacker News Reader

Great post. It's always good to see more examples of people putting these tools to work.

With that said, I consider getting the AMI id dynamically to be an anti-pattern which undermines the principles of infrastructure-as-code. Specifically, it introduces an implicit build variable "time of `terraform apply`", which is not tracked in version control. Happily, because of Terraform's design, this sort of thing mostly won't cause your infrastructure to drift into unexpected states (e.g. production instance 1 running AMI X and production instance 2 running AMI Y). Within an environment, things should be consistent, but your staging environment may run AMI X while production is running AMI Y, and you wouldn't know from looking at your Terraform definitions.

I previously wrote about similar ideas in the context of pinning dependency versions, where wildcards can and often do get you into bad states. https://jonathan.bergknoff.com/journal/always-pin-your-versi...

Interesting point. Here is the counter-argument.

With a Continuous Delivery Tool Like Go-CD (Not Continuous Integration) - you solve this problem with connected pipelines (value-streams.)

In this case - the ami-id is resolved dynamically, but the particular version comes from the pipeline, is entirely repeatable and traceable (and dare I say - immutable).

The scenario being that the first pipeline builds the ami, then stores the ami ID as a value/variable/text-file that is then passed on to the terraform apply pipeline. You can repeat the terraform apply with the same ami-id. You can run a new ami build, get a new terraform apply with the ami from that build.

With respect sir - I believe there is room for more nuance in your claim that 'dynamic ami-ids in terraform are an anti-pattern' - this is a solveable problem. This has been solved.

We just store the AMI in a terraform file itself, and update it as needed. I mean, if you are already updating a text file, might as well update a a text file called ami.tf with the new AMI ID and store that in source control as part of the build process. This ensures that the terraform plan is an easy to read source of truth for what is supposed to be up in production.

This is a really important point. Our company is using Terraform in a limited way, but Puppet is our primary automated configuration management tool. Similarly in Puppet, `ensure => latest,` on package resources isn't necessarily dangerous, but it can add a lot of confusion without intentional commits against the control repositories.

Puppet rules are typically applied constantly in short intervals. The installed version should always be the same everywhere, the latest available from the repository. The risk is more that you can get updates installed at inopportune times. Of course what's right is that the version change is not reflected in the configuration management, but this isn't normally a problem for minor version changes.

I think the default settings for puppet agents are typically 30 minutes if I'm not mistaken. I've inherited a bit of a DevOps mess with some 600-700 nodes in various states of management.

In previous positions, there was a great hew and cry that the run intervals were increased from 30 minutes to 60 minutes... eventually every four hours for production resources.

In my current position, production nodes are provisioned to run puppet once daily as a rule, triggered by cron jobs at a pseudorandom minute between 0200 and 0400 defined at server provisioning time.

How do these intervals compare to what you've seen?

Well I've seen many environments, and some only ran puppet manually whenever they would before had used ssh and made a change directly. Another allowed one group of admins to just continue to log in and make small config changes, then got alerts and diffs of those changes from running puppet in noop and incorporated those changes into their git after the fact.

It's a tool to be used however you see fit and what you describe sounds reasonable, but the most common (because default) setup I've seen is the 30 minute interval.

Which is why I would argue "time of last puppet run" is mostly fairly recent and consistent for all machines, but terraform apply is most often not run automatically, not even at daily intervals.

> The installed version should always be the same everywhere,

Agreed.

> the latest available from the repository.

Highly controversial statement. It depends on the policy of the pancake maintainer. I’ve seen too many subtle bugs introduced by changes in configuration file behavoriour, new defaults, etc.

I would say if you can guarantee “latest version” means “this version plus security patches” or the maintainer is absolutely pedantic about semantic versioning, have at it. Otherwise, consider the pros and cons of stability vs being up-to-date, and make a judgement call accordingly.

You should be mirroring repositories locally and pushing out new copies after they've been thoroughly tested if this is a concern to you. This is where tools like Katello/Red Hat Sattelite shine, you take snapshots of your upstream repositories and promote them through your various lifecycle environments to test the packages before they even hit your production systems.

AMIs in general are a pain in the ass in AWS, though. If you're using custom-built encrypted AMIs in multiple accounts, the easiest way is to build the AMI once in each account, which then leaves you with a different AMI ID in each account. So now your template with hardcoded ID that works in one account won't work in another.

See: https://github.com/hashicorp/packer/issues/4772

Since you're probably also rebuilding AMIs on a regular basis to keep up with upstream updates/patches, your best bet is to use the latest version of an AMI in your template and be sure that you're also pruning old/incompatible AMIs from accounts regularly.

Reply on news.ycombinator.com