Terraforming 1Password
blog.agilebits.com
blog.agilebits.com
With that said, I consider getting the AMI id dynamically to be an anti-pattern which undermines the principles of infrastructure-as-code. Specifically, it introduces an implicit build variable "time of `terraform apply`", which is not tracked in version control. Happily, because of Terraform's design, this sort of thing mostly won't cause your infrastructure to drift into unexpected states (e.g. production instance 1 running AMI X and production instance 2 running AMI Y). Within an environment, things should be consistent, but your staging environment may run AMI X while production is running AMI Y, and you wouldn't know from looking at your Terraform definitions.
I previously wrote about similar ideas in the context of pinning dependency versions, where wildcards can and often do get you into bad states. https://jonathan.bergknoff.com/journal/always-pin-your-versi...
In previous positions, there was a great hew and cry that the run intervals were increased from 30 minutes to 60 minutes... eventually every four hours for production resources.
In my current position, production nodes are provisioned to run puppet once daily as a rule, triggered by cron jobs at a pseudorandom minute between 0200 and 0400 defined at server provisioning time.
How do these intervals compare to what you've seen?
It's a tool to be used however you see fit and what you describe sounds reasonable, but the most common (because default) setup I've seen is the 30 minute interval.
Which is why I would argue "time of last puppet run" is mostly fairly recent and consistent for all machines, but terraform apply is most often not run automatically, not even at daily intervals.
Agreed.
> the latest available from the repository.
Highly controversial statement. It depends on the policy of the pancake maintainer. I’ve seen too many subtle bugs introduced by changes in configuration file behavoriour, new defaults, etc.
I would say if you can guarantee “latest version” means “this version plus security patches” or the maintainer is absolutely pedantic about semantic versioning, have at it. Otherwise, consider the pros and cons of stability vs being up-to-date, and make a judgement call accordingly.
See: https://github.com/hashicorp/packer/issues/4772
Since you're probably also rebuilding AMIs on a regular basis to keep up with upstream updates/patches, your best bet is to use the latest version of an AMI in your template and be sure that you're also pruning old/incompatible AMIs from accounts regularly.
With a Continuous Delivery Tool Like Go-CD (Not Continuous Integration) - you solve this problem with connected pipelines (value-streams.)
In this case - the ami-id is resolved dynamically, but the particular version comes from the pipeline, is entirely repeatable and traceable (and dare I say - immutable).
The scenario being that the first pipeline builds the ami, then stores the ami ID as a value/variable/text-file that is then passed on to the terraform apply pipeline. You can repeat the terraform apply with the same ami-id. You can run a new ami build, get a new terraform apply with the ami from that build.
With respect sir - I believe there is room for more nuance in your claim that 'dynamic ami-ids in terraform are an anti-pattern' - this is a solveable problem. This has been solved.
CloudFormation has supported YAML for at least a year or two now, and it’s leagues more readable and compact, not to mention maintainable—you can even add comments to your code with YAML (something that is impossible with the old JSON format).
I’ve spent a lot of time working between the two, and while Terraform does have a lot to offer and is often a very valid option, CloudFormation has its virtues, especially the fact that almost all bleeding edge AWS features are first available to be managed via CloudFormation (sometimes with weeks or months of lead time), and many bugs can be more readily ironed out with AWS support (assuming you have it).
Again, not saying don’t use Terraform, just that I don’t think the decision is quite as black and white as this blog post seems to make it.
Recently I've come around to using Troposphere [1] to write cloudformation templates, it's actually very pleasant to use. You just write your infrastructure in python, and it will generate you a template at the end. The developers seem to respond quickly to changes AWS make to the CF templating language too.
YAML, despite its warts, is much more readable and maintainable for CF templates than JSON, particularly when you are doing non-trivial things and need to use a lot of intrinsic functions and string manipulation. Or you want to put comments in your template.
I've worked on huge JSON and huge YAML CF templates (I'm talking templates that are thousands of lines long--in YAML). YAML is without a doubt easier to maintain.
I can't recommend any CF libraries for generating templates either, unless you want to wait around for new CloudFormation features to be implemented or suffer from half-broken existing implementations (or waste time hunting down bugs and submitting patches). Sometimes it makes sense to use a template engine like Jinja or ERB. But I'd stay away from libraries that generate CF templates--they're mired with missing edge cases and they're mostly an unnecessary dependency.
What hasn't been helpful is what others here and elsewhere have reminded me time and time again... "JSON is a subset of yaml"
http://yaml.org/spec/1.2/spec.html
They provide a huge amount of examples, and YAML is actually I think a lot more powerful than most folks realize.
Terraform supports it, CloudFormation doesn't...
Yes, this has been known to happen, here is one example: https://stackoverflow.com/a/42142791/2518355 - this AWS feature was added Sep 21 2016, Terraform resource released support on May 11 2017, CloudFormation resource released support on Jun 6 2017.
However, several points should be added to this comparison:
- CloudFormation supports Custom Resources so you can always implement new AWS features yourself with a few lines of JavaScript, without waiting for the official resource implementation to be published. - CloudFormation resources have extensive documentation, and are officially maintained + supported by AWS.
There are also decent tools atop CloudFormation for using an actual programming language instead of whatever HCL wants you to think it is. The Cfer[0] project that I contribute to (and use as the underpinnings for the Auster[1] cloud workflow tooling) is the thinnest possible wrapper around CloudFormation that we could come up with. But it just by existing lets us do things like...y'know...if statements and loops...without breaking our backs. Code reuse is better than whatevery module reuse, too; I wrote a gem, which this reminds me that I need to open source, that lets me roll out a standardized three-tier network of varying size (important because AWS tends to work best with /24 subnets but hey, you might want more than three of them per tier) without really thinking about it too hard.
(Back when Terraform was new, I tried writing a halfway decent DSL on top of it; turns out "it's really just JSON under the hood" was untrue and no testing had been done on that path. I assume it has since. But HCL still exists, and HCL is still pretty awful.)
I would say not to use Terraform if you're using only AWS, because Terraform has a nice habit of hosing your state when you look at it funny and I've had it literally regress to the point of making states in version X unreadable in version X+1. But those issues are separate from the clunky DSL.
I've found preprocessing CloudFormation YAML using a standard template language (e.g., ERB) to be the thinnest possible wrapper around CloudFormation for providing if statements and loops without breaking our backs. Cfer looks nice and lean, but it still adds another domain-specific language on top of the stack while ERB is part of Ruby's existing standard library.
Granted, using an existing template preprocessor or a lightweight DSL can both work well and I think it's largely a matter of preference as to which feels thinner/easier to work with.
I agree with your assessment of HCL, a range of tooling choices for CloudFormation are largely possible because CloudFormation runs on standard JSON/YAML documents rather than the domain-specific, vendor-specific Hashicorp Configuration Language that lacks as robust tooling and support across languages/IDEs.
Objects just work way better for reuse and aggregation than text, IME.
Which seem to mostly revolve around AWS vendor lock-in. Which, you know, can be a valid business decision - accepting vendor-exclusive tooling for better vendor support to try and simplify initial deployment and lower near-term, initial ramp-up costs - but it has its downsides too, like losing control of costs over the long term and making advanced work (like multi-cloud deployments to improve reliability) more fragile, difficult, and sometimes de-facto impossible.
Seems to me like the question of CloudFormation vs. Terraform has more to do with whether the business makes a strategic decision to allow for vendor lock-in or not, rather than mostly aesthetic discussions over the merits of YAML vs HCL.
While I agree that CloudFormation is a vendor-specific tool which makes Terraform the obvious choice to avoid vendor lock-in more generally, in this particular case, the author is fully entrenched in the AWS ecosystem without any hint of desire to avoid lock-in, and gave as reason #1 for his decision a mostly aesthetic discussion over the merits of YAML vs HCL.
* troposphere (Python): https://github.com/cloudtools/troposphere
* cloudform (JavaScript, TypeScript): https://github.com/bright/cloudform
Driving infrastructure with a general purpose programming language actually works pretty well in practice.
As things have developed, I've come to question infrastructure-specific external DSLs, like Salt, Terraform, even Chef to some extent. Infrastructure is not any less programmable than payments, machine learning, graphics, app servers. Why can't the tooling be provided as libraries, consumable from ordinary programming languages?
The video is a bit slow because it's meant to be instructional. There are other videos on the same page which show how the model (graph) can be viewed logically instead of isometrically (what we call spatially).
I'm the CTO, happy to answer questions.
I think that Tufte needs to make a resurgence with this generation of designers.
You're correct that 2D makes for greater readability and information density.
Is the readability just due to the size that the images get reproduced at? If that's the case, it looks like the text isn't great on the "standard" size people reproduce the graphic in (landing page of your site), but it looks nice inside the editor (second section on your site, very crisp). Optimizing it for size, or scaling up the representation somehow would make this a lot more pleasant to look at for me.
Nobody else in this whole thread seems to be complaining about state management. I think it's insane that Terraform encodes where a given resource is in your filesystem / module hierarchy into the state JSON structure. (Unless something has changed since I last looked,) if you want to move things around in your .tf source, terraform can only apply that by tearing down the old resource and recreating it.
For our large setup, in order to adopt Terraform, we've had to spend a ton of time upfront thinking very hard about how all of our .tf sources are going to look, and it's delayed our deployment by months.
The key takeaway for me was "we really only need to consider the nodes that map to the physical resources of our infrastructure when we are planning our state surgery. This means we can ignore all of the nodes that correspond to data sources, variables, and providers."
So after a refactor, this is what I do now: (1) run plan to get the names of everything terraform wants to delete and recreate; (2) pair all the resource nodes manually and translate them to state mv commands; (3) re-run plan and verify that terraform is now convinced there is nothing to do.
It would be nice if terraform could do this for me, of course, but I find that it is generally possible to avoid delete and recreate if all I've done is a refactoring.
Also, Terraform state on the file system? Do you have the luxury of solo development without the need for a Terraform remote backend?
That is, it's about the file system layout of the .tf tree, not the state.
it does suck to have to copy the contents of an s3 bucket just to move a bucket inside of a template tho, for sure
The whole point of infrastructure-as-code should be the ability to spin up environments on-demand, do work, and then spin down. There's no reason to spin up an always-on shared development environment when developers can easily spin up their own environment when they need to do testing and kill it when they're done. Most development tasks can be tested in a single AZ, let alone region. Similarly, QA shouldn't need an always-on, 3-region setup.
I'd wager that they'd have far more issues from sharing a development environment between developers who are all pushing unreleased code to the same environment than they'd have from differences with the production environment. It should be pretty easy for a developer to spin up a simplified environment on demand from a branch or specific commit to do their testing. As a bonus, anyone reviewing code can spin up a similar environment from a pull request to ensure that code does what they think it does.
Infrastructure as code is a good first step. Embracing the freedom you get from that reproduceability is the next step.
It is does require extra time/effort though. We deployed 1password.ca and 1password.eu just a few months ago and never really got to that.
The only thing we "optimize" at the moment is the smaller number and size of EC2 instances in non-production environments.
Always-on non-production environments are, to my mind, a vestige of the time where you had physical servers that needed to be provisioned by a person and cost roughly the same amount when switched off. Or at least the time where OPS built and maintained each AWS VM by hand. On-demand just offers so much more flexibility...stuff like the example from my response to a sibling comment--you should be able to type a single command and spin up an environment from a pull request any time a code reviewer wants to do testing to ensure that what code looks like it does is what it actually does.
The more you can leverage the work you've done to get to where you are, the more you can drive down your AWS costs while giving greater flexibility and isolation to your non-prod workload.
To respond to some specific items mentioned in this post:
- "Terraform has a more straightforward and powerful language (HCL) that makes it easier to write and review code."
It's easy to pipe configuration through your favorite 'straightforward and powerful' templating language of choice to generate the stack template used by CloudFormation. I definitely wouldn't use CloudFormation at all without an extra preprocessing step of some sort.
- "Terraform has another gem of a feature that we rely on: terraform plan. It allows us to visualize the changes that will happen to the environment without performing them."
CloudFormation has a similar feature called "Change Sets" (released in March 2016).
Finally, two more things to note:
- CloudFormation supports "Custom Resources", which allow you to write Lambda-function scripts to perform any custom operation you want (e.g., interact with third-party APIs, or support new AWS resources that don't yet have an official CloudFormation resource implementation). There's not as extensive a library of providers Terraform supports, but you can often find open-source providers to fill the gaps on AWS resource until an official implementation is released. And if not, you can just write a quick and dirty implementation yourself. Terraform supports a similar feature in "Custom Providers", but they are Go-only as opposed to the various supports Lambda-language runtimes.
- CloudFormation and all official AWS-resource implementations not only have excellent public documentation but are officially supported by AWS Support, which is a big deal if you're deploying something with any degree of complexity or cost/risk.
As for Terraform's strengths: its extensive set of resource providers beyond AWS ecosystem can't be beat. Also, it has great support for mapping existing infrastructure resources, which is a feature still sorely lacking in CloudFormation. (I'm surprised the author of this post didn't take advantage of this for their migration to avoid downtime!)
The tricky bit is what kind of logic you can apply to values that are the result of provisioning/looking up resources in the first place, isn't it? Like, what dynamically determined resource IDs are exposed to the language and in what places you can reference them. Basically the capability of terraform's interpolation language vs. CloudFormation Fn::* constructs, I suppose.
cat template.yml.erb | ruby -rerb -e "puts ERB.new(ARGF.read, nil, '-').result" > template.yml
The CFN coverage is not as complete as I would like but I've built and managed production workloads with it and it does the job. Since it's built on NodeJS so if you know JSON and a little bit of Mustache it's not hard to understand. Anyone want to help?
CloudFormation is a great tool, don't get us wrong. It is one of the first cloud IaaC tools.
And it has it's downsides:
- inability to address specific object in the state, similar to `-target ` option in terraform
- forcible re-deployment (`taint`ing) of the resouce is not possible too, well, not as easy as in TF
- when creating ChangeSet in CF, it is absolutely useless on nested stacks, all you see in it is "Stack will be updated", even if there is nothing to update
- targeted `destroy`s do not exist, as well as destroy `plan`s: similar to what `terraform plan -out plan.out -destroy -target aws_resource.name; terraform apply plan.out` does
- The work with state, renaming, reassigning, deleting and importing resources is missing in CF as a class
- TF refreshes state of the resources each run, checking their actual state and trying to revert any changes applied to them outside of the flow, CF assumes that nothing is changed and gets very surprised when it is drifted or simply doesn't match its expectations. ( I know there is drift detection, but does it actually restores the desired state? I've never had a chance to check that..)
When we deployed new TF stack, we imported part of the resources created by old CF template. And now I am a bit worried to clean it up, because it might try to delete old resources, even with '"DeletionPolicy": retain'. I have no visibility or control over its action. Basically: apply the template and pray.
ChageSets also failed me once, trying to do not what it was telling me in the plan. Terraform plan once captured into the file will do exactly what it promised me, when applied.
To sum up: both tools got their 'pro-'s and 'con-'s, and personally I feel more comfortable with the Terraform.
Of course obscurity is not security blah blah blah. Still I can’t help feeling that writeups like this could backfire down the line.
By sharing this with the world, you are encouraged to face any vulnerabilities that you may have overlooked.
It's just like when talking about secure communication, you explain exactly how the public/private key exchange works, what algorithms are used, and how the entire handshake takes place. You don't say "We are keeping those details secret just in case it might aid some hacker".
The added benefit of everybody examining the process and agreeing that it's sound and secure is just too great to give up.
If they listed out security group and IAM configurations, or how exactly they’re connecting through the bastion, then it would be a little more risky, yes.
I think it's really great that they've talked about this, it's quite rare to hear about these kinds of internal migrations, and it's something I do a lot with clients but it's not really glamorous enough to talk about.
The difference between StackLeap and sites like acloud.guru is that I'm focussing on the day to day stuff rather than high level concepts you need to know to pass the certification exams.
I'm hoping to launch a beta in the next few weeks.
You might not be ready for that level of integration, but see if you can get a demo or trial.
I would love to go into more details about the environment and some of the things we did to build on top of the default AWS security settings. It was too much information for this post, maybe we will do another one that focuses on security.
Sending several million emails could be a challenge considering that most of our customers depend on 1Password apps and usually not affected by the downtime.
We do have a status page and Twitter feed where we make announcements:
P.S. sorry for the self-promo
One suggestion re: TF files is to keep each service in a separate TF file/state and keep all your state files on S3 that way if someone does an apply it's always consistent. Keeping things separated means that you don't have to worry about hitting other services when you do an apply
Terraforming github repo https://github.com/dtan4/terraforming http://terraforming.dtan4.net/
One of the greatest strengths of terraform (vs say, cloudformation) is that you can adopt existing resources, and ZERO downtime is needed to migrate.
Why take downtime?
Given a bit of time with Terraform, the need for modules becomes obvious as you identify common resources for the infrastructure you're modeling. I thought writing modules first was the "right way," but starting with modules ended up being a waste.
Modules are really neat but I don't think they're a sensible starting point if you're not doing a lot of repetition or don't exactly know what you need to build yet.
For me, I choose cfndsl to build my cloudformation templates. This gives me the ability to actually write code. JSON, YAML, HCL aren't full languages. Yes tf and cf try to provide language like concepts (iteration, etc) but really at the end of the day they are just a definition of your environment and not strictly code.
Note: I maintain cfndsl, so I'm totally bias.
Kubernetes is powerful, it is modular, and it makes everything a lot more efficient, but setting it up - especially in such large deployments as 1Password would have here - is never easy.
Also, I recommend that you first try using kubernetes before you dive in with setting it up. Kubernetes.io has a live tutorial where you can work with a minikube cluster in your browser, afterwards you might want to use Google Cloud's free tier for a small Kubernetes cluster or Minikube until you're comfortable with kubernetes.
For seting up your own cluster, I've heard great stuff about kubernetes the hard way – not to actually set one up, but to learn how the internals work, so you can then fix issues in the cluster you’ll set up with kubeadm/kops/GKE
It might be easier now that AWS is starting to support Kubernetes.
You can grant IAM roles to individual pods running in k8s using kube2iam, though there are new advancements seemingly coming, or already out, now that amazon has announced eks. https://github.com/jtblin/kube2iam
Kops can provision and manage a cluster that incorporates multiple subnets allowing you to have a multi-az buildout https://github.com/kubernetes/kops/blob/master/docs/high_ava...
Are you kidding?
I don't see any mention of Terraform Enterprise here, either. I imagine they're perfectly capable of pursuing Terraform with the foss version, although the enterprise complement has some pretty great additional features.
The blog post is patting itself on a back for migration that caused downtime.
It included this gem:
> Couldn’t you’ve imported all online resources? Just > wondering.
> That is certainly possible, and it would have allowed us
> to avoid downtime. Unfortunately, it also
> requires manual mapping of all existing resources.
> Because of that, it’s hard to test, and the chance of a
> human error is high – and we know humans are pretty bad
> at this. As a wise person on Twitter said: “If you can’t
> rebuild it, you can’t rebuild it“.