Open sourcing Terratest: tools for testing infrastructure code

I'm one of the creators of Terratest. Happy to answer questions.

The main question I've seen so far seems to be how Terratest compares with various "spec" tools (e.g., inspec, serverspec). Most of the spec tools focus on checking the properties of a single server or resource. For example, is httpd installed and running? Terratest is largely for end-to-end, acceptance style testing, where you deploy your real infrastructure, in a real environment (e.g., AWS), and test the infrastructure actually works as expected.

For example, let's say you wanted to test a module for running Vault (https://github.com/hashicorp/terraform-aws-vault), which is a distributed secret store. With a spec tool, you might test a single Vault node to check that Vault is installed and the process is running. With Terratest, you'd check that the whole Vault cluster deployed correctly, bootstrapped itself (including auto-discovery of the other nodes), that you can initialize the cluster, unseal it store data, retrieve data, and so on.

I use Packer and Terraform extensively for my consulting company. I appreciate the work on trying to make Terraform/Packer testable, but I wish this tool was Golang agnostic and written in HCL or a markup language.

The biggest concern with Terraform is that small changes (ex: changing Terraform variable names) often causes rebuilds of large chunks of resources, which when in production is scary and sometimes cannot be applied without downtime. Point being, not sure testing (expected results) is the core problem. The problem is confidence destroying and editing resources will not produce unexpected downtime (modifying ELB's, RDS, IAM, or security group's) are examples.

It's weird to think of infrastructure/deploy automation in terms of "testing". Yes, you are always testing things, but not always in the "func test" sort of way. There's automation testing, and there's testing of automation, and there's tests that are part of automation. Terratest seems to be the second, but it's the third one I think is most useful.

If you build your infrastructure/deploy automation correctly, you should be able to redeploy your full stack all the time, and when it succeeds, throw traffic at it, and if you don't detect any anomalies, make that the new production service. On the detection of anomalies you simply move the traffic back to the previously deployed incarnation (or re-deploy the old incarnation, if necessary). For sufficiently large systems this gets more complicated as you can't just duplicate your resources, but the smaller pieces that have actually changed can be shifted around.

The idea of a "rollback" is really just "return to a previously known good state", but it's misleading. It was previously good before now. Now things have changed, and it might not still be good. So just as much as you can test newly deployed changes before you make it the production service, you should probably also test the previously deployed changes to make sure they will work again if pressed back into service. So, regression testing for infrastructure, I guess. (You'd do this if you were building a physical product like a network appliance to make sure your old appliances still work with newer software releases, but we rarely think of software-derived infrastructure this way)

How does this compare to severspec https://serverspec.org and inspec https://www.inspec.io

Also i would hate to be on a team who is scared to change their code, be it infrastructure or application code!

If you are scared to change your infra code then you probably are not following true infrastructure as code practises.

The missing piece in all of this is the possibility of a clean rollback. If all infrastructure could be described "as code" and is version controlled in Git or something you could roll back. That doesn't take into account the omnipresent state in databases and so on. I have yet to see a system or environment that doesn't save state in databases or similar systems.

also related: Testinfra https://testinfra.readthedocs.io/en/latest/

How does this tool compare to Test Kitchen [1] generally, and kitchen-terraform [2] specifically?

[1]: https://kitchen.ci

[2]: https://github.com/newcontext-oss/kitchen-terraform