Hacker News Reader

Comment by Androider | Hacker News Reader

In my opinion, databases are not cattle, and don't need to be automatically created (and destroyed!) in your main Terraform plan.

It's perfectly OK to have a completely separate Terraform project that just configures the DB initially (or even manually, I see lots of places running DB's that predate Terraform with immutable infrastructure for everything else), and applies minor non-destructive changes in the future. This way you get the benefits of IaC, but the DB plan doesn't participate with the rest of your infrastructure that IS ok to blow away and re-create at will.

BTW, Amazon RDS backups work the exact same way: Destroy the database and the backups are also destroyed. Therefore, same region automated RDS backups are fine for day-to-day, but in a true "DB goes poof" disaster you should expect that you WILL lose them too! You need cross-region, or even better, cross-account DB replication or snapshots to survive this.

View on HN

> You need cross-region, or even better, cross-account DB replication or snapshots to survive this.

Excellent point.

AWS provide a couple of decent options for cross account backups, the Aurora Snapshot Tool [1] and the AWS Backup service. I've used both successfully.

[1] https://github.com/awslabs/aurora-snapshot-tool

It's worth checking out "terragrunt" it helps one breakdown larger terraform sites into re-usable modules which can be ran independently without having to redefine common configuration such as backends.

It allows for running a plan or apply on the whole site, or each individual module in the site.

So in this case it's easy to just apply the database changes, and then run the whole site plan to make sure everything is indeed in the right state.

This is what I've always done.

Databases (or cattle, how or what ever they may be) are always created as a separate stack in your IaC of choice. The values you need for this stack are always outputted for consumption into other pipelines.

The benefit being you can make lots of small changes to your upstream stacks without breaking your database. And you can place additional controls around the database stack to prevent instance replacement from occurring.

> Databases (or cattle, how or what ever they may be)

It's a reference to the idea that you should treat your servers as "cattle, not pets". In other words, they're disposable and you should be fine with destroying one and creating a new one to take its place.

The parent is saying that it's fine to treat databases more like pets, where they're pampered and looked after carefully, and fed treats - much like DBAs are, I believe.

I don't know much about Terraform, but at least in Pulumi you can mark resources as protected to prevent accidentally deleting them.

https://learn.hashicorp.com/tutorials/terraform/resource-lif...

You can do it both in Terraform and also at the resource level in Azure itself.

That's chock full of bugs unfortunately. For example, if you lock an Azure DNS Zone to prevent it being deleted, you then cannot delete any DNS record under it! It's a strict hierarchy, there's no way to turn off the inheritance.

(It is possible to create a custom RBAC role that excludes zone deletion only, but this is very fiddly and not-quite-the-same in complex ways.)

An Azure resource group structure along those lines that I've come up with is:

    APP-LOC-SHARED       -- wildcard certs, deployment scripts, etc...
    APP-LOC-ENV-Common   -- gateways, etc...
    APP-LOC-ENV-Data     -- databases, storage accounts, pets.
    APP-LOC-ENV-Web      -- cattle
    APP-LOC-ENV-App      -- back-end cattle

DBs are cattle for dev environments. However, i agree they are not cattle for acc and prod.

One of the benefits of creating databases in terraform or some other IaaS tool is having the DBA teams pre-define files to create databases along with the standard set of monitoring, logging, and other devops goodness. That way everyone gets the same set of “golden signals” monitoring and dash boarding for free.

All you have to do is set the final_snapshot_identifier and leave skip_final_snapshot off, right? Never done this, so I don’t really know.

Reply on news.ycombinator.com