Hacker News Reader

Comment by ryeguy_24 | Hacker News Reader

Where do people recommend logging infrastructure changes (e.g. database changes, server configuration) for all things potentially not in code?

View on HN

Whether something is "in code" or not is orthogonal to the problem of tracking infrastructure changes. E.G. You can commit a change to Terraform, but that doesn't tell someone when "terraform apply" occurred.

What you should use depends on what tools you have available. We currently use Datadog, so we use their API for publishing events. We can then search for them in the event stream or overlay them on dashboards.

You could look at the datastores Grafana supports for annotations and choose whichever of them you're comfortable operating.

http://docs.grafana.org/reference/annotations/

We need more context. Server configuration should definitely be in code/source control these days. Database changes, well, what kind of changes are you talking about?

These days, as we have multiple departments and services, it’s hard to track down any issues we identify without knowing what kind of changes have happened to potentially cause it. As we don’t have a single development platform (multiple acquisitionss) this is hard for us at least until we have unified systems (will take time). I’m referring to any infrastructure or operational change to the stack. For example, deployments/roll outs, scaling servers up or down, port changes, updates to software on key servers, changes to database table structure, manual changes to database data. Any changes at all to the production stack.

You might want to take a look at Architectural Decision Records in addition to Change logs. A lightweight format for tracking changes to infra that can be checked into source control.

One example - https://github.com/joelparkerhenderson/architecture_decision...

Choose a documentation source-of-truth (Confluence, etc) and keep a changelog[1]. I've also seen a dedicated email group (i.e. change-mngt@company.com) used for tracking non-code changes.

[1]: https://keepachangelog.com/en/1.0.0/

Terraform checked into version control is pretty common

It's all in code now... At least how I do it.

Reply on news.ycombinator.com