Ask HN: How do teams remember why infrastructure decisions were made? | Hacker News Reader

Ask HN: How do teams remember why infrastructure decisions were made?

8 pointscurious_sre11 comments

On teams I’ve worked with, we often run into systems where nobody really knows why certain configs, services, or architectural choices exist. Docs are outdated, Slack history is messy, and the people who made the decision are often gone. When something breaks, we end up rediscovering the same context over and over. Is this just inevitable on long-lived systems, or do experienced teams have a better way of preserving this kind of context?

View on HN

ADR records. Store as markdown file(s) in the repo.

https://adr.github.io/

https://github.com/adr/madr

That makes sense. In your experience, do ADRs actually get revisited during incidents or onboarding, or do they mostly exist as reference docs that people forget about?

Rarely useful for incidents in my experience, mostly useful for onboarding those new to a codebase or part of a system. It’s a mechanism to preserve institutional knowledge that would otherwise evaporate due to not being represented as code; formalized documentation.

The benefit of this information existing in markdown files is this can also be used with LLMs and RAG if getting a natural language interface to the knowledge might be relevant to your enterprise.

That’s helpful context, thanks. In practice, when incidents happen, where do people actually go to reconstruct the “why”? Is it mostly Slack archaeology and asking senior engineers, or is there something else that works better?

As you said, chat logs, logging source of truth, and any context that can be provided by subject matter experts on the system(s) in question.

Google's SRE resources on this topic are somewhat helpful, consider reviewing and evaluating for your environment.

https://sre.google/workbook/postmortem-culture/

https://sre.google/sre-book/postmortem-culture/

Good but needs to be driven by a head of architecture type role incentivizing the practice. Otherwise it's a patchwork of records and largely ignored.

Ah I wanted to make a product to solve this problem. I posted here to see if anyone thought it was worth solving, but nobody seemed to care.

The way I wanted to do this is to create dashboards that would serve at the same time as infrastructure diagrams for documentation and live health monitoring.

Right now, most documentation solutions aren't used on a daily basis so become out of date, because people don't think about it when making changes and fixes.

And monitoring solutions only show you charts of things you're supposed to already know. They're very technically-oriented, and not business-logically oriented, if that makes sense. Like they'll tell you that process x is running on machine m, and that it's running out of ram, but nothing will tell you that process y that depends on x's outputs is going to fail as well.

After I read your comment, I remembered I posted about it too, and came to the same conclusion ^_^

Thanks all, this has been really helpful. One recurring theme seems to be that anything manual eventually gets ignored. Out of curiosity, has anyone seen any system that captures context automatically, without relying on people to write it down?

I send an email to the team, that has a few notes. It's actually an email for me in the future, and the notes are in case the future me forgot the details.

Confluence pages, ADRs in github. Not perfect though.

Reply on news.ycombinator.com