Are we the only ones dealing with this? How's everyone else handling it?
1. If it's not written down, it does not exist. This needs to be a ruthlessly enforced rule. I know we all love those Mel the Programmer stories, but that's a story about fragility. Similarly if you are relying on SRE or dev heroics to solve field problems, you have a gun to your head and you need to solve it yesterday.
2. The TCO of knowledge is dominated by the maintenance cost, not the creation cost. I see this mistake made a lot. Anyone can create a confluence page, Slack comment, SharePoint site or a git comment. The org must make it someone's job to grow, weed, and maintain these gardens. It does not matter what system you use, but someone needs to look after it.
You get a bonus for building information "data structures" that make it easy for people to pull data out as your org changes, but this is probably something you should accept that you'll get wrong as your org grows. AI hides this problem because it makes search so easy, sometimes, but it will come and bite you hard one day, so be ready for it.
Importance is the more obvious one - there is critical info you absolutely need as a basis to understand the system, then there is also minor details. The difference is in an emergency, lack of the former will break your attempt to fix the system, lack of the latter will merely inconvenience you.
The Frequence of Use axis is the one that brought value to our operations when we introduced it. That's the question "How often will I need this info?". Regularly re-occuring, detailled procedures need to be accessible easily, so they go to a prominent Confluence page. But then there is information that you will hardly ever need, some arcanae about the system that only the old timers knew but that needed to be fixed in writing.
The most vexing bits of info are those that are highly important but are rarely ever needed. Do you plonk them into Confluence? Might not be the best idea because then you don't know how to quickly find it when need be. So we had an extra category for just that sort of thing which became our go-to info hub for all sorts of highly urgent but at first intractable issues with our system.
In my experience:
1. Well meaning people try to build wikis/confluences/whatever that turn into backwaters of slowly rotting stuff
2. Beyond a certain extent that they can hold in their heads, most other people give up.
3. Some people hoard knowledge and use it for personal advantage.
I dont know what the solution is. Maybe an ai that reads every chat and email and commit - and builds some kind of browsable, queryable, executable knowledge trove.
As to whether people like it: I was thinking of a work context, where expectations of privacy are lower. I guess it could be an opt-in for group chats.
1. People would go in and “clean up” what they thought was a rotting Confluence. They were new-ish to the company and didn’t have the knowledge or experience to know what they were looking at. Rather than ask, they just deleted stuff. Some of those things they deleted were docs I wrote, several over 20 pages long that I referenced regularly… gone.
2. Constant re-orgs and tool changes meant it was becoming difficult to know where anything goes. Not long ago I had 4 bosses in 4 years. I got tired of starting over and shuffling things around.
If I keep what I need, no one else is going to delete it without consulting me, and I don’t have to try and migrate and re-organize it every time I get a new boss or management decides we’re using a new system.
I guess a third thing, which is less of a trend, is friction. The collaboration platforms are much higher friction than something like a local Obsidian vault.
I’ve probably written and shared more documentation than everyone else on my team combined, but at some point, I got tired of the uphill battle. I still do write and share stuff I think will benefit the team, but I no longer try and make everything public for the reasons listed. If someone asks for something I have, I will always send it over, and having my notes in markdown now makes that easy.
This behavior forcing every individual to skip the other necessary activities like RCA and documentation and straightly trying to deploy the product into production.
This way, new and updated context is not lost on the next user.
I do like the idea of having a live editable wiki as the knowledge base a LLM can reference.
i think one way to solve this is having an someone or an agent update the initial spec with any new changes as and when they are made.
If the LLM is going to write the doc anyway, why not just do it on demand from the code if/when it’s needed, instead of burning tokens on documents that no one will ever read?
I also think a lot of the value in those historical docs are the reasons why certain decisions were made, as it can be important context if there is every a discussion on doing things differently. The LLM likely won’t have that information, or may just make up a bunch of reasons the business doesn’t actually care about.
Our "project knowledge" consists of artifacts of such types:
- product documentation (made by human). Intended product behavior and capabilities.
- technical documentation (maintained by human and agent who work together on feature). Implementation documentation.
- feature session journal (made by agent and human who work together on specific feature). Per feature session. Contains original problem, gaps, problems, misunderstandings and failed attempts.
- wiki and ontology articles for targeted knowledge retrieval. Built by Knowledge Keeper procedure under human supervision on demand. Consolidate the knowledge across subsystems.
- code and tests (mostly written by agents).
All these knowledge artifacts (except code and tests) are mostly markdown articles. They are stored in git - this allows versioned storage, authorship and diffs. All articles have front matter with summary, keywords, status (draft or reviewed and approved by human) and links to related articles, so we and agents know what is relevant and what the source of truth is. Wiki and ontology articles hold the semantic relationships for better retrieval.
Typical process of changes: As agent and human work on the feature, this is captured in the session journal. At any moment in this process, the engineer who works with the agent can ask the agent to hand off their work to other agents, integrate in the technical documentation or handoff to a Knowledge Keeper. Such handoff includes the message with documentation links and the link to the session journal, so one who continues starts with the context.
The Knowledge Keeper is a supervised procedure that keeps wiki and ontology aligned with the product documentation, technical documentation, known issues. Knowledge integration process is currently manually initiated by the handoff of the human who worked with agent on the feature. The Keeper prepares the integration plan which a human approves. Keeper builds the index on top of wiki and ontology after integrating the knowledge and tests the knowledge retrieval via MCP to check if integrated knowledge retrieves correctly and reports to the human maintainer. MCP allows the consolidated view of the product and its capabilities. Its answers point to the primary docs, therefore users do not need to find one person who remembers everything.
The problem of scattered data is to put data in one place.