Ask HN: Organizing company knowledge?
In the world of collaborative systems and wikis, what are some good resources for learning how to organize and structure knowledge at a company level?
Too often this falls by the wayside because documentation of processes and preserving institutional knowledge, while extremely important, isn't the type of work that gets recognized or rewarded.
If anyone can recommend a good way to align incentives here, I'd be very interested.
In fact one of the best organization I ever lived was in a small company (<30 employees) and everything was just organized in folders. Just Folders, no special tool, no wiki, nothing. In fact the main document describing the organization was maybe 10 pages long and the first thing you had to read when joining. As everyone was following it it made finding anything a breeze.
There was basically a semantic document number indicating what the document was about, which revision it was and where it was stored. Each Project/Product had an index linking to the document that was maintained by hand. This even extended to software versions where you had to publish some zipped version of your software at each release.
I wonder however how it would scale...
I would say that the culture + making sure your projects and teams are well organized is the key to success. In addition, I'd recommend to follow some protocol when documenting things. This will make both writing and reading a less intense and more predictable task, resulting to a better overall experience.
The best case scenario I found was in a bank. Each line of code was documented and rationale explained in plain english. This way hypothetically even a newbie could run the system.
While different places have different issues, I found that the quality of documentation was inversely proportional to people's insecurity about their job. More insecure they were about their job. More resistance to documentation was found.
Wow. Every line.
There is such a thing as too many comments or too liberal use of comments in code.
APL/J are highly compressed and any reasonable explanation would require more than one line of documentation per line of code. I know there's some J in financial institutions.
For most code it will be overkill but there are some companies where it's integral for core systems.
I would love to work in a place where documentation was cosidered a valuable asset, but I think that your example is off mark.
The best rule of thumb is, make the code so clear that it is itself a comment, and only if the code cannot relay the meaning, then add a comment.
on the other hand, if it was named something like number_of_records it would be clearer.
That’s only true if comments are kept up-to-date. More often than not, they get out of sync with the code itself, causing even more confusion.
Can be sold as an on-boarding tool if you are in a large team some moderate turn over. Or as an insurance policy if you have a very small team with a few key team members, if we lost them for any reason would be a substantial setback to productivity.
In general I think people like good docs but fail to think about how to get there. Time must be spent.
Other tools I've seen used with success generally have the characteristics of being lightweight, not too complicated, and having a relatively frictionless interface for making small changes (e.g., "press an Edit button and start typing").
I've been in a culture that desperately wanted to do documentation, but the tool that management forced people to use was SharePoint, and it was horrible (search functionality that never worked, very bad response time, very difficult to link or move documents, and worst of all a policy to delete "unused" SharePoint sites after a while . . . the result being that important engineering documentation was irretrievably destroyed). PMs loved the busywork that SharePoint created for them. I wound up putting all of our documentation into source control and snapshotting it to SharePoint periodically. This made me unpopular with the PMs, but I'm pretty sure the documents are still there, a decade later.
So tooling doesn't matter, unless the tooling stinks badly enough.
Effective knowledge management practices are something we think about a lot at Spoke (https://www.askspoke.com). We've studied other companies' practices and have come up with some organizing practices/principles:
1. Adopt the right tools: - Adopt cloud-based knowledge management tools that are simple to use. If new tools are overly complex, no one will use them. - Find solutions integrate with the applications employees are already using. (Be where employees already are, like Slack.) - A central source of information is best. If knowledge is spread across multiple tools, it will still be difficult for people to find.
2. Take advantage of new technologies: - Adopt tools and technologies that use AI to process and catalog resources. Ideal solutions may automate the processes of updating knowledge and/or automatically categorize and tag new content to make it easier to find. - Look for tools that use machine learning to improve as data is collected. Machine-learning technologies learn how people search for certain types of information, getting better over time at helping users find the exact information they’re looking for.
3. Document important processes: - Set aside time once a month for employees to create documentation on the tasks they’re responsible for. - Save all documentation on the cloud or some other shared server so everyone has access to it and to prevent document loss.
4. Find creative ways for employees to share tacit knowledge: - Establish a mentor program that pairs new hires with long-time employees. - Make sure managers know how to perform the most critical tasks that their teams are responsible for. - Set aside time for employees in related roles to cross-train. This will expand institutional knowledge, provide a source of backup when employees take time off, and reduce the likelihood of total knowledge loss caused by unexpected turnover.
I think the tool choice doesn't matter much but having a well defined tool or process helps a lot. In my company it's never clear where to our documentation or where to find it. So every time you have to think about this and often don't get started . In my team we have started using OneNote for everything and now we always know where to put things and where to find them. Suddenly documentation is getting better and is actually useful because you can find what you need.
Previously we used SharePoint and data just went into it and disappeared to never be seen again. Totally useless.
My quick answer which may sound trite is compensation. An organization can compensate the author(s) either during their employment or even after. Yes, after. Award either non-voting shares. Set up a micropay system for each time a current employee "votes" the documented solution.
Documentation (and organization of documentation) is my thing. I hear so much talk and see so little progress. The most expensive work hour is the unbillable hour yet organization after organization continue with status quo of inaction.
The motivation is simple, the implementation is more difficult.
People will document things if they feel as though they derive a personal (read -> personal, not party) and immediate benefit from doing so. That's it.
This is why getting motivated to doing unit tests is so difficult. There are many long term and group benefits, but the immediate and personal benefits are limited (especially in relation to the amount of effort).
My tips:
1. Maintain a clear single source of truth
2. Make sure everyone can contribute to it. We do this by requiring at least to edits to the handbook during onboarding
3. Embrace it as an organisation: if not everyone is committed to it, it's hard to maintain it as truth
4. Constantly iterate and improve it. Structure is highly dependent on the content.
it is used throughout the data org and has been extremely popular and useful. there is a blog post here with more information: https://medium.com/airbnb-engineering/scaling-knowledge-at-a...
I just wanted to give a shout out to all of you and say thanks for giving so much back to the community.
There are other strands of thought that may also be relevant. E.g. I think HR people often think about "workspace learning" or "training and development" (e.g. https://en.wikipedia.org/wiki/Association_for_Talent_Develop...).
And in the world of management there's the concept of a "learning organization": https://en.wikipedia.org/wiki/Learning_organization
(You will get a lot of answers involving tools. Tools are the last thing to think about.)
https://en.wikipedia.org/wiki/Knowledge_management
Also related is the field of "Organizational Learning".
https://en.wikipedia.org/wiki/Organizational_learning
It's a huge topic, and would be hard to do justice to in a HN post.
There are a lot of good books on the topic. A few I'd recommend include:
https://www.amazon.com/Handbook-Organizational-Learning-Know...
https://www.amazon.com/Wellsprings-Knowledge-Building-Sustai...
https://www.amazon.com/Winning-Knowledge-Transfer-Michael-En...
https://www.amazon.com/Only-Knew-What-Know-Knowledge/dp/1451...
https://www.amazon.com/New-Edge-Knowledge-Management-Changin...
https://www.amazon.com/Fifth-Discipline-Practice-Learning-Or...
This is a document which exists to link to other documents. I use a shared Google Doc for it (which is almost equivalent to a wiki).
Since it's a document (and not a folder or a spreadsheet) each linked document can be combined with some explanatory context - is the document current? What does it cover? etc.
I also like splitting my doc-of-docs into different headings, for example:
Current
Outdated
Related projects from other teams
The key benefit of the doc-of-docs approach is that the answer to EVERY question about "where can I find documentation for X" is "Look in the doc-of-docs". And the follow-up answer if they don't find it there is "Go and find that document, then add it to the doc-of-docs".
(Disclaimer / I worked there in the past)
There's no Linux download, no github, and especially, no server - I stay away from products like that, sorry.
Almost every company uses external services for this sort of thing right now. Google Drive, Confluence, whatever. Notion allows you to export to PDF/Markdown as well.
https://www.notion.so/Export-to-PDF-or-markdown-5406d98f17d2...
However, I work at a large company that has adopted Confluence, so I'm stuck with that for organizational knowledge.
The lack of an Android client might be a blocker, although according to their Twitter, one is only weeks away.
Now, documenting things is a different story...as gringoDan pointed out here (https://news.ycombinator.com/item?id=16811654), whether or not your documentation process/tooling gets anything done is directly 1-to-1 correlated with company culture on documentation. I've personally tried numerous times to get documentation going, and it always fails because no one keeps up with it.
What I liked about Slab is the way better performance over the Confluence, quick integrations from their team (took them about 2 days to add draw.io integration on request). I liked how clean it was and easy for non-devs in my team to understand the structure.
How was machine X configured? --> read the setup script.
How do I import the CSV data from server Y? --> check the steps in the script.
How do I add a new user to system Z? --> check the steps in the script.
The point I am trying to make in case it isn't clear is that a system that comes with plugins and allows you to structure the documentation to suit your organisation is best.
As others have noted, not many developers love documentation. In my experience, it seems to happen in bursts. It's remarkable how useful and relevant a piece of information is even 5 years down the line. [1]https://www.drupal.org [2]https://www.drupal.org/project/feeds
I hate it, but it is the only thing we've found that meets all of our needs, isn't $500/mo+, and gives our mid level managers control.
We use it for everything from on-boarding, to training, to system documentation, to competitive intelligence documentation.
Sure we could build something custom, but overall it works and it's given the team leaders control over their "domain."
Also we found that making people specifically responsible for documenting their job/building training for others before they "moved up" into a higher role was really effective. Wikis didn't allow for the granular control we wanted over who got to see training/documentation.
The only time when people are absolutely required to document something is when it's a deliverable to a customer...so the customer needs to understand the product we're delivering to them.
We're all supposed to be using an engineering notebook to help document designs...but those really only come into play when someone leaves the company and the company is trying to retain as much IP as possible. Though they are incredible useful on an individual level.
Other than that, it really comes to down to finding time to draw a diagram or do a brain dump into a document describing how something works. Maintaining ICDs (Interface Control Documents) describing software/command protocols or hardware interfaces is pretty easy and provides a lot of information.
It is good for defining structure and permissions at an org level for files and folders and solves that pesky problem of documents not being accessible if the Creator of them has their account shut down.
its honestly a market i dont understand, i'd love a high level primer on it.
For example, when I worked at a company that used Google Apps for Business for everything, I set up our documentation in a Google Sites Wiki. Now that I'm working somewhere that uses Microsoft for everything, I've set up a Sharepoint wiki.
In both cases, the tools wouldn't have been my first choice from a page-editing or layout standpoint, but the ease with which my less-technical co-workers can search and find information and process documents with their existing credentials has been key to encouraging broader use.
Just paste in new information into the comments section and then work it into the summary in the card description. You can also assign each topic to have an owner to make sure the card stays fresh.
When there's a much larger document, we create a card for it and link to a GSuite doc. For dev knowledge however, it's updated in our repos directly (e.g., README)
Very much still working on it :)
But, as others have pointed out, adding a tool won't create knowledge. You need to foster a culture of creating documentation, writing notes, just writing everything down. It's so easy to relax after a hard problem, but if you document it, then it will be easier in the future.
Some other things I've learned... Wiki's suck, they become stale, they are disorganized, very informal. I recommend just keeping big ass user guides (can be different ones for different systems or even user guides about procedures that are not about products) using a well-organized documentation system and a well-kept table of contents. Don't waste tech writers on internal docs. They'll spend half their time asking the dev anyways, it will get stale, and asking a dev to write something is not asking too much. Keep the docs close to the code, but not in the code like comments. Docs close to the code can let people have less friction between code and doc commits. Each non-trivial repo should have a manual (a.k.a. user guide) anyways. And if you have to, say, grab a bunch of RST files from a few disparate Sphinx doc locations, so be it.
I personally use documentation as a point of pride. My favorite part of any project is documenting the awesomeness I did. I just make sure I order it in a reasonable order and don't bloviate.
If you don't have a librarian at hand try something that forces more structure at you.
As others have said, your company culture has to support it, lest you use the resources to set it all up and no one uses it.
If your company does not prioritise a written culture, then there is no tool that can come in and magically do it for you. Your tools will only amplify this phenomena (your team will use the tools for some time, and then quickly think of it as "another thing to do" when the shine wears off and stop using them)
This is because of the way current tools are set up. I have seen companies hire interns to help keep their internal knowledge updated. Despite this, there is always some document to update. I view updating documents as work that is required to keep other (perhaps more "important") work seamless. I can draw a comparison between maintaining internal documentation and writing unit tests (both equally important, while being underrated and easily postponed for later) At scale it either compounds to a technical debt and can bring the whole company to a halt momentarily.
It is really simple, everyone who can use Word can use it. It is also very wiki-like in that it is quick to edit, searchable (and fast), versioned and it is easy to see when something has changed (bold tabs).
Personally I used to prefer dokuwiki (and before that I made a custom tool in Javascript to simplify use of mediawiki to document procedures.)
We’ve been using a moin moin wiki. It’s ok, but we’d like to move to a simple, hostel solution so we don’t need to maintain a moin moin server.
We don’t need fancy, just simple with attachments, images and links.
Because large amounts of knowledge are locked away in individual emails, having a central storage repository with good search and easy control over the flood of information makes it possible to find the knowledge in-context later.
- people don't feel awkward to modify other people's content because there is a collaboration and review process in the form of pull requests
- you can assign different people to oversee different parts of the knowledge base. They make sure the organization of the knowledge base doesn't get messed up by reviewing changes of other people to it.
- you can keep others in the loop about important updates (big and small) by tagging them in pull requests
- verification can be automated via bots and CI, for example using https://github.com/originate/text-runner
This is difficult and time-consuming to setup. But if you cann afford the effort, this gives you a very flexible system where you can decide what to store in a structured or unstructured manner. Structured data can of course be queried, aggregated, vizualized etc. Together with the Page Forms extension this can replace many Excel Spreadsheets and MS Access solutions.
If you really know what you're doing, you can even do moderately complex CRUD Applications. But then you really need to know your way around several extensions and of course wikitext.
For simpler use-cases I've also used Drupal. For this use case it's not as flexible as MediaWiki, but you can do a lot through the GUI and build custom Forms / Views.
At my company I do internal nurture marketing (thank you MixMax) to remind employees to use knowledge resources, and to contribute to them.
I've seen companies using forums, Google docs, and even Github repos.
The information is almost always 3 clicks away, but people just want to ask another person, or behave in a paralyzed manner as if the information couldn't possibly be found.
And if you work in a "lotta meetings" company culture, even moreso.
Starting out can be tough. Experience and observe the company/department/team you are seeking to organize and structure the knowledge for.
* Do they care about capturing, authoring, presenting and maintaining knowledge? How much do they care?
* Enough to insert steps into their existing processes or create new ones?
If so, you have a chance. Otherwise, focus on organizing and structuring YOUR OWN company knowledge. When others look over your shoulder at a meeting, read your notes, your reports, your guides, you may be able to entice.
Examples speak louder than words.
Tooling can be tough. If you work in a small outfit, you can likely spin up the best KM tools available. If you work for a dinosaur, good luck. They are likely already entrenched with some sharepoint mess that busily employs 10's if not 100's of people. Bringing your "new tool" to the IT department will get you the run around. (I've been trying to implement Gitbook (my fav KM tool) for over 1.5 yrs.).
Large companies may also have problems with cloud deployments. And honestly, in light of data breaches, etc, I get it. Many tools (most of the ones in this Ask HN) are cloud-based so this may limit your options.
I'm currently using OneNote to author and present ~300 pages of docs to ~3000 users. Being the SOLE author, this is fine. The style and structure are consistent, but I don't like it. OneNote doesn't support markdown (are you serious Microsoft?!? you don't have an f'ing markdown editor yet?), locks you up in a proprietary format, etc. etc.
Its also too "willy nilly". Its meant as more of a personal KM tool rather than a something for general consumption.
Where I lean:
* Static site generator under git version control
* Hook up with your local "Write the Docs" Meetup
* Check out the annual "Write the Docs" conference (coming up in a few weeks in Portland!)
* And start with YOUR OWN company knowledge.
Many people (including dev) use it on a daily basis, and most people on a weekly basis.
Definitely recommend.
2 classic pitfalls with knowledge are - what do you put in this word (for instance are meeting minutes "knowledge"?) - how static you think it should be
If you only envision knowledge as static processes, traditional wikis work. If you want all your team to contribute and for this knowledge base to be a daily used tool, with all the information that matter you need another setup and we are building Slite for this (happy to have feedback!).
Edit: I see https://news.ycombinator.com/item?id=16476388
I have spent my career with technology teams, either 2 guys in a closet or 40-thousand people working on multi-year, multi-billion-dollar efforts.
Something weird happens between the 2-guys scenario and the 40k version: critical information stops moving around effectively. Many times big organizations seem to exist and profit more out of a sense of inertia than strategy.
Looking at this, over the last several years I've been trying to identify why some organizations do well in how they handle information flow and how some (most) organizations don't.
There is a ton of detail here. It encompasses everything from Lean Startup to Business Modeling and Extreme Programming. (Way too much for an HN comment, sadly!) There are a few folks talking about how to think about information at the meta level, but not a lot. Happy to point folks to some free online resources. Just let me know specifically what you're looking for.
Best of luck with the book!
We spend so much time and effort on building-things-right. It's much more important to build the right thing. Usually (not always) boils down to information flow and feedback loops.
I'm so far in the weeds right now with the book publish date coming up it's for me to provide a good overview. Here are some comments from my alpha reader group (I do not know these people and they are not paid in any way for these recommendations)
Hey Daniel,
Got info about second part of Info-Ops, will get to the details later today. But I just wanted to give you quick info (while not very useful at this stage I assume, but still...) - few days ago I finished the book. I loved it. It's written in simple language, easy to understand and digest even for not-native speaker like myself. So that's very good.
The bad: I wish I read it two-three years ago, would make me much better software developer!"
Daniel,
"You know, when I first got to know you, I thought you were a funny guy, but arrogant. (Yes, I am aware of the irony.) After reading the beginning of this book, I realize you're not actually arrogant - just really fucking capable. This may be the best book about process analysis I've ever read. OK, I have to admit it may actually have been the only book about process analysis I've ever read, so there's an inherent grain of salt there, but damn. This stuff is truly fantastic. Very, very well-considered and thought-provoking. I am truly and unironically happy you've given me the opportunity to read it...Seriously, I find this roughly on par, in its level of thought-provocation, as Doug Hofstadter's work. I might have a slightly weird way of looking at it, so don't derive any expectation of universal acclaim from it, but I do mean it. Damn fine work."
"Daniel. Wow. I didn't know you had it in you. I'm eager to keep moving forward. ... I like your approach and wondering if you can or were at all all looking to at some point to speak/teach/consult on the topic? (We could use at current client!)"
"I'm Ola Ellnestam a software developer and team coach/lead. My job is to balance software development with regards to process, delivery and collaboration. I didn't know what to expect from Info-Ops, I thought it had to do with DevOps and information. A few chapters in though I realized it gives the reader a concrete way of looking at, understanding and in a way, manage the information that flows through a software development effort. It helped me understand underlying problems with that flow and how to deal them. It also gave me new ideas on how to work with semantics in software development in a more pragmatic way. If you want to know how to create a shared mental model in your team and beyond, read this book and I am sure you can become the grease between business and tech."
It's still our main point of truth for a lot of processes and details, though we've been considering moving to Phabricator which we're using for our code.
Create small teams that work well together, give them lots of responsibility and autonomy, and let them work on the things they really care about deeply.
to use open source, wiki is not structured and i really don't like that much, folder-files are hard to manage once you have many files.
CMS could be used(e.g. drupal), however I hope CMS can support markdown _natively_ which is not the case for now.
I have spent so much time to find ways to organize knowledge myself, to no avail, I use drupal 7 along the year, which is not great, but I could not find anything better.
I position myself as
- Virtual Chief Knowledge Officer aas
- Virtual Chief Quality Officer aas
- Virtual Chief Information Officer aas
-----
Issue 1: "Knowledge and process is seen as a dichotomy, but shouldn't be".
All businesses that I have worked with, and all discussions I have seen or been a part of, draw a distinction between actual operations, and the workflow & guides that govern them.
The problem manifests as such:
- The business doesn't update info, as it is buried away (out of sight, out of mind)
- This fundamentally undermines the information immediately, making it even less likely to be used
- The information is therefore less likely to be updated, compounding the issue further.
Instead, these should be tightly integrated. For example - If a ticket/job is generated, the relevant guide should be proactively attached by the system as part of that ticket. When a human comes to that job, the information is there waiting for them. People aren't going to go out of their way to bring up information.
-----
Issue 2: "The Business Manual is verbose, not effective or up to date, and people don't like it"
Most business manual's aren't encouraging use anyway, in that they aren't relevant, are too constricting in areas, and are verbose.
1. Simplify (streamline the process, break the process down into procedures, break procedures down into atomic steps. Everyone in the organisation should know exactly what your processes are)
2. Automate (any steps that can be automated, should be)
3. Steer (Any steps that cannot be automated, should have a guide attached - But only to the extent that you need).
The last point is critical, as I often see strict, long instruction documents that stifle creativity, have a negative impact on job satisfaction, and don't cater for everything possible anyway.
Instead, guides should be curated to the extent required only, using the following attributes:
- Budget hours (if you exceed this let's regroup and decide whether to re-plan)
- Instructions (only where specific and accurate steps need to be followed)
- Considerations list (exhaust this before escalating, but allow creativity)
- Notes (general hints and tips / knowledge base that should be added to over time)
- Team (a section which shows who can train and undertake the task, and who is on the 'to-train' list)
- Reference (reference information or items)
- Tools (any potentially relevant tools)
This strikes a balance between paternalism and laissez-faire, and uses each where it is most effective. It solves a problem in that business manuals tend to be 'ineffectually authoritarian', so people resent them anyway.
-----
Issue 3: Information gets stuck
The following should be standard practice (knowledge transfer is king):
1. An entity is generated in the system
2. Template auto attaches
3. The template has a list of people authorised to undertake that entity, and a list of people to induct to that procedure/task
4. One of those authorised people undertakes the task, along with someone on the 'to train' list
5. Repeat 1-5 until trainer can 'pass' trainee. Trainee's name added to the top of the authorised list
6. Trainer moves down the list of members as people are inducted and added to top of the list, therefore gets used less and less for that task and is freed up to move 'up the process chain'
This creates a moving machine within the organisation, where people are proactively provided information to complete their job, and proactively are able to pass that knowledge on and faciliate personal development within the company.
-----
Issue 4: Reactive thinking, lack of proactive thinking
The following should be deeply ingrained into the culture:
1. The business becomes aware of 'the system' no longer comprehensively and accurately reflecting reality
2. The interface node (user / system) that discovers this difference should log it immediately in or as a system entity (service ticket, general client, technical documentation update, client information change, new client user, data breach, potential sale, etc.)
3. All relevant information should be auto-attached
4. Whenever any entity is being closed off, the team member considers whether there are any potential follow ups that they can derive (including improvements to the template/guide), and triggers them
-----
"3 clicks away" is not good enough. "2 clicks away" [was never] good enough. Information needs to be a) in the user's immediate view along with their work, or b) a single click away (to hide peripheral information, so as not to overload the primary view).
Documentation is unfortunately often an afterthought, or at best is a follow-up task for a system that is not at the forefront of people's vision and mind.
As a final thought, here are two contributions to the suggested software so far (no affiliation):
- IT Glue (if you are in the IT industry) (www.itglue.com)
- Process Street (www.process.st)