The case for continuous documentation
virtuallifestyle.nl
virtuallifestyle.nl
1. If the docs are in the same repo, a commit that changes the code can update the relevant documentation (in addition to the tests) as part of the same unit of work
2. This means it can be enforced during code review: if a developer forgets to update the docs they can be reminded before they land their PR
3. This also provides a version history for the documentation which is synchronized with the code history. This is really useful when looking at history and trying to figure out what changed when.
4. This also works great with branches, PRs and releases. New features can have their documentation developed alongside the code in a branch, which makes it easier to understand a proposed change. If your software is deployed in multiple places as multiple versions (or even just staging vs production) you have a way to view the correct documentation for each individual deployment.
5. Added together, all of this builds trust. A common problem I've seen with internal documentation is that no-one trusts it to be up-to-date. Making it part of the regular code development lifecycle can fix this.
6. If you do this, you can write automated tests that enforce aspects of your documentation! I call these documentation unit tests, and wrote about them here: https://simonwillison.net/2018/Jul/28/documentation-unit-tes... - even something as simple as a test that fails if a new API endpoint isn't mentioned in a markdown file using simple string matching can ensure no-one forgets about the docs when they add a new feature.
That's a great idea. A related thought I had was testing assertions about architecture by looking at the graph of imports or calls.
From your blog post,
> if a change doesn’t update the relevant documentation, point that out in your review!
What you get at but don't seem to state explicitly is that finding the relevant documentation is hard, for both the author and reviewer. Finding any relevant documentation should be easy, but ideally we're finding all relevant documentation, and we need to be reaching sufficient confidence that we're missing little enough that we're able to hold back entropy enough to keep the docs useful.
Your tools address this for some cases! Doctest addresses this for other cases. From TFA here, it sounds like Swimm.io tries to address this for more cases (my gut says the article oversells it but I intend to look more closely).
To get further, an idea I've been toying with is to treat claims (implicit or explicit) in documentation as requiring citation, pointing not at sources but at tests. Ideally the test runner, when a test fails, can then surface all references to that test. In addition to highlighting portions of the docs that may need to change, this seems likely to also provide crucial context when fixing the code and/or the test.
Some examples:
- https://docs.datasette.io serves documentation from https://github.com/simonw/datasette/tree/main/docs - which has documentation unit tests here: https://github.com/simonw/datasette/blob/main/tests/test_doc...
- https://sqlite-utils.datasette.io/ serves from https://github.com/simonw/sqlite-utils/tree/main/docs - unit tests here: https://github.com/simonw/sqlite-utils/blob/main/tests/test_...
- https://django-sql-dashboard.datasette.io/ serves from markdown in https://github.com/simonw/django-sql-dashboard/tree/main/doc... - I don't have documentation unit tests for that yet
Those three are all hosted on https://www.readthedocs.org but I've also used this trick on web app projects that host their own documentation deployed as part of the build process.
Emacs org-mode would be a great candidate as well, for things like runnable code inside the documentation as examples and export to many other formats and nice tooling for viewing the files. Unfortunately many git hosts do not support it well and render crappily.
I would recommend against switching to a non-standard Markdown dialect. Why switch away from reST, which offers all of the things nessecary for a good technical documentation, to some Markdown dialect, which has many important features only bolted on?
That said, one thing I noticed happening with this approach is, that people think "Oh, I have my documentation in my comments of the code! I don't need to write anything else!" And then I end up reading documentation like: "def get_a(): ..." Docstring or comment: "Get a." Wow, thanks, how helpful!
In short: There is no simple way to have good documentation, except for writing good documentation. Docstrings probably will not be sufficient, unless you write whole novels in your docstrings. A good documentation needs usage examples and rationale of why something was done in a specific way, what kind of gotchas there are and probably other stuff, that does not come to mind right now.
Honestly, the main reason is that I've encountered developers who have an almost alergic reaction to rST - they genuinely hate writing in it, and will be deterred from writing documentation if they have to figure it out.
Custom Markdown flavours are more likely to get buy-in.
I have used Markdown, Pandoc Markdown, probably Github Markdown, probably other dialects of Markdown and I kept missing features for writing documents.
I have written a thesis in rST and it worked very well, even though Pandoc at the time did not understand citations and arbitrary document internal references (both of which standard markdown does not have) in rST well enough. I had to write my own pre-parser for it. I could not have written that thesis in some markdown dialect incapable of expressing things, that one simply needs in academic writing. There was Pandoc Markdown, but it felt bolted-on and rather ad-hoc in comparison to rST, which brought all the things out of the box.
I remember also looking at AsciiDoc at some point. Not sure, why I did not choose it.
Then I discovered Org-mode. It has been a journey to uncover more and more of org-mode capabilities. Org-mode does not suffer from the same eco-system split up as markdown does. It does not have this bolted on feeling, that many markdown dialects leave me with. It usually has everything I need. In fact, I have written technical documents in it as well and have discovered things like literate programming. The tooling in Emacs is so good, it is really nice to work with org files. Good, that I can still export to primitive markdown, for anyone not knowledgeable in org-mode. Even though org files are merely human-readable plain text documents. It is unfortunate, that VCS hosts are unable to render them properly.
I would really dislike going back to markdown only. It feels so limiting to me now, that I have developed an aversion against writing documentation in an unsuitable format like markdown (talking standard markdown, probably commonmark). If there is already markdown only documentation, then well OK, I guess I can write it, although it surely will not feel great.
Can't speak for anyone else, but for my part there is animosity towards any format, period. Documentation should be plain ASCII - or, when strictly necessary[0], UTF-8 - text, and readable as such with only human-generated ad hod syntax such as *emphasis* or
+------------------+
| ascii-art tables |
+---+--------------+
| + | corner |
| | | side of cell |
| - | bottom/top |
+---+--------------+
Admittedly, I'm not necessarily a repesentative example of anything.
What I see as disadvantages are: It is not possible to render it properly or have support for it beyond showing plain text. Many users might write things different from each other, which might affect how easily a reader picks up what each document's style or conventions are.
Sure you can, and indeed that's what markdown was originally supposed to do (at least as it was first described to me). The catch is that you have to treat the plain text as the authoritative version: you do not edit the plain text in order to produce changes in the rendered document; you edit the plain text for it's own sake, without considering the effect on the rendered version, and the rendered version is derived from that.
Missed this and too late to edit.
0: Which historically meant "not stupid-quotes", with a addendum for stupid-dashes, stupid-ellipses, etc, but now also includes "not emoji".
The most annoying thing about Markdown is that most of the time (90%, I'd say) it's added some kind of formatting, I was not trying to format anything at all.
One of my projects[0] builds and deploys a static documentation site[1] on every push to master. The static site generator (Nanoc, in this case) imports the library and uses it to publish its own documentation. All the examples are snippets of code[2] that are both displayed as-is and eval'd into the final output.
The guide can never be out of sync with the library.
[0] https://github.com/dfe-digital/govuk_design_system_formbuild...
[1] https://govuk-form-builder.netlify.app/
[2] https://github.com/DFE-Digital/govuk_design_system_formbuild...
Include hashtags for topics, e.g. #authentication, #language, or #netscape.
Use the text file indexing and management system to browse and update the documentation.
You can have the programmer write bad documentation and file an issue for it to be improved. You can then enforce that releases don't go out until those issues have been resolved by the copywriter.
You can also implement new features in a branch with multiple authors. The branch doesn't get merged until the documentation is in good shape.
The copy editor updates the repo and while their changes won’t be in the same commit, they should be nearby.
So you still get the benefit of docs history, and being in the same place.
The biggest issue I've see is management and product who seem allergic to the actual repos for some reason.
I have a small disagreement on that. I always feel documentation should be outside of code as it might need to be reviewed by people who do not have access access to the code base. Then there are also the "Why are we doing this" part of documentation that is difficult to mark out in code. Documenting in code is great for addressing "how are we doing this" part IMO.
The higher level strategic stuff can absolutely live elsewhere - in my experience Google Docs or some kind of company-wide wiki often come into play here.
Though there are reasons to do both(some docs from the source itself, and some from docs/) but that's a different debate.
How do i build this? Run the build command.
How do i test this? Run the test command.
How do i run only the unit tests? Run the unit test command.
How do i start this locally? Run the start-local command.
How do these components interact? Run the contract-test command.
...
That fixes "reference" type docs better than any reference doc but there's still a place for technical guides around a code base but short screen recordings voiced over by an experienced dev on the project navigating their IDE will beat any written guide on any metric (time to write, usefulness etc.)
If it's customer facing docs, treat them as code and host them inside the application in some way. There's few things worse than reading the wrong version of a doc.
If you have multiple active versions, like you often do with software released to run on someone else's machines, coupled documentation "works," but has a big downside. Namely, it prompts people to "refactor mercilessly"/change everything all the time, together with the documentation.
When you need to maintain multiple versions at a time, having a single version of the document explaining the differences between all the live versions can somewhat curb the enthusiasm for gratuitous changes (since whoever does the changes must also maintain the increasingly long and ugly description in the single document describing all the versions.) And someone needing to work with all those versions has these differences nicely laid out and those areas not having differences also clearly visible. Whereas with multiple versions of the document you need to "diff" these versions if you want to build a mental model of what changed.
Sadly (for those agreeing with this), I presume that the above is a minority opinion.
I enjoy reading a nice documentation website maintained by the open source organization; it also gives me a touch-point with the organization which created the library. I also agree with your nuanced argument concerning incentives. Arguments related to incentives are almost always discarded by managers but they are very important.
I do think decoupling the documentation encourages people to think about documentation more carefully as a distinct and important activity. I find that in-line documentation tends to be neglected; as a developer, when you're in the middle of coding an important feature which requires your full attention, you don't want to be distracted with updating the comments all the time because it breaks your train of thought. Usually developers tell themselves that they will do it later and they often forget. Comments are often neglected in the PR review process too.
There is no way around it, you need to set aside some time to write or update the documentation as a distinct activity. There is a time for coding and there is a time for explaining.
Also, if the library is a sub-dependency which the developer doesn't interact with directly, why should they download the documentation for it? They will never read those comments in the code anyway.
Docs are like code - the less your write of it, the less you have to maintain. Documentation should be treated as inherently evil. The only worse thing than no documentation is documentation that is not maintained and out of date. There is nothing more infuriating than following the docs only to find out from someone later that it was antiquated. Why is it there?
There are common sense rules. Why would you have the docs on how to set up a fresh checkout NOT live with the checkout? How would I know it lives somewhere else? How would anyone update those steps if those steps were not code?
That's a fantastic feature for a popular open-source framework, as it means the documentation remains up to date.
I'm not sure at what point it's worth the effort for an internal project, though. If you have a cultural problem with incorrect, out-of-date or missing documentation this could make things worse. I'd look for the root cause of that first (training, motivation?), before trying to enforce it with technology.
As a security tester, I can't even count the number of times I've gotten API documentation with omitted info like perhaps-trivial-to-them-but-blocker-to-me how to actually authenticate against the API. If they actually had it running somewhere, that sort of thing can't be missing. (Also, the number of times I've asked for API docs in an API-only test and they go "umm, let me task someone to write that real quick"... like, what did you think I was going to work with, balloons and thin air?)
When appropriate, I'll definitely be recommending clients to include their documentation examples in testing. But they will probably ignore it like the rest of our non-high-risk advice (today's 'low' findings are tomorrow's stepping stones for ransomware).
Everything can be made into code, but at a certain point it's just so complicated to do that you're spending more time and money automating your docs than your applications. So until we have solutions for all that, you will have to maintain some docs manually.
For those manual docs, how do you keep them fresh? I've thought of automatically sending an email to warn that in 30 days the document would be deleted unless someone updated it, but even if people agreed to such a system, they could just update some punctuation and it would remain stale. Even blank pages, people seem to want to keep around rather than fix.
How do you navigate your docs? Search engines actually suck for the most part. Search is a hard problem to solve, and a home rolled search will usually net terrible results. On the other hand, most people don't have the time to maintain a governance structure for their docs, much less an enforcement mechanism, so the docs invariably become terribly organized.
People also seem to need training to learn how to write good docs. I know there are some trendy pages being passed around about some kind of "golden framework for docs" but they don't explain how to write them either. I know how to write docs, but I feel like I'd need to write a whole book to get it across. One thing I found really useful was Atlassian's newer Confluence page templates, which come well organized and primed with examples of how to write the docs.
As a philosophy, I really, really love GitLab's Handbook First model. Their handbook is incredibly detailed and covers pretty much their whole organization, and is fairly easy to update. I feel like this one one of the magical missing links in getting more documentation for the important things that aren't code.
Of course I also enforce that contributors must add/modify documentation at the same time as the code, but that’s easy, because if you modify most of the code, the documentation is also right next to it.
In theory it should work like this:
I) "User should be able to do X" <- requirement
II) "X can be achieved by performing steps A, B, and C" <- description of the implementation (high-level, user-perspective)
III) "A works by using components 1 and 2" <- technical documentation (design-level, architecture perspective)
etc.
I) generates your index (what can I even do with the software)
II) generates the documentation (how can I do it)
III) and below is for technical use only (extending, modifying, porting)
Stuff like rationales for design decisions can be structured in the same layered way.
I don't know how something like this can be extracted after the fact, but no matter the development model (waterfall/agile), a structure like this should arise naturally anyway and the absolute amount of documentation isn't a problem. Lack of proper structure, however, is.
The Python ecosystem has been building docs this way for over 10 years now, and it works great.
Perhaps it’d be fair to rename that article “The Case for Continuous READMEs”.
On the other hand, in-code documentation is hard to follow because it's scattered all over the source code, relies on special IDEs (more corporate lock-in) and developers often forget to update the documentation anyway (even more easily than they would forget to update the website). Not to mention that it takes up a LOT of space and requires more scrolling; IMO this has a negative impact on the readability of the code. Well written code is simple enough that it doesn't need much in-line documentation.
I don't know why, but these days, when it comes to software development, I find that I disagree with 90% of all the top links that make it to the top of the HN front page. A lot of the practices which are being advocated are inefficient, bureaucratic and they seem to align with corporate interests as opposed to developer interests.
The agenda seems to be about making developers more reliant on proprietary tools, IDEs, subscription SaaS services - All at the expense of free software principles.
There is also an agenda around making developers more reliant on teams and less independent in the software development process. I remember coming across some outrageous claims such as "Good full stack developers don't exist". Also there is a push towards monorepos and other corporate structures which limit the degree of possible decentralization and autonomy of different projects and their dependencies. The shift towards static typing is also part of the trend towards centralization, de-modularization and high inter-dependency with proprietary tools and services.
It's kind of ironic that tight coupling used to be considered one of the main signs of low-quality code but this concept is barely mentioned these days and the agenda is to promote it without saying outright what is going on.
A very common one is when a library was written before some new Type/Interface was introduced as part of the core language and the library had invented its own abstraction which does the same thing... So the interface exposed by the library became redundant. Statically typed libraries require a lot more maintenance and this may also explain why companies are increasingly pushing for a monorepo structure which facilitate this constant maintenance which would have been unnecessary with a dynamically typed language.
It's not particularly sophisticated - just some tests which introspect the code and then use dumb pattern matching against the documentation text to check that different concepts from the code are mentioned at least once in the docs: https://simonwillison.net/2018/Jul/28/documentation-unit-tes...