Etsy’s experiment with immutable documentation
codeascraft.com
codeascraft.com
Consider a page like this: https://firebase.google.com/docs/firestore/manage-data/add-d...
It has 12 "how" sections of code snippets, each with 12 language options. That means 144 code snippets need to be correct for this page to be 100% correct! And we don't auto-generate any of these snippets, since auto-generating idiomatic code is a rabbit hole that's consumed many a good developer.
So we, like Etsy, have come up with a separation of concerns between How and Why that works for us. The text on the page is maintained by a tech writer. The code on the page is a series of blocks that look like this:
{% includecode snippet="android_firestore_write_doc" %}
We have a system that resolves these, at publish time, to match snippets on GitHub. For example: https://github.com/firebase/snippets-android/blob/5f94890d7b...
These snippets can be maintained with all the wonderful tools that exist in the world for maintaining code! We don't need to think of them as just docs. So that "snippets-android" repo has continuous build, a robot that updates dependencies, PRs/Issues for third-party contributions, etc.
What we've found is that this system allows us to maintain many thousands of code snippets with only a handful of people watching over them.
Feature request, though: include a url in the source code that takes you to the snippet's usages in documentation. Something like // [START https://codesamples.google/firebase/set_document ] or just the url as a separate comment.
Disclaimer: Nothing to do with them, just at a company that uses it and finds it very good at this part of docs
The wikis I've used also have slow editing interfaces because they have separate read and edit modes. Does anyone know of a modeless wiki, which probably means one that allows you to edit-in-place?
The problem with modes is that to edit something on a sizeable page you must,
1. Load the page in read mode
2. Read/search to locate the spot on the page where your edit goes, read the context around it, and plan the edit.
3. Load the page in edit mode
4. Read/search to locate the spot on the page where your edit goes
5. Type your edit
6. Reload the page in preview mode (a form of read mode)
7. Read/search to locate your edit and verify it
8-?. Repeat steps 5-7 until your edit is correct.
N. Reload the page in read mode.
Compare to a modeless wiki:
1. Load the page in read mode
2. Read/search to locate the spot on the page where your edit goes, read the context around it, and plan the edit.
3. Edit until satisfied
In Rust (and many other languages), when you document a function, you usually add a code snippet that explains how to use this function. That code snippet will be treated as a unit test and ran when testing the functionality. In this way, documentation is always up to date (assuming the tests are ran).
Why can't tool documentation be the same? The tool documentation usually contains a sample tool invocation. All we need to do is run the tool in a controlled environment, and ensure that the invocation runs successfully. When a tool gets updated, run the "documentation tests" to ensure all the commands in the docs still run successfully. If they don't, then update the docs. Surely, there are tools out there that already do this? Does anyone know?
It's designed so that you can write tests that are the basis of the "how to" documentation - with a small script and some jinja2 you can generate readable documentation in the form of markdown from the test:
Original test (behavioral example) : https://github.com/crdoconnor/strictyaml/blob/master/hitch/s...
Example docs generated from that test : https://hitchdev.com/strictyaml/using/alpha/scalar/email-and...
It's also designed so that the testing framework can be programmed to rewrite the test automatically for simple code changes (e.g. changing the wording of that exception).
This makes building tests much quicker and much, much less tedious and perfectly in sync docs almost becomes a side effect.
Essentially "how to" documentation and example driven behavioral tests are the same thing for all projects. It's just that most people don't realize and don't treat them as such.
"What do you want to do?"
o Get a list of databases
o Connect to a database
"Do you want to connect to dev or prod?"
o dev
o prod
o I don't know (this option could link the user to documentation on the different environments; maybe the user is a student on his or her first internship and has never even heard of the concept)
"What's the shard number?"
"Which side?"
The CLI could output the equivalent, non-interactive command before performing the requested action, making it a teaching opportunity as well. Maybe the "I don't know" option or this approach in general is overkill, but I generally think tools should be smarter. You don't need as much documentation if your tools are inherently capable of guiding you to the right answer.
Point taken, but I don't think it should take that much effort to implement. Some CLI frameworks already have built-in methods for interactivity.
2. A tool's popularity/usage could be a way to judge when it's worth the effort to add this sort of system.
3. With the Etsy approach, I would also be wary of the cost of interruptions. Getting constant Slack notifications is not conducive to getting work done. You could silence them, but then what if someone can't move forward without getting an answer to their question?
4. There is a nice balance to be found between wasting time documenting everything possible upfront and not documenting anything until someone asks a question.
5. Personally, when I find documentation that needs to be updated, it's usually as simple as editing a wiki or editing Markdown and making a pull request in GitHub's interface. To me, that's not much effort.
Any system inevitably acquires dependencies. Maybe it's fine to break those dependencies if you're building some public website where you can always have the latest version of the code running, but if you've got customers who depend on having a stable build which was initially branched three years ago and you still need to build, run and test that, you damn well better have a consistent and stable mechanism for dealing with that codebase which spans multiple years. Your ops and QA people will hate you otherwise.
Our own approach has been to pull out the coarse-grained operations and have an 'environment' script which examines the version it's working with in order to spit out relevant details. The 'porcelain' scripts are never allowed to change their API in any breaking fashion, and their behaviour is adapted based on the environment's output.
One of the big picture value propositions of technical writers IMO is that we're shepherds of organizational knowledge. For example, if someone sends an email to me asking how to get a particular task done in Chrome DevTools (which I write the docs for), I will ask their question on Stack Overflow, provide an answer for it, and then send the link back to the asker. It's a similar amount of work, but I'm making the knowledge public, rather than one-off.
What's really innovative about Etsy's idea IMO is that they have found a way to make tribal knowledge shared on an ephemeral platform like Slack more permanent. My big issue with real-time chat up to this point has been that all the beautiful knowledge that you share on the platform isn't recorded in a permanent way. I acknowledge that a lot of people prefer the real-time messaging format, so it's really cool that Etsy has figured out a way to get the best of both worlds.
Blog posts like this are always optimistic and celebratory. Sometimes it's warranted, sometimes it isn't. I'd love to hear how people on the ground at Etsy actually feel about it.
We got runbooks/ - these are hard about how. And they are prone to some degree of decay, I won't lie about that. Thus, runbooks/ is encouraged to be changed, extended, modified, clarified. runbooks/ isn't anything pretty, runbooks/ is the oil- and coffee stained notebook with more loose paper than bound paper somewhere on top of a machine and we're all going to die if you lose runbooks/.
And on the other hand, we have doc/. doc/ doesn't change without a lot of discussion, and doc/ defines hard standards. doc/ won't generally help you with an immediate problem. doc/ explains why things are defined the way they are - or if this is just an arbitrary naming convention.
While I agree with you, the latter problem is particularly frustrating to encounter.
Solving the conceptual problems feel more like an exploration or a learning experience, and I rarely feel like time spent solving those problems is wasted even if it's very meandering.
Solving the "how do I do this?" problem rarely feels like more than a very isolated learning experience, and more often it feels like banging your head against a wall. Doubly-so when the documentation is obsolete and leads you in the wrong direction.
Learning the basics of media compression is interesting and worth spending time on. But I'm going to hate myself if I have to spend 6 hours figuring out how to convert a file using ffmpeg.
Here's an example: I need to generate a report where I need to cross-reference data from A and B. I need to know where A is going, where B is going, how to modify the schema of the events going into B to add a specific piece of data, where we can process inputs from A, how to do the join, and how to add the report into the dashboard.
On the one hand, I could ask the surface level question "how do I generate the report for ____?" but really there's a whole lot to unwrap, and what I need is the information to do the unwrapping. I don't need a course on statistics, I just need to understand our specific architecture.
The answer (from my experience at Etsy, anyway) to questions like:
> how to modify the schema of the events going into B to add a specific piece of data,
was "Run schema_modifier", the docs for which were frequently obsolete. Figuring out how to "run schema_modifier" was usually a very frustrating experience in that particular instance, whereas unwrapping the actual problem was usually less frustrating.
The thing I suggest here is that if you are going to document the function, it should only be the WHY. The how can change and the documentation doesn't really decay. If you are making a new function you have to start with the why and if you can't then you shouldn't be adding a new interface. This also helps build a better picture of how the pieces interface together.
The other is when someone write “if you need to do X” with no information about how you’d know if you need to do X
I would barely consider random snips about a thing/keyword, scattered over time, to be documentation.
Certainly, a problem was likely solved with this FYI system... but it wasn't a documentation problem.
I also prefer the 'documentation is code' approach, but it doesn't solve the same problem.
see also: BSD-derived os's commit history, etc.
A more robust implementation would parse the code to object-code, then emit a document(s) for the developer to fill in.
The killer feature of such a system is that non-code changes (whitespace, etc.) would not trigger a need to re-document, but seemingly benign changes that affect distant parts of code (or a -O4 being added to the Makefile) would trigger a round of documentation
I do agree that this is a useful distinction.
I'd have called how-docs and FYIs "howto", as I think that term would be recognised more clearly, but that's a quibble.
Immutable documentation, at its essence, isn't new. Wikis are in many ways immutable documentation due to the diff. The interesting thing here is trying to organize the "small bits of hard to organize" information around slack. More often, however, the real problem is that we need to properly organize this documentation at the point the organization becomes more evident.
For a tool, that should be obvious: --help should be obvious. It's not true in all cases.
This is a great way to articulate something I've been trying to talk to people about for a while - is this something you've thought of for yourself, or do you know of anywhere that talks about these patterns?
Sure, getting everyone to maintain docs with their features is useful. But given enough time, you'll end up with so many "close but not really close" concepts in documents it's very hard to stay on top of without someone putting some organization into play.
I can imagine it working for simple one-line commands and their switches.
I can also imagine it failing to work for any application that requires multi-line code blocks and/or more detailed how-to examples.
For example - as a hypothetical, imagine you change your API from Oauth to Oauth2, and you have to provide examples of new standard code, perhaps in different languages.
That's very much a how-doc and not a why-doc problem, so supposedly it doesn't belong in the Wiki.
But I really can't see the changes being explainable in a fyi.
Of course you can say "The new code is in repository X instead of X-1." But if there are other questions and fyi's around Oauth2 - and there will be - there's no guarantee that important fact is going to be at the top of the read list.
For complex breaking changes that read list is going to get very long.
Simpler changes are going to have the same issues on a smaller scale.
So this isn't really immutable documentation - it's more of a database-hosted command cheat sheet.
That's a useful thing in itself, but I don't think it's a complete solution for documentation.
If you do that.. then..
You never edit docs. You only add to docs.
Never refactoring your interfaces means that you can never improve the design as you gain new information. It means you can never change to something more appropriate if the underlying use cases change. It means that you can't easily consolidate common functionality in a different place as new similar interfaces are constructed. In other words, you are locking yourself in to a brittle system that will degrade into a mess of spaghetti that nobody can understand.
There are, of course, also downsides to refactoring. No simplistic rule of thumb will tell you your best approach.
If you can do it then great, but it's just not always technically feasible or cost effective.
A lot of times I find that developers lack experience of how legacy applications get the way they do. Either they have only ever worked on green field applications before (a surprisingly large contingent!) or they assume that the crappy legacy code was created because the previous developers were stupid. The reality is that most legacy systems were made with very reasonable decisions at the time, but that the world shifted under them. The mistake was locking themselves into their original decision and making it very expensive to change. The decision to never deprecate old APIs is one of those decisions that makes change expensive. Such decisions should always be made looking at the trade offs in cost. How expensive would it be if we changed this API? How expensive would it be if we don't change this API? Usually the second is far more expensive in the long run, but you have to take it on a case by case basis.
Yes, you should never change your interface.
But docs can poorly written (bad implementation) and deserve edits, just as code can be poorly written and deserve edits.
Maybe the docs are history/execution aware so they can list the top N ways the binary is executed.
Maybe the docs are also team aware so they list the top N ways the binaries have been executed by your team.
So you say, "Here at CoolTechCompany, we built a tool to help control our widgets. Please see the `cli widget` command. And everything kinda works automatically.
With something like this tool it puts it right in a primary communication tool.