The Opposite of Documentation is Superstition (2020)
buttondown.com
buttondown.com
Now, for some interfaces, this isn't too bad. Most people don't know why US AC power plugs are polarized, or what the ground prong is for. Electricians have to, but users do not.
For more complex interfaces, it means that many functions will either be misused or undiscovered. This is the source of the plaint that only 10%-20% of a product's features are used.
On user interface design, the classic is "Tog on Interface", Bruce "Tog" Tognazzini, 1992. That's from the Mac UI era. A more modern take is "The Gamer's Brain", by Celia Hodent, designer of Fortnite's UX.
Software internal documentation seems to suffer today from a mindset that comments are unnecessary and waste space. Especially in the Javascript era. It's worst in languages that don't have data declarations. There's no place to properly document the data.
Rust has good conventions and tools for documenting data. There's a well defined place where the documentation for each structure and field goes, and reasonable tools for checking it and turning that into documentation. If you fail to do this, when you publish your crate on crates.io, the documentation pages will come up blank, which screams "loser".
Rust is weak on function parameter documentation. There should have been a defined place where each formal parameter gets a comment, which then appears in the documentation of the call.
Most other languages don't take such a hard line. More should.
I like to think of it as "cargo-cult programming", along the lines of the well-known essay by Feynman.
The people who take DRY and SOLID as absolute rules might create codebases that will be harder to maintain, but probably won’t go up in flames.
I think cargo cults absolutely do work, same as how people pick Nginx “because it’s fast” while they might need to consider that their Python API probably wouldn’t be bottlenecked even by Apache2 (depending on what you’re doing Nginx might still be the right choice, but for different reasons). Similar to how people might lean towards either Java or .NET for a plethora of shallow reasons - either will work for most problems, so the choice will be good even if the reasoning behind it might be bogus. Same for AMD vs Intel or AMD vs Nvidia etc.
Unless you need to be web scale, then the right answer is clearly MongoDB. /j
Have you ever read the manual for your browser which you are using to make this comment? The OS on which the browser is running? The device on which they are running?
I think there are interfaces that need to be usable assuming documentation will not be read. These are of course user interfaces, not necessarily APIs. But if you are creating something like a library of basic math functions I would rather read your code than your docs.
Some of the worst docs I’ve read lately have been for ffmpeg and mp4box. The former sometimes straight up has things like “so one guy tried this and it worked for him so we recommend you try it and see how it works for you”. The latter has like 4 examples with no explanation of what you are looking at or how to make anything else work. The project used Stack Overflow for their docs because nobody can really figure out what to do with it right away. I’d rather take readable code in this case over the terrible docs.
> Have you ever read the manual for your browser which you are using to make this comment? The OS on which the browser is running? The device on which they are running?
Have you ever read the manual for the tools used to build the building that you live in? A manual for the trowels, tape, mud, and drywall that built and makes up your walls? Or a manual for the Romex in those walls? A manual for the concrete foundation of your building?
No? That's because using/depending on those things doesn't require nearly anything of you; you stand on the floors and lean on the walls without thought.
You browse the web constantly, so you do it without thought. Not so with your taxes, despite their similar constance and ubiquity, because they are not constant and thus haven't had the same abstractions engineered into their very DNA.
But even things like buildings/floors/doors with that level of "no need to think about them" only afford that to you based onto their frequent use. Few other pieces of software afford that same consistency. With time though, they will. I think we'll see a lot of software grow to be everore of a "skin round a spreadsheet", especially as we comoditize the few remaining frontiers of UX design (e.g. mobile devices) in the same way as desktop design has converged on common patterns like the chat-log-and-history, the search bar, the photo gallery, the text editor, and the spreadsheet.
I am laughing at ‘remaining frontiers’ of UX design. This year for Christmas I wish for mobile devices with browsers that don’t crash and lose my form entry. This year, I wish for YouTube that doesn’t find new ways to turn on autoplay (TV queue being the latest one I had to turn off, despite having already turned off AutoPlay on that device in settings… each of my devices has autoplay turned off… yet TV queue came with autoplay turned on… crooked YouTube!) But sadly, this kind of thing is not what the state AGs are looking at—-but to me, the constant bypassing of disabled AutoPlay is anti-consumer AF and must be stopped. That is the remaining frontier.
In the UK, the government automatically takes tax out of my income and tax is indicated as part of my pay cheque. It requires absolutely no thought from me.
For niche situations, yes, e.g. different browsers having different keyboard shortcuts for hard refresh. It's quite common for people to assume they know how to do it in one browser because they knew the shortcut in another, but when they just get a normal refresh they don't realise their error.
Software written to be used by billions of people per day must be written to have a very low barrier to entry, and even if it contains sophisticated and complex features they must not be essential for >99.99% of the software's overall usage.
The downside of this approach is that many people never learn about features in everyday software that would make them much more productive. Keyboard shortcuts as above are the classic example and are almost always documented, but there are many more unexplored features and not all requiring significant memory or technical skill to make use of. I've not heard of a regular office job in the last 15 years where there would be induction training on the core tools (OS, browser, office suite). It's assumed that you're not employable if you don't know how to use them, and by implication that knowing how to use them is a binary thing rather than a spectrum.
Talking about software specifically: this is the exact wrong way to do it. A programming interface should be easy to use and "self-documented" to some extent (by, e.g. using the type system as much as possible to make bad states unrepresentable), but only very simple interfaces are usable that way. For most real-world interfaces, there's just no way you can encode all the "rules" in the type system and the names. You absolutely need to document how your interface works in general, what each part is used for and expected to do, special cases and so on that are just impossible to know otherwise. Now, if you just read the implementation instead (and notice that you would need to read all implementations, most interfaces are likely implemented in many different ways) you're not coding against an interface at all. You're assuming the implementation is the interface. You're ignoring what the implementer intended to be private and subject to change. Don't do that.
Can't really put finger on it precisely, maybe its too large ego (my time is too valuable for some pesky documentation for idiots, I am a superstar! No you're not, you are at vee best a brilliant junior...).
Or maybe lack of basic empathy, surprisingly high number of humans can over time learn how to work in society but any novel situation requiring some empathy they handle like mammoth ina porcelaine shop.
Or something else in similar vein. That was quite shocking, entering adulthood and discovering vast number of folks have such issues
Usually a long comment is a smell. If the way something works isn't obvious and must be explained, it can probably be better designed.
Code changes and then comments have to be updated, and often aren't. If they aren't just wrong, the context is often so changed that they aren't useful. You wind up reading the legacy code anyway, and you and the author both wasted time with comments.
My team is smart and competent. I need them to read the code to make sure their changes are safe, anyway. They do not need their hands held. They don't need me to waste time writing comments that will soon be outdated, or that explain something I should have designed better.
Finally, management doesn't apply an OKR to intangible code metrics like internal documentation. Should they? I don't know, but it's their code, not mine, so I'll write it to the standard they apply.
These are the reasons you have to work in code that is poorly documented. It has nothing to do with a lack of empathy -- if I were to allow my employer to extract extra value from me by using my empathy for my teammates as a lever that would be exploitation. If I spend more time on documentation -- which the employer doesn't value -- instead of functionality, I'll be punished at review time for wasting time on worthless things.
Maybe you should stop letting your company manipulate you into doing more work than they're paying for. If they value internal documentation, they'll create an OKR or KPI or whatever to make sure you're paid to do it. Otherwise it's unpaid labor for a for profit corporation. I'm not doing that. Are you?
But in most cases explaining what your code does is like explaining a joke: if you have to explain it, it’s probably not very good.
I just don’t trust the docs to be accurate anymore. The implementation contains an accurate description of what it actually does.
I still read the docs… with a grain of salt.
Even stellar code just tells me whats going on. Doesn't tell anything about why its happening, what consequences does it have in other systems, what are the overall constraints of this domain etc.
I agree generally too long comments aren't best, or even just non-flexible rules. But denser the logic is, the more spread its across multiple systems and environments, the more info the better.
And to be changing some dense code and completely ignoring comments just next to it describing them... that's just rude to be polite. Shows who is a team player and who isn't. Trust me, if you want to make friends and have some respect in some long term team, this is one sure way how to do it.
The fix? Stop leaving it optional. Tie documentation to performance reviews, bake it into deliverables, reward teams for knowledge-sharing. If you can’t convince them with logic, align incentives so ICs can’t ignore it. It's either that, or accept that you believe ICs have better things to do with their time. But make a conscious choice either way, I'm begging you!
In Rust however you can get away with way less documentation because the type system and signatures are self explanatory. And if you're doing things right, the set of values you can pass to your function is limited.
Something like Hoogle becomes a necessity if you want your type system to replace the need for documentation.
They key point to me in TFA: "The feature is a black box"
This doesn't apply to software internal documentation. It's not a black box and you can give any deep you want to know what is and isn't implemented, what has explicit tests and who wrote it.
Whether comments on those are a waste of time of not will come down to how long it will take to write useful comments, how long it will take for the reader to understand the class/function/method otherwise, how much maintenance will happen on the comment itself after it's written, what it will cost when it will "rot", and what else (e.g. tests ?) the dev could do with all that time.
Typescript is very common too. It has all the types needed to document the data. And vscode understands all the references and navigates through them by default.
That's not a complaint, it's an observation, and it's not due to discoverability. It's because products are made for many kinds of users with many kinds of needs, and their needs often don't overlap a ton.
I probably don't use 99% of what ffmpeg can do, but it does everything I need it to. And with word processors it's still the same principle -- I've never in my life needed to do a mail merge, or apply a watermark, or 100 other things. But other people do. And it has nothing to do with discoverability.
Maybe because I'm a child of the 20th century that I value documentation that gives me a path from understanding what and why to learning how I can use it, with a master index so I can look up exactly what the “Mung Until No Good” command does, and see which other commands relate to it.
Perhaps one day generative AI will be good enough that we can feed in one of these websites, with its pages with titles like “Migrating from V4.7.2 to V4.7.3” and “Building for OS/2 Warp” (I exaggerate, but only slightly) to documentation that is useful for learning, use, and troubleshooting. I live in hope.
I've found that it is sometimes a good idea to reinforce mental models that don't actually reflect the operation of the software (in my case) under it.
Documentation is good, but, in my experience, almost no one ever reads it. I can tell, because of all the questions that I get, that are in the docs.
That's totally my fault. A really good tech writer could probably do a better job of conveying information.
That said, I think that it's important to present good, discoverable UI, that doesn't punish users for exploring.
But the actual documentation of the tool has room for improvement. I needed a YouTube video to get started and that's rare for me.
So what I want to say is, that you need an intuitive, discoverable UI, but also a documentation that has each case (and if it's just for linking in 1st level support cases) _and_ is discoverable. And by that I mean both easy to grasp (e.g. following https://diataxis.fr/) and also can actually be found. I've had cases where a tool had good documentation, but actually finding it was the hard part.
(maybe you eventually want a bath tub and a toilet in each room? maybe not?)
It really depends on the nature of the project, but UI design often requires a lot of "Paving the Bare Spots"[0]. It's really just too damn complex and counter-intuitive (or too intuitive) to catch in Requirements.
Software allows us to iterate this incredibly quickly. Hardware design also does it, but at a much slower pace, and a much greater cost.
[0] https://littlegreenviper.com/the-road-most-traveled-by/#pavi...
This is different than commercial work where in addition to the original plans, you also get as-builts, which can be expected to be accurate, and are expected to be updated.
If you only want to document once, it makes sense to do it once the thing is built, rather than before, because there's a good chance the actual thing will be different than the plan. If you will update it, it might make sense to start documentation before the thing is built.
Of course, if you never get around to writing documentation, it never needs to be updated.
I wonder if LLMs can generate docs at any reasonable level of usefulness and accuracy
Speaking of which, I just tried using ChatGPT to create a React app with typescript and rollup. What it generated didn't work (got errors). Of course I could tell it the errors and it fixed them but it suggests they aren't yet ready to write accurate docs
In fact, "superstition" very neatly defines most prompt engineering for LLMs.
Its weirdly fascinating how this is developing. Like some people boldly claiming that if you offer to tip your LLM it does better. And if you say things like "Think hard and check your work" it gives more accurate results.
Works pretty well. Could still [theoretically] go out of date, but that hasn't happened, so far.
The one thing that does happen, though, is copy/pasta errors. I'll copy a bunch of code, including the headerdocs, and paste, but forget to adjust the docs.
You need some overalls documentation to explain the concepts and how to put things together. When the user understands that you can use in-code documentation but even then only if you need to use one function at a time - there needs an explanation of how to put things together.
With Apple's docc, you can drop READMEs throughout the system, and they show up in Xcode's doc browser (and can be referenced from the code).
I usually create full-fat test harnesses[0] for my stuff, as that gives a totally realistic implementation example.
But I think having a well-written README is more likely to be useful.
API/implementation documentation (whether docs or tests), are much more likely to be read, than end-user docs.
LLMs can put into words what a programmer can infer from struct definitions, function signatures, names and even taking a cursory look at a function body.
LLMs cannot retroactively find out the _why_ of design choices. If you don't document the reasons behind decisions while making them, they'll be lost, which is exactly what leads to superstition. As a result, changing code is error prone because you never know when code is written in a particular way for a good (and still valid) reason or just because.
A lot of times documentation isn’t read by anyone for stretches of time till it’s needed. I imagine documentation by LLMs will make the situation worse if nobody verifies its corectness. I don’t believe in magic solution, good documentation that is not out of date requires work and if a company doesn’t allocate resources no magic will fill in the gaps.
The docs for SwiftUI are ... less than ideal.
I'm pretty sure that it can do what I need, but the docs don't cover it at all. I'll need to keep playing with code completion (which kinda sucks, right now, because Xcode keeps making up nonexistent APIs), and looking at the actual system exported headers.
I'll get it, eventually, but this is pretty crazy bad.
I needed to have charts that you can pinch-to-zoom.
A) insufficient docs B) programmer can't be bothered to read the docs C) programmer doesn't trust the docs D) programmer had a bug once, blames the underlying library, not his code E) programmer had a bug, changes code, bug goes away, jumps to conclusion.
Trust in the docs is a big thing. If the docs are wrong a lot the the programmer is skeptical even when they're right.
But I've also noticed that most developers simply don't read them. "I'm too busy (fixing bugs) to read the docs." Or "my backlog is so long I don't have time to learn how to do it right, I just want it done fast "
Superstition occurs with the absence of knowledge - the existence of docs is necessary, but not sufficient, in overcoming that absence.
In the past I made an effort to address this by creating a unofficial but quickly searchable document repository documenting every question I have gotten from a teammate by either answering it or pointing to where the authoritative answer could be found.
What I found was the following:
1. Questions are repetitive, you can quickly build a base of tribal knowledge and point people to the answer. 2. If the document tool is quick and efficient, team members will generally check it first instead of asking. 3. Sometimes document tribal knowledge conflicts with someone else's tribal knowledge. This is a great opportunity to hash out the more correct answer and update the document. 4. Tribal knowledge gets stale fast and needs constant curating. 5. The less friction to find an answer, eg fast access and fast/accurate searching, greatly increases success.
NB. GPT is a perfect front end for such an effort.
In my last job, I used a blog to write informal reports. I set pretty loose rules for the blog for my team. We would post meeting minutes, outage postmortems, scripts, tricky one-liners, general lessons learned, and so on. We used SharePoint, which is kinda meh but it works. The "Alert Me" feature in SharePoint made it easy to subscribe my teammates to the blog via email, so you got the benefits of email (push notifications) without the drawbacks (employee mailboxes are unsearchable once they leave).
I believe email contributes to this tacit ("tribal") knowledge that the OP names. I think my blog went a long way to counter this effect. A major goal in our blog posts was to capture the "why" behind our decisions. Often, we fail to remember the nuances of our own decisions in hindsight. I always thought it was important to name our present priorities and known-unknowns.
The trick is to get people to get their tribal knowledge down in public Slack channels or their docs system.
Recording most Zooms with AI summaries may help.
If the workers values it, it would likely be documented knowledge. Undocumented knowledge can be job security.
You're mostly right. It's paradoxical in some ways. Workers generally won't go through the trouble of documenting tribal knowledge. However they greatly appreciate if someone else does.
Almost every human will do this for something IRL. As semi-irrational creatures, we tend to forcibly seek out patterns where there are none.
Some people become so encrusted with such behaviour that it legitimately impacts their ability to operate effectively in modern society.
Not surprisingly, the religiously-afflicted tend to pack the upper end of this spectrum.
It works with the example given in the article: OneNote is proprietary software, and it is documented, though incompletely. Had it been open source, one could see exactly how shape recognition works, bugs, quirks and all. No documentation will give that amount of detail.
Sure, not everyone has the time and skills for that, but maybe someone on stackoverflow does and have answered the question with proof, that is, not a superstition.
No one has time to read the code in depth so it is also mostly reading tea leaves like „oh it has a lot of stars and a lot of downloads so it must be good”.
Don’t get me started on left pad ;)
But if the shape drawing process isn't random, I think the author's experience of feeling unable to articulate the rules AND gravitating to a set of behaviors is a good example of procedural memory (implicit vs explicit).
Explicit rules would probably help speed things up, though!
Programmers have the luxury of documenting the previously undocumented for their repos (assuming management allocates time).
End users would need to reverse engineer but that assumes predictability. Without that you get superstition as per article.
As in, the reason it isn't documented is that no one has a clue...
Welcome to the stupid singularity.
Literal interpretation of anecdotes lead to superstition.
Ironically this leads to them becoming documented.