Documentation Driven Development (2022)

Writing documentation up front is a good idea, but only if one treats it as the first iteration of the program with the expectation that the initial documentation may well be thrown out. The process of writing the documentation no doubt will help you organize the plan for writing the code. But there will also be things that the documentation does not capture or problems it does not foresee that emerge as the program is being written.

The danger is that either you fail to update the documentation to account for the changes in the system that emerge in development or you only update parts of the documentation as you go which causes the documentation to become inconsistent and unreliable.

A middle ground is write the documentation up front and then rewrite it after the system is done. The initial draft helps guide the design and the final version captures the full and complete essence of the finished program, which is nearly impossible to do up front.

I think that the usefulness of any “X driven development” methodology is so highly variable based on the particular problem being solved and the particular brain solving it, that it’s impossible to make any general recommendation for anything. Like everything else in programming, it depends. Documentation driven development might be great for one person, but for another person it could be a tedious waste of time. Everyone should feel free to find their own way. If it works for you, good. If not, let other people do what works for them.

I think a lot of XYZ Driven Development advice can basically be boiled down to: write down some kind of plan before you start coding.

Certainly one should think about the design before implementing, but I think writing acts to force active reflection on the design - be it a plan, or documentation, or a test.

At this point my approach is to plan out my current ticket into TODOs before I make a branch. I do this in phases until I get down to the level of what functions need to be modified/made. If I know there is a function that is going to be doing a bunch of branching or is otherwise complex, then I know it will need a good bit of unit testing, and I might opt for TDD - though its rare. I'm used enough to planning my tickets now that it is usually sufficient to get to a good design.

I would recommend trying to do some sort of planning of your work. It can be hard and time consuming it first but it can be a superpower if you get into the habit of it.

We are trying out some DDD for an upcoming pigweed.dev tutorial. I'm docs lead for the site and I'm working with 5 or so engineers. It's going decently. As a technical writer, it's hard to flesh out exact content until the features are built out. Especially in a tutorial, where each section builds off the ones that came before it. So we have a lot of placeholders in the tutorial. All-in-all I think it's been fairly helpful for me to repeatedly say: "This will be our main artifact. This is where people will learn of your features. Please make sure your work and features are captured in this document, otherwise they kinda don't exist (because this doc is where people will most likely learn of the new features)." I don't ask them to fully flesh out docs themselves; I'm just asking them to dump notes into the draft doc and insert commands/code as things become fleshed out. I think it's also a useful focusing exercise for the team - there's so much we could work on but if it's not going into our primary documentation artifact and it's not foundational work that is prerequisite to what we want to show in the tutorial then that sometimes makes it easier to decide that the thing is lower priority. There was also an interesting feedback cycle where the team worked on some really important stuff that is usually kinda "behind-the-scenes" and low-level, and we asked ourselves "we have to work this into the doc and make people aware of it... how do we do that?"

For designing APIs, my favorite approach is a hybrid of the two. I'm not sure if there's an official name, but I've taken to calling them walking skeleton tests.

The gist is that you start by writing example client code demonstrating small but useful interactions with the API, including explanatory comments. They won't work yet, because the API hasn't been implemented. But you're already in a position to talk them over with your team and start thinking about user experience issues. And they give you a nucleus around which you can start building high-quality user documentation.

And they're your automated end-to-end acceptance tests.

Once things look good (not perfect), then you can start implementing the API. For this portion I don't necessarily do TDD, because I remain unconvinced that writing unit tests first is inherently better than writing unit tests afterward. But I might. Depends on my mood. But I do keep running the walking skeleton tests, which serve as my primary feedback mechanism for evaluating implementation progress and success.

The more important thing is that I can start stepping through those walking scenario tests and seeing how well the interaction actually works in practice. And I can use those observations to help inform an ongoing conversation about the API's design and functionality. I can also get quick feedback on whether some aspect of the high-level design isn't going to work as well as we had hoped. Ongoing updates to the design are memorialized and communicated by updating the walking skeleton tests.

This discussion reminds me of literate programming. An old idea for a more civilized age.

It also reminds me of clitest[1] (very similar name to the author's lib, but it is another thing).

You can write examples with prose in markdown, then let clitest verify them for you as if they were tests. Best of both worlds.

[1] https://github.com/aureliojargas/clitest

This is actually how software and systems engineering works in life sciences.

There are a set of artifacts which are designated the validation artifacts which includes a user requirements specification, functional design specification, technical design specification, and configuration specification.

Before any release, the team will start by identifying which of these documents need to be updated and which sections need to be updated. This is codified in a formal, signed document.

Then work starts and an updated version of each of the document is created. Prior to releasing the software, the validation process kicks in and basically verifies the correctness of each of those documents in reverse order: the configuration specification -> installation qualification, the functional design specification -> operational qualification, the user requirements specification -> performance qualification (there is no explicit qualification of the technical design). The qualification phase has a mountain of evidence of testing.

The result of this rigor is: 1) the products tend to have very low defect rates because every change is accounted for, 2) the release cycle can be as long as the build cycle, 3) it's very hard to fix bugs "on the fly" as these then need to be documented as a deviation and accounted for, 4) anything that breaks is easily accounted for, 5) it's very, very hard for startups to succeed in this space because of how much rigor is required.

I initially hated it because I wanted to move fast. But after leading a few release cycles, I could see a lot of the benefits as we rarely "built the wrong thing", we rarely had major defects, and work-life balance was generally pretty good because every release cycle was meticulously scoped since we had to start by describing the planned changes starting from the documentation. There is less scope creep once work on the release starts since the start of the release has already identified what will change and adding scope means adding more paperwork after the fact.

The author seems to conflate several different types of documentation, like requirements, code comments, and api explanations. Additionally TDD isn’t intended to work in the absence of any documentation, the concept is that a test models the requirements.

Maybe a paradigm to consider more is:

Literate programming (LP) tools are used to obtain two representations from a source file: one understandable by a compiler or interpreter, the "tangled" code, and another for viewing as formatted documentation, which is said to be "woven" from the literate source.

https://en.wikipedia.org/wiki/Literate_programming

I honestly believe documentation is more important than tests in some cases. The time spent by a dev every time to understand why a method was created that way adds up quickly. The what is more evident from the code.

The idea of writing documentation before the code is interesting, it's thinking out loud

I really love this idea in theory, and I believe that for some system, specially mature ones, it may work well. I see good documentation as a super power; it empowers readers and motivate people to understand more about the system without being caught in the weeds of reading the source.

The source has baggages, and the intent of every single function calls is not always evident. Writing documentation up-front can help direct the source, but this is a tug-of-war environment. Each affect the other in its own ways.

And for that reason, documentation driven development can be a real drag. You start writing documentation with the best intentions, everything works great for this first release. But 2 months down the road you need to modify something and it has a ripple effect on many of the things you documented. It's a non-negligible cost.

I've been working on this open-source tool(https://github.com/pier-oliviert/sequencer) and I've spent a lot of time on the documentation. And what I described above happened. I wanted to make a not-too-big change, and it required me to rewrite 30% of the documentation. I still love the documentation aspect of it, but it definitively has a higher cost than tests, in my experience.

I have lived through all:

Rational Rose UML RequistePro. Doors. Just plain prototype something through many iterations until is good then document. Doxygen

All of the above except the prototyping a waste of resources more or less.

If I had my software shop (which I don't) I would expect a software module to come with a README.md Which a brief explanation of what is is and no more than 3 step instruction for a reader to see the software module running and doing something useful.

Many of these arguements are moot – because for any top-down function definition technique, the How to Design Programs recipe covers virtually all the bases.

- https://course.ccs.neu.edu/cs2500f18/design_recipe.html - https://cs110.students.cs.ubc.ca/reference/design-recipes.ht... - https://docs.racket-lang.org/htdf/index.html

Between a signature, purpose statement and examples, you've declared most of what documentation provides short of a longer contextual statement of the functions role in a codebase.

For larger modules there is How to Design Worlds.

- https://world.cs.brown.edu/1/htdw-v1.pdf

Few can afford not to follow the basic (DO --> SECURE --> DOCUMENT --> ANNOUNCE) workflow. Working in any other order and you end up having to redo some if not most of the previous steps for whatever you where enterprising.

So why can't I use documentation as a requirements spec and just feed it to a LLM for implementation ? With lots of examples in the documentation that can serve as test cases for the generated code.

Except for trivial examples, all real development needs to be undertaken iteratively. Design a bit, build a bit, rework both based on what you found out. Unfortunately most people - and definitely most organisations - don't want to rework anything as it sounds like a waste of time/money rather than an essential step to take working code and turn it into production code. So instead they look for silver bullets. "If you just do the steps in this order..."

Writing out a design before starting to code is often helpful, especially when working in teams.

A high-level design is best not written in code, but in a separate document. Writing documentation in tickets is just plain silly, IMNSHO.

Writing design documentation used to be commonplace. If it helps to put a name on this, then that's great.

Everything is already natural language driven development, but management/execs often like to avoid accountability by prompting their human LLMs verbally off record.

That's mostly how Knuth codes (with a flavor twist in the form of literate programming). It's a huge endorsement of the idea presented in the article :)

I might be missing something, but in my experience as a product manager, we would refer to these as requirements.

> Making sure tests pass can be challenging to handle with the unknowns of implementation detail. After all, if you expect parseInt to act one way and it behaves another, you will likely have to rewrite all tests that worked off that assumption.

Isn't this kind of the point of TDD? TDD makes it harder to write tests based on assumptions about implementation details, making it less likely that you'll make such assumptions. You end up with less unnecessary coupling between your tests and the implementation details of the code they test, which means fewer false failures (tests that fail because the structure changes, rather than the behavior), and a more robust test suite.

bullshit