Comment by franciscop | Hacker News Reader

Comment by franciscop | Hacker News Reader

I had an "oh f_ck" moment around docs in my OSS project: I made a small mistake. I wrote "@include" instead of "@import" in the homepage of a fairly well-polished project, https://picnicss.com/. Now, I don't want to imagine how many developers copy/pasted that snippet into their code, saw that it didn't work and blamed their code a hundred times until they ultimately gave up and left the library or SCSS altogether.

But it wasn't until a poor soul opened an issue in Github (which made me feel horrible) that I realized how much responsibility people writing the docs have. So I decided I had a to do something for my next project, and I did. Every snippet of code should be tested, see this as an example: https://serverjs.io/documentation/context/#-options

    const mid = ctx => {
      expect(ctx.options.port).toBe(3012);
    };

    /* test */
    const res = await run({ port: 3012 }, mid, () => 200).get('/');
    expect(res.status).toBe(200);

In the build process, that snippet will be captured, the test extracted and run. That way, as a one-man dev, I can make sure that my snippets are tested and working automatically. I've gotten busy since then and haven't had the chance to refactor the code or test all the examples as I'd love, but many of those tests are still there, making sure the code examples are working.

That's cool that you've found a way to test the code that's embedded in your docs. Google has infrastructure for inserting code from a GitHub repo into docs [1] but I haven't heard of anyone extracting the code from the docs and testing it.

One thing that you might want to look into is conditionally displaying parts of the code. For example, suppose that for a given doc, you only need to display one line of code. But to get the code runnable, you need to put in some boilerplate. There should be a way to only display the one line of relevant code.

    // START_EMBED
    fetch("example.com");
    // END_EMBED

    // Boilerplate infrastructure that the user doesn't need to see.
    new Test().run();

One advantage of the Google approach is that you write the code once, but can insert code one line at a time into the doc. For example, suppose you start off with the full code sample.

    const service = new Service();    
    service.init();
    service.doStuff();
    service.terminate();

But then you want to discuss the first line:

    const service = new Service();

Then the second, and so on. The Google markup looks something like this:

    // START_FULL
    // START_CONSTRUCTOR
    const service = new Service();    
    // END_CONSTRUCTOR
    // START_INIT
    service.init();
    // END_INIT
    service.doStuff();
    service.terminate();
    // END_FULL

And then to embed a certain snippet into each doc you just do something like this:

    {% includecode "https://github.com/.../code.js" snippet="FULL" %}

[1] https://news.ycombinator.com/item?id=17835045

That is very nice idea as well. I didn't consider that, but definitely I could pull tests from the test files and display some parts of that. I wouldn't make the full START_FULL bit, it'd be a bit overkill for my use case/resources (or I'd simplify it to something like <test-case from="1" to="4">).

I think it is important where you start a project as well. Many times I will write the docs even before writing any code (totally or partially). If I think the docs and snippets are going somewhere I'll continue with the code. So in a way, it makes more sense for me to pull the data like docs -> code -> tests than the other way around. That way I know where I want to go, and then I try to get there. Examples of my projects like this [1] [2].

I'll very likely try it though, so thank you!

[1] https://superdom.site/

[2] https://github.com/franciscop/magic-promises

Rust has a cool feature where all examples in doc comments are automatically run as part of running the tests of the package (`cargo test`). So when your CI is running tests it's also testing your examples.

There's even some built-in magic to make it nicer. Like if the example needs some code to compile but which would otherwise distract from the point of the example, that code can be prefixed with # and will be hidden in the generated docs.

I've done this, 100% tested documentation for a complex project, and it was a nightmare to maintain (eventually abandoned).

Maybe it makes sense for simple projects, but for my project it was an utter disaster.

It shouldn't be an incredible nightmare as long as your toolchain is set up to handle this. Realm.io's documentation followed the "docs as code" notion of having documentation under revision control[1], but took it a step further -- the docs repo was tested by CircleCI every time you pushed to master, and the tests extracted the Cocoa-based examples and ensured they compiled without error. During the time I was there, they hadn't built an equivalent for the JS and Java examples, but it should certainly have been possible.

[1]: "Docs as code" sounds obvious to people at startups, and we did this at RethinkDB and do it where I am now, but it's relatively new to the traditional technical writing world. (I wandered in here from webdev, so it made sense to me right off the bat.)

The problem was that all the interesting examples had nontrivial state that couldn't be mocked.

Reply on news.ycombinator.com