Doctest.js: a humane Javascript test framework
doctestjs.org
doctestjs.org
Is there anyone here who's been using DocTest and has some experiences to share?
I love how, as a side effect, this approach to unit testing encourages implementing a good toString / __str__ on each class. In those languages that have that, of course.
Test your documentation, but tests aren't documentation.
In other words, it's great practice to ensure that code snippets inside your documentation execute correctly. Your primary benefit is that your documentation is correct, complete and up to date. Any testing benefit is purely secondary; you need a complete test suite regardless. You can use a doctest framework to create this test suite for convenience, but recognize that your doctests are going to be either illustrative or complete, but cannot be both.
Examples:
Here's a tutorial written in git comments so that each step is fully executable: http://cookbook.hobocentral.net/tutorials/agility
Here's a manual page written in rubydoctest: http://cookbook.hobocentral.net/manual/scopes
## Capitalizing text
The `capitalize` function receives a string and returns
it in capitalized format. Example:
```javascript
print(capitalize('some words'));
// => Some Words
```
If you could just do:
```javascript;commenttest print(capitalize('some words')); // => Some Words ```
and if javascript;commenttest got translated into <pre class="commenttest">, then you'd be good to go right now, without any special support from doctest. I believe if the element is syntax highlighted that doctest will just ignore (and wipe) that highlighting, but it depends on the details, I've never personally tried it. If you could add multiple classes you could use javascript;commenttest.hidden for setup code that you want to run as part of the tests, but doesn't add to the narrative. (There's a class that makes setup blocks compact, but I suppose I could extend it to do hidden blocks too: http://doctestjs.org/reference.html#compact)
I don't think there's an advantage for doctest to parse the Markdown directly, as opposed to parsing what is rendered. And extracting just the code also seems unnecessary, as it is helpful to see the context of any failures.
The reason we don't is because good tests are very rarely the same thing as good examples. Good examples often elide bits of code that aren't pertinent to the API function being demonstrated ... and good tests often involve the edge cases that are poorly suited to learning examples.
While testing your examples is all well and good, it's also limiting -- you can no longer write:
makeRequest({
url: 'http://example.com/endpoint',
data: ...,
})
My main concern with it is that it forces you to write the document in the same order you want the code to be extracted, which may not be the best order for explaining things. That is why classic Literate Programming tools like noweb[1] allow you to name the chunks of code and then arrange them into the generated files as you wish. You can see an example of this in action when I'm assembling noweb.php[2].
Also, agreed on the point that not every piece of example code is suitable for running as a test, so you'd optimally want some sort of annotation for that. Github-flavored Markdown[3] has the code block fencing syntax that would allow metadata like this. Something like:
```coffeescript
# Just a regular, non-testable example
```
```coffeescript;test
# Example that is also a test
```
1: http://en.wikipedia.org/wiki/Noweb
2: https://github.com/bergie/noweb.php/blob/master/README.txt#L...
> My main concern with it is that it forces you to write the document
> in the same order you want the code to be extracted, which may not
> be the best order for explaining things.
I think that, unfortunately, the tangle/weave paradigm is a large part of why literate programming is barely used -- it introduces an entire level of build-time complexity that doesn't have any direct effect or benefit on your code. Here's hoping that the simplicity of having a file that is at once both executable code and publishable markdown is something that feels more usable to more programmers.
It's also good practice to test your documentation. Far too many examples in documentation have subtle errors that could easily be caught by regularly executing them.
$ makeRequest({
> url: 'http://example.com/endpoint',
> data: 'foo',
> success: Spy('makeRequest', {wait: true}),
> });
makeRequest(...)
Still, supporting CoffeeScript shouldn't be a big task, it would just require abstracting out some of the parsing, and having a way to select the language (probably just another CSS class on elements, and/or the page body). The only parsing doctest really cars about is chunking the text by comment.
In regards to abbreviated examples: you can write your test in one big chunk, and I like to do this when I'm really testing and not explaining – doctest.js is not intended to just be a documentation tool. But if you are composing docs, then doctest will only test things that are explicitly marked to be tested. It's not as strict as Python's doctest, because it's using essentially invisible information to determine what's a test (a CSS class). You can also have it test things that aren't visible in the docs themselves, but are necessary to create a usable environment for the tests.
Of course a lot of things, like makeRequest, are only interesting if you handle the async aspect of the code. Spy() is handy for testing, but will look peculiar in documentation. It occurs to me that for documentation you might want:
makeRequest({
url: 'http://example.com/endpoint',
data: 'foo',
success: function () {
print('I got a response:', this.status);
},
});
// (later) => I got a response: 200