Hacker News Reader

Comment by MattGaiser | Hacker News Reader

We need a way to unit test documentation to see if the code snippets still compile and the things that are referenced still exist.

View on HN

This actual exists in some places! Check out moduledocs and "doctests" for languages like Elixir. Was something really cool I liked while I have been learning Elixir.

If you write a unittest in D underneath a function, it will automatically be included in the generated docs as an example. This is how all the examples in the standard library documentation are created, so you know for a fact that those work.

Sphinx supports this for Python: https://www.sphinx-doc.org/en/master/usage/extensions/doctes...

This is actually something from Python's stdlib: https://docs.python.org/3/library/doctest.html

Really underrated.

I seem to recall a talk given at one point that demonstrated a scheme that would reverse engineer functions out of whole cloth based only on docstrings and doctests, but of course I can't find it now.

My job is mostly Python. Did not know this existed.

Use contracts and propery-based testing. See Hypothesis library in Python (and other laguages). I wrote a library for contracts in Python (http://github.com/Parquery/icontract, see its readme for further references to other libraries).

This is why I prefer to link to actual code in the repo, in an examples directory, which gets compiled during every build.

This is not only simple and comprehensive but also doesn't require any special support from the tooling or language. Love it!

Unfortunately that doesn't help if the documentation is written by a different team and in a different format.

Getting out of sync happens very quickly if docs aren't generated from source and aren't an integral part of the release process.

That's definitely something that's still missing in technical documentation. The tooling still isn't there yet, and that's sad.

Every change in API or behaviour should be automatically blocked by CI tooling in release builds if its documentation is missing or outdated.

But alas, even in 2020 technical documentation is still just an afterthought for most companies - even big ones.

My company is currently trialing this with the Open API Spec. The workflow at a high level:

1) Make code changes to a specific microservice. (note: The Open API Spec also lives in code.)

2) CI/CD pipelines get triggered.

3) New microservice is built.

4) Call every REST endpoint defined in the microservice's Open API Spec and validate it using the example requests/responses.

5) If successful, regenerate the API documentation from the Open API Spec.

This ensures that our documentation and services will stay inline with each other.

It's far less extensive than what you presented, but on the current project we are working on at work, we are using https://github.com/swaggo/swag

It's somewhat specific to golang but so far it has been relatively good for us.

The spec is directly next to the code as code comments, so it has far more chances to get updated when changes are made compared to an external documentation.

Also, the payloads and responses are directly derived from the Golang structs, so, at least on that aspect, changes in the code are automatically reflected in the spec (apart for description and examples).

We also put the interactive documentation directly in our API under /doc. It is a useful tool for developers when implementing a new url/handler or modifying an existing one. It also creates incentives to keep the documentation matching.

Overall, we had very few mismatches between the spec and the actual implementation overall, despite not having deeply tested it (be it manually or automatically). Apart from one or two mismatches that were fixed quickly, I was able to take the spec, generate client libraries from it (swagger-codegen), and use them for quite extensive demos without issues.

We are still early in the project (not in production yet), and there are definitely some aspects we need to improve (integration/automated tests to be sure doc and code are 100% matching) or to completely figure out (ex: how to handle several versions of the same API). But overall, using swaggo/swag has been a pleasant experience.

Internally to $WORK I've got a documentation system consisting of interactive F# notebooks, put together automatically from marked-up F# code (so you get type-checking while you write them) with output assertions checked in the CI pipeline for the corresponding library (and before a pipeline pushes out the latest version of the documentation). Hoping to open-source the system if I ever get time; I think Mathematica's documentation system is absolutely first-class, and I want that in any language that supports notebook-style interfaces!

obligatory "Rust docs can do this"[0] (quite a nice feature tbh).

[0] https://doc.rust-lang.org/rustdoc/documentation-tests.html

Python can too, and I think most other languages are inspired by it.

https://docs.python.org/3/library/doctest.html

oh cool, didn't know that was where it came from, thanks for the pointer!

Obligatory reference to https://en.wikipedia.org/wiki/Web_(programming_system).

But yeah, Python made it usable :)

We do this for our project (https://hail.is) and it's a game changer. Saves so much time and so many bug reports.

Can you specify how are you doing it exactly? Very interested.

Comes standard with Rust

Reply on news.ycombinator.com