The Problem with Linux Kernel Documentation, and How We're Fixing It | Hacker News Reader

The Problem with Linux Kernel Documentation, and How We're Fixing It

blogs.s-osg.org

73 pointsbuovjaga33 comments

Related: Tomorrow [29-SEP-2016] afternoon at 15:00-15:40 CET, Jon Corbet of LWN, is giving a talk[1] at Linux Recipes conference in Paris on this subject.

URL[2] to live streaming.

[1] https://kernel-recipes.org/en/2016/talks/kernel-documentatio...

[2] https://air.mozilla.org/kernel-recipes-2016-09-29-PM-Session...

The main problem is not the format of the documentation, but lack thereof. Many important functions and high level implementation tasks are not documented or the docs are inadequate.

Perhaps there's little documentation _because_ it has a poor format.

Not because of Programmers who think that the time spent writing documentation would be better spent programming some more? ;)

No reason it couldn't be both, I suppose.

Definitely the case in my office. A couple of opinionated and (relatively) senior devs that think the code is the documentation. No, I can't read this 40 yo fortran code and use it as the basis for a new program on a different architecture. It uses gotos and variables like "yxtgn" that mean nothing to the reader (turned out y meant it was an integer, weird). I spent a month on that because there was no documentation, which took up a page once I was done writing it after my reverse engineering effort.

And then documentation, where it exists, is often poorly CM'd word documents, and poorly OCR'd PDFs (or doc->pdf that somehow lost the ability to be searchable and selectable text, what option did they select?!?).

The offending systems are only maintainable because a few people haven't retired (< 1 year left for several of them) and sheer force of will for the rest of us.

> It uses gotos and variables like "yxtgn" that mean nothing to the reader (turned out y meant it was an integer, weird)

It meant "ynteger". Elementary, my dear Watson.

I (being the kernel documentation maintainer) have already seen an increase in activity around the docs as a result of the change - it makes things far more accessible.

It's also not really correct to say there's not much documentation. We have quite a bit if you look at it, especially if you count the 55,000 kerneldoc comments in the source itself. The quality of some of it and the organization of all of it is another matter...but we're working on that.

It is almost certainly correct that this is a contributing factor. It is unlikely to be the only factor or even the primary factor.

The very human element, that most programmers simply don't like writing documentation is certainly a larger factor. If this holds true for the Kernel no new format can fix the problem by itself. At a minimum it will require a encouragement from the top to improve.

They missed a format that's ubiquitous, has a lot of formatting options, and has search facilities. Manpages. When someone's looking for a piece of documentation, they are usually working on a system as they do it, and man -k <thing> is a lot easier to remember than some other search tool that requires its own set of packages, etc.

Nothing prevents their new solution from generating man pages as a way to create end user accessible documents.

Manpages are not a format to write the documentation, but to display it.

You can `make mandocs` in the linux tree, then `make installmandocs` to make them easily accessible with the man command, i.e. `man struct_sk_buff` will display the definition of an sk_buff.

The problem with that approach is that you are limiting yourself to fundamentally primitive documentation (and by primitive I mean non-interactive). Granted, for a kernel it might be less important, but it's great when there is a good architecture diagram/image instead of 7 paragraphs.

Manpages can have a rather large amount of interactivity. Documents can be formatted as text, as HTML, with links, etc. The problem is that manpages are an afterthought in too much of the Linux ecosystem, so people have low expectations and low beliefs in what they can actually do. They can have tables, they can have equations, they can have graphics.

Out of genuine curiosity, could you link me to an existing guide from the Linux community that is based on manpages and features images and/or other interactive content?

Not exhaustive, but I've written code that processes 50,000 or so man pages and haven't hit one.

You can build man pages with Sphinx.

http://www.sphinx-doc.org/en/stable/builders.html

Have you actually tried to use this for something real? Sphinx source doesn't seem suited for man pages at all, and formatting seems the least important bit that doesn't fit.

Happy to see they pick ReST over MD. ReST is vastly superior for these purposes, in my experience.

Depends. We use Markdown pretty extensively and the drawbacks listed on the site don't really apply - you can always either:

1. Embed HTML (more or less preferred, depending on your situation). 2. Create an extension.

>Embed HTML (more or less preferred, depending on your situation)

But then you can't embed markdown inside the HTML, like code blocks inside a table. Sure you can use <pre><code> tags, but then you lose syntax highlighting.

Actually I've always wondered why no MD renderer supports something like

    <code data-lang="js">

which it then treats the same as a

    ```js

code block. Or a pseudo <x-markdown> element...

Kramdown supports this with markdown="1" as an attribute on html tags: http://kramdown.gettalong.org/syntax.html#html-blocks

Not only are you correct, you allowed me to find an equivalent feature in CommonMark! If there is an empty line between some content and its wrapping HTML tags then that content will be interpreted as markdown. So

    <td>
    
    ```js
    console.log("Hello, world!");
    ```
    
    </td>

will work as expected. http://spec.commonmark.org/0.26/#example-118 and further discussion at http://spec.commonmark.org/0.26/#example-153

VSCode's markdown preview renderer ( https://github.com/markdown-it/markdown-it ) implements CommonMark, so I'm very happy right now :)

> But then you can't embed markdown inside the HTML, like code blocks inside a table. Sure you can use <pre><code> tags, but then you lose syntax highlighting.

Pandoc supports this with the `markdown_in_html_blocks` option.

3. Pick a language that already supports it and is stable, standardized, without having to whip out some specialized version just for your codebase.

Not like you have to do anything to get it to work OOB, but your mileage may vary. One of the benefits of Markdown is that it's easily understood by people new to the doc space. RST is slightly more complex and has more rules added in.

Mind explaining why? I like Markdown but would like to know what ReST does better.

ReST can be better extended without creating a variant of the markup language [1]. Extension support is first class [2], and it is by that means that a good number of the core features are implemented.

[1] For example, github flavour, stackoverflow flavour, reddit flavour, ...

[2] Roles and directives.

One large difference is that Markdown exclusively targets HTML as an output format where ReST tries to be format agnostic. If you only ever target HTML this might ease some things but not if you want html, man pages, pdfs, plain text, …

ReST is a whole multi-document system, as opposed to MD which is just the formatting (at least before you mix in the dozens of mutually incompatible extensions). ReST is a superset.

I find it surprising that Michael Kerrisk's Linux man-pages project isn't mentioned here [1]. He and other contributors done a major job documenting the Linux man pages. It's usually the first place I go to when I need to read up on the Kernel APIs or C libraries.

- [1] https://www.kernel.org/doc/man-pages/

Are there any specification for ReST? With tests that confirm if a parser works?

Otherwise it will be harder to get more tools.

They should contact the freebsd-doc@ folks.

Reply on news.ycombinator.com