Show HN: Zero-Config Documentation Websites for Python | Hacker News Reader

Show HN: Zero-Config Documentation Websites for Python

timothycrosley.github.io

233 pointstimothycrosley81 comments

Tried this on our project, GitHub.com/hail-is/hail.

Sphinx is a persistent thorn in our side, but we've made it work. A couple things I had to do to get this working:

- something thinks a filename with dots (e.g. example.8bits.bgen) means there is a python module that needs to be imported (e.g. example), and fails when that does not exist; we have files like this in a data directory

- I could not quickly figure out how to ignore files, so I had to delete a Sphinx conf.py file.

- Sphinx resolves the _templates folder relative to where it is called, not relative to the doc string's source file, so I had to add several symlinks (rather than manually edit every doc string).

---

OK docs are generating. Quite slow, maybe a couple minutes. Looks like one core is at 100% during this process. We've got ~400 files. Maybe its parsing some data files? The whole directory is 318 MB.

---

Now I'm looking at the docs.

First thing I notice is that Python type hint forward references break everything. Consider:

    class GroupedMatrixTable(
        parent: 'MatrixTable',
        ...

The generated doc page is missing the class name and the entire class definition is inlined as a preformatted block (with highlighting :shrug:).

All my python function def's seem to be missing the function name?

No arg functions look really weird

    def (
    )

So all our docs are in ReST (thanks Sphinx :|). This means they're not valid Markdown. It seems that invalid markdown in a class doc string can break the formatting of all the methods (presenting them again as one reformatted block).

---

On the bright side, the mobile version looks great and the search works better than my experience with Sphinx.

EDIT: formatting

Thanks so much for this great feedback! I haven't done any speed optimization on portray yet - as both at work and online I tend to organize my own projects as small self contained repositories. Clearly there is a lot of room for improvement there, I have a lot of ideas to make it faster.

You can manually define the modules for portray (which you probably figured out to get documentation rendering): https://timothycrosley.github.io/portray/docs/quick_start/4....

And, yes write now Markdown only. I will say - this was a week only project: https://timothycrosley.com/project-2-portray so I think there's a good chance I could improve these points with one more week of time spent :)

Thanks!

~Timothy

Yeah of course! In retrospect my initial post seems a bit neutral to negative. I'm really glad this project exists, I'm endlessly frustrated by Sphinx. I'll keep my eye on your project!

A little off-topic, but as someone who's just started producing documentation in Sphinx, what are the problems/downsides that you've run in to?

I've used it for two projects so far: documenting a relatively small python module, and for documenting a RESTful API. I do a lot of embedded work, so I'd like to use it as a place to document a piece of hardware in it's entirety, from schematics/layouts to firmware and build toolchains to actual use of the device. Do you (or anyone else reading this) have any comments on that use case?

I think my main complaint at the moment is that it doesn't play nice with Markdown, so I had to re-format a bunch of pre-existing documentation for it to work in Sphinx. ReST seems to render alright in Gitlab though, so that's a plus.

Sphinx should support Markdown, we're using the instructions at https://www.sphinx-doc.org/en/master/usage/markdown.html for our project and haven't had any issues.

Haven't tried it yet but you should also be able to enable rst blocks through recommonmark so you get the best of both - https://recommonmark.readthedocs.io/en/latest/auto_structify...

If you are on 3.7 you can add this import:

    from __future__ import annotations

And then forward references don't need to be strings. I wonder if this could fix the generated page.

My team at present use sphinx-doc[1] and generate both HTML and PDF. We write the documentation as part of the code using docstring and then other documents manually in rst. We use Sphinx plugin to automatically generate document from code.

I will give portray a try and see.

Is there any comparison between Sphinx-doc and portray?

[1] http://www.sphinx-doc.org/en/master/

We've historically done this as well. One issue we perpetually run into is that the docstrings get out of sync with the actual signature / types of the methods/etc that they purport to document. I suspect there are plugins that generate documentation from types such that there is less to keep synchronized. It is also unfortunate that Python projects have to set up infrastructure to build and publish documentation packages. I really like the effortlessness that Go's ecosystem brings--https://godoc.org just reads your source code from your git repository, so there is are no explicit steps for building and publishing documentation.

With Sphinx's Napoleon extension (https://www.sphinx-doc.org/en/master/usage/extensions/napole...) Python 3 type annotations will be added to the generated docstrings so you don't have to write the annotations both in the function definition and in its docstring. I've found this to be very helpful in keeping docs in sync with the code.

> the docstrings get out of sync with the actual signature / types of the methods/etc that they purport to document

But that's on the devs, not Sphinx, eh?

It's kind of on the whole dynamically typed model. If you can have static type annotations that are validated for correctness, then you can use those annotations to validate or generate your documentation. Python supports annotations and validation thereof via Mypy, and assuming the project takes advantage of these, then Sphinx could leverage them to validate/generate documentation.

Which plugin do you use to go docstring->docs? I found it difficult out of the box to determine w/ Sphinx how to get everything to play nice.

I use sphinx-apidoc[1] command to extract docstrings from code and create Sphinx documentation files out of them.

Enter this command to look for modules in a package, say, foopkg, and create .rst files to generate documentation for each module in the package and subpackages:

  sphinx-apidoc --module-first -o docs/api foopkg

I am assuming that the Sphinx documentation project resides in a directory named docs. Each .rst file would contain automodule directives for Sphinx to automatically pull docstrings from foopkg and render them in the generated documentation. However, for Sphinx to understand these automodule directives, the autodoc[2] extension must be enabled in docs/conf.py.

  extensions = ['sphinx.ext.autodoc']

If the documentation is written using Google style or NumPy style docstrings, then the napolean[3] extension must also be enabled.

  extensions = ['sphinx.ext.autodoc', 'sphinx.ext.napoleon']

The sphinx-apidoc command above would also write a file named docs/api/modules.rst which would be used to render the starting page of the automatically generated API documentation. So include this docs/api/modules.rst with the Sphinx toctree directive from the some other page that the user is likely to visit. For example, the start page of the project documentation would likely be rendered from `docs/index.rst`, so add this to `docs/index.rst`:

  API
  ---

  .. toctree::

     api/modules

Now render the documentation with Sphinx normally:

  cd docs && make html

[1]: https://www.sphinx-doc.org/en/master/man/sphinx-apidoc.html

[2]: https://www.sphinx-doc.org/en/master/man/sphinx-apidoc.html

[3]: https://www.sphinx-doc.org/en/master/usage/extensions/napole...

The biggest differences between Sphinx-doc and portray:

- Markdown - Zero Config required - Focus on Simplicity

https://timothycrosley.com/project-2-portray goes into why I created the project and how I viewed the current state of Python documentation tool.

This is great! You hit the need for configless docsites that many Pythonistas who are in data science and research have: we write a lot of experimental and prototype code. Docstring documentation in itself feels like a considerable investment in these contexts so anything that helps publication without any hassle is more than welcome.

Are you planning to support Restructured Text next to Markdown? I find ResT is used more often (in the machine learning ecosystem at least).

Thanks!

> Are you planning to support Restructured Text next to Markdown?

I will look into supporting this, I agree that it definitely makes sense as a feature!

I just used this on a random project I happen to be building documentation for. I'm pretty in love with the minimalism of 'portray'. I wish there was a chance to get my team to use it, but I'll definitely be using it in my personal projects going forward!

It's always kind of interesting to look at a document created by the tool it's describing.

I thought the GIF within a GIF was very clever. Did you notice? It's "playing" itself back.

I think the number of playbacks is the number of times they had to re-record the video. :)

Ha, I wish it was accidental! In reality, I created the gif many more times than needed just because I wanted to create the effect.

Very neat, I especially like it's compability with mkdocs-themes. Will try this out on some project :)

I am confused, all I get is "No Python project found in the given directory".

portray requires your project to be packaged for it's auto discovery - if you get this error and your project has a setup.py or pyproject.yaml in the root (and this is where you ran portray from) let me know and I'll create a ticket! Otherwise - I highly encourage looking at using poetry to setup your pyproject.yaml file: https://poetry.eustace.io/

my project doesn't have a setup.py because it is not intended for packaging and it doesn't have a pyproject.yaml because to be honest, this is the first time I heard about such a file. But then again, the title says "zero-config".

portray no longer requires a config to be present, instead you can just specify modules directly from the command line with `-m`: https://timothycrosley.github.io/portray/TROUBLESHOOTING/#no...

I have this problem when using the src folder tree conventions (poetry new --src new_package).

Excited to try this. From a Perl background, used to POD, Python's doc systems have been a sewer.

How is the search implemented?

portray uses MkDocs default search plugin which uses lunr.js behind the scenes.

How easy is it to theme for when Material starts looking dated?

https://timothycrosley.github.io/portray/docs/quick_start/4....

portray is compatible with all MkDocs themes and plugins out-of-the-box

Awesome, looking forward to trying this out on a new project.

Kudos to using toml for configuration.

I was super pumped by this, and took some time to look into pdoc3 too.

- https://github.com/pdoc3/pdoc/issues/87

- https://github.com/pdoc3/pdoc/issues/64

That's a hard pass from me. I'm not going to link my employer's name (much less my own name) to anything with a swastika on it.

I saw that as well, late into the project creation. The pdoc3 maintainer claims it's not that - but there are other reasons to want to want to avoid pdoc3. I'm going to be working on replace the pdoc3 dependency with pdoc in a way that is transparent to users.

Great to see you are receptive to this change. I didn't notice it earlier but then when I see on the right hand corner of the website I was astonished it's there.

The change away from pdoc3 and to a new fork of the original pdoc (called pdocs) has landed:

https://github.com/timothycrosley/portray/blob/master/CHANGE...

https://timothycrosley.github.io/pdocs/

Thanks!

~Timothy

Fantastic! I look forward to this :)

The change away from pdoc3 and to a new fork of the original pdoc (called pdocs) has landed:

https://github.com/timothycrosley/portray/blob/master/CHANGE...

https://timothycrosley.github.io/pdocs/

Thanks!

~Timothy

There is no question that these are Buddhist swastikas; they are clockwise and also rotated 45 degrees compared to the most common nazi version.

It is difficult to appreciate how ubiquitous swastikas are in the Buddhist world until you go there and see them everywhere. I have no doubt that this was an innocent decision.

It stopped being an innocent decision once it was pointed out and he doubled down.

Put another way: I cannot put my or my employer's interests in Western markets at risk because something is acceptable in Eastern markets. The maintainer can do whatever he wants to do; I'm not questioning that. I'm just saying I won't be using it, even if it would save me from sphinx-autodoc fragility.

As the original author of pdoc, thanks for not using pdoc3, which---even separate from the Nazi imagery---is something that I consider to be an unethical fork. You might instead consider using the original pdoc, which is now maintained by mitmproxy[1]. Although, it does seem like activity on the project has waned.

[1] - https://github.com/mitmproxy/pdoc

> I cannot put my or my employer's interests in Western markets at risk because something is acceptable in Eastern markets

My employer works in both markets, so rejecting something because it's Eastern would look much worse than rejecting it because someone in one market might not know what it is.

Anyway, it's a tiny icon on the boiler plate of a dependency of a project used to generate documentation. It would be a huge stretch to say your employer supports Nazis because of this.

I highly doubt the absence of any religious iconography is going to harm my employer in eastern markets.

> There is no question that these are Buddhist swastikas; they are clockwise

They are paired mirror images, so they are both clockwise and counterclockwise.

His github profile picture is the swastika counter-clockwise. It's very clear from that his intention is to include things that appear like the Nazi swastika but are technically not. That's bad intention in my book.

In all fairness to the GitHub account, the Python logo itself could be perceived as the same logo made out of two snakes.

But troll culture masquerading as innocent plausible-deniability really is ruining the internet because it's hard not to assume the worst (we see the same symbols being used on purpose to illicit a reaction and that person did respond with "Made you look. :grin:").

It looks like it's not on the website anymore?

It's true that that swastika (usually going in the other direction) is a buddhist symbol, but it looks like the person who forked it is from Slovenia, which has never been a buddhist country but was briefly a Nazi country. It's hard to believe that was an accidental oversight.

Still here: https://pdoc3.github.io/pdoc/ (bottom only).

Seems like he means well, but if he means that well, why not remove them?

I think being told what you are not allowed to do by strangers on the internet can elicit a certain stubbornness. I don't blame either side, but think it is strange for people to be so sure of another's intentions.

For what it's worth, I'm not claiming I'm certain he's a nazi sympathizer. What I am claiming is I cannot risk media or competitors making a big deal out of it, even if he means well. Surely that isn't a far out proposition, right?

oh man that's sneaky. They're very faint. The sneak factor does not belie good intentions, regardless of claims.

These barely appeared on my laptop's screen. I went back and checked twice because of comments before actually seeing it.

It's only sneaky if you assume bad intentions. Otherwise they're discreet.

Symbols get irrevocably besmirched all the time. I'm not going to be the person roping in a documentation tool that results in "<FAANG Company X> uses software linked to Nazis" as a BBC headline, you know? That headline alone is going to roflstomp any amount of nuanced rhetoric - and that's giving the maintainer the absolute most kind benefit of the doubt, which I'm not even remotely convinced they actually deserve.

The site has been updated (see issue 64)

But the site hasn't been updated and that issue dismisses them as not a problem?

I didn't find the symbol on the site linked anymore, and it's different from the screenshot. I may be missing something though?

It's still in the footer. From the screenshot I thought it was directly around the quote but it's actually closer to the edges of the screen when in desktop view.

My bad. I was looking in the wrong spot, they are indeed all the way out to the side and pretty light. Probably in my blind spot on this particular screen size!

I thought it's common knowledge the Nazi symbol is rotated 45° and vents clockwise. Still, agree that removing it is the smart move to increase adoption.

Many comments are addressing the fact that there is a difference between the clockwise and counter-clockwise symbols and that they should not be confused, but they are both on the website?

It's common knowledge amongst Nazi apologists, internet trolls, and actual Nazis.

The way your comment is phrased is disturbing.

Its not a problem of adoption. It's a problem of normalizing hate. If you don't understand that, I don't want to be associated with you.

Ok. Put me in the internet trolls one, that seems the least offensive (I know this from 40 years of watching Holocaust Memorial Day movies)

I feel bad for the author. To be labelled a nazi for using the counter-clockwise svatiska is undue.

Edit: Can I have any argument instead of simply downvoting me?

My reasoning to critizise him goes like this:

It is quite apparent that he knows about the meanings and uses of the symbol, therefore he could (and should) have anticipated the discussion. I grew up in an environment where nazi insignia are forbidden, and a very common way for convinced nazis to get around this was to just flip the damn thing and claim it is ok now, while in spirit you still go and use it to hail hitler with your friends.

I grew up in southern Austria next to the Slovenian border (which is where this guy is based). This part of Austria had a 40% right wing majority in the past and after a few beers right wingers still openly hail hitler in a pub, despite that beeing illegal. My grandfather who passed away a few years ago was a convinced nazi — this thing is still alive here.

The author is based in slovenia, literally a hour by car from where I lived. There is no way somebody from that area uses that symbol accidentally, without having considered its nazi connotations. This is an area where every other person had somebody in his family killed by nazis or were profiting from it in some way.

So either this person is incredibly naive and tries to overwrite history that is still very much ongoing or it is somebody who in full knowledge of all of this still decided to go with the symbol. In any way it doesn’t look good to me.

He rotates it the other way in his github profile picture.

You are right about it, didn't pay attention to his github picture.

i don’t think people were labelling him a nazi; i think they had issue with their own projects, or their name being accidentally associated with nazis

Do you have the same bad feelings for neo-Nazis who try to rules-lawyer the anti-swastika laws of Germany by displaying the mirror image?

We're talking about an EU citizen, who is professing to be ignorant of what he learned in school, after he is informed about the issue.

You know the Python symbol is a swastika, right? Technically it's a couple snakes that are coiled like a swastika, but if it's "anything to do with a swastika" we should avoid it, right?

https://www.flaticon.com/free-icon/python-file-symbol_28884 https://www.python.org/static/img/python-logo.png

I think the average person is going to have to have you point out the relation and squint for a while before they see it, if at all. I had to close one eye and really look at it before I saw what you were talking about.

Syntactic whitespace is pretty high on my personal fascism chart.

Reply on news.ycombinator.com