I wonder if LLMs can generate docs at any reasonable level of usefulness and accuracy
Speaking of which, I just tried using ChatGPT to create a React app with typescript and rollup. What it generated didn't work (got errors). Of course I could tell it the errors and it fixed them but it suggests they aren't yet ready to write accurate docs
In fact, "superstition" very neatly defines most prompt engineering for LLMs.
Its weirdly fascinating how this is developing. Like some people boldly claiming that if you offer to tip your LLM it does better. And if you say things like "Think hard and check your work" it gives more accurate results.
LLMs can put into words what a programmer can infer from struct definitions, function signatures, names and even taking a cursory look at a function body.
LLMs cannot retroactively find out the _why_ of design choices. If you don't document the reasons behind decisions while making them, they'll be lost, which is exactly what leads to superstition. As a result, changing code is error prone because you never know when code is written in a particular way for a good (and still valid) reason or just because.
Works pretty well. Could still [theoretically] go out of date, but that hasn't happened, so far.
The one thing that does happen, though, is copy/pasta errors. I'll copy a bunch of code, including the headerdocs, and paste, but forget to adjust the docs.
You need some overalls documentation to explain the concepts and how to put things together. When the user understands that you can use in-code documentation but even then only if you need to use one function at a time - there needs an explanation of how to put things together.
I usually create full-fat test harnesses[0] for my stuff, as that gives a totally realistic implementation example.
But I think having a well-written README is more likely to be useful.
API/implementation documentation (whether docs or tests), are much more likely to be read, than end-user docs.
With Apple's docc, you can drop READMEs throughout the system, and they show up in Xcode's doc browser (and can be referenced from the code).
A lot of times documentation isn’t read by anyone for stretches of time till it’s needed. I imagine documentation by LLMs will make the situation worse if nobody verifies its corectness. I don’t believe in magic solution, good documentation that is not out of date requires work and if a company doesn’t allocate resources no magic will fill in the gaps.