Show HN: LLM-docs, software documentation intended for consumption by LLMs | Hacker News Reader

Show HN: LLM-docs, software documentation intended for consumption by LLMs

github.com

28 pointseigenvalue7 comments

I was inspired by a recent tweet by Andrej Karpathy, as well as my own experience copying and pasting a bunch of html docs into Claude yesterday and bemoaning how long-winded and poorly formatted it was.

I’m trying to decide if I should make it into a full-fledged service and completely automate the process of generating the distilled documentation.

Problem is that it would cost a lot in API tokens and wouldn’t generate any revenue (plus it would have to be updated as documentation changes significantly). Maybe Anthropic wants to fund it as a public good? Let me know!

Open articleView on HN

Oh! Recently I had the experience of working with someone who was using LLMs to build something using my JS canvas library. The code the LLM was producing for this person was ... sub-optimal. Over-complicated. Not a surprise to me as my library is very niche and the only documentation around it is the docs/lessons I've written myself. So now I'm in the middle of an exercise to write documentation[1] that tries to explains everything (that I can remember) that the library does.

The problem is, I've no idea how useful that documentation would be for LLM consumption - does anyone know of an "Idiot's Guide to writing documentation for LLM consumption" so I can review my work to date and improve the docs going forward?

[1] - In this branch. I'm writing the documentation in .md files which get converted into .html files (using Sundown) during a build step: https://github.com/KaliedaRik/Scrawl-canvas/pull/119/files

It’s a cool idea. I’ve wasted a lot of time over the past few months futzing around with beautifulsoup, Playwright and others I forget, or cloning entire repos and trying to figure out exactly which incantations for which build tools are going to get me the built docs I need, all in service of setting them up for retrieval and use by LLMs. Some projects (e.g. Godot, Blender, Django) make it very easy. Others do not (Dagster is giving me headaches at the moment).

I would probably prefer to receive unmodified, plain text/md versions (with the heavy lifting done by, e.g., docling, unstructured) than LLM summaries though, since I’d rather produce my own distillations.

I would pay for that kind of thing. I think the intersection between ethical scraping and making things machine-readable is fertile ground. For a lot of companies it’s something that can be of great value, but is also non-trivial to do well and unlikely to be a core competency in-house.

I feel like humans don’t have great intuitions around the ideal format for an LLM. But an LLM probably does have a good sense for what would work well for itself.

And also I think that we shouldn’t necessarily expect to find LLM targeted documentation to be intuitive or helpful as human reader.

Ultimately you could empirically measure how well a given distilled documentation file works compared to another one with a given LLM. Just give it tasks to do that rely on the documentation in thorough ways that give good coverage (would be easy to have Claude generate such code from the docs).

Then see how well the LLM does on a battery of tasks given a particular distilled documentation in its context window along with a request for a particular coding task.

The distilled documentation file that gets the best average score with the shortest length is ideal and that should be the one you use for this.

Yeah, agree that one could / should definitely put it through evals if it were going to be a product. There’s quite a bit to a good evals program, though.

I think the main implementation challenge here - as in most of these things - would be how you retrieve / traverse the distilled documents. I think it’s hard to disentangle your distillation method from your preferred retrieval method. Surfacing stuff that’s relevant to a query is pretty trivial, surfacing that in a way that takes account of complex cross-document references, less so. All old-school search skills, though - suddenly (or soon to be) in big demand, no doubt.

Dagster's docs must be LLM-readable somehow because their LLM that lets you ask questions about their docs is the best RAG experience I've had yet.

That’s good to know - haven’t tried it, tbh, since they’re usually so poor, but will definitely check it out. Good stopgap solution.

Why does it need to cost a lot of tokens, and why can't it incrementally update?

How do the VS code integrated AI helpers do it?

Reply on news.ycombinator.com