Track changes to CLI tools by recording their help output | Hacker News Reader

Track changes to CLI tools by recording their help output

simonwillison.net

10 pointsappwiz6 comments

My AWS tracker has caught a ton of a activity - they really do ship new feature in that tool every day of the week: https://github.com/simonw/help-scraper/commits/main/aws

My scraper against the GitHub GraphQL schema catches some interesting details too: https://github.com/simonw/help-scraper/commits/main/github

For AWS, you don't need to hit the cli in this circuitous manner. The clis are all generated from their API models, which are published publicly in json format.

Right - but actually comprehending the differences in different versions of that JSON isn't at all easy.

The diff of a CLI help output is a lot more readable than the diff of a JSON file.

Interesting idea, it’s like running an integration test for someone else’s tool.

I did something a bit like this for a few internal tools I built at Airbnb. I built a docs system that indexed all the markdown files in every repo or project’s .infra/docs folder into a central documentation browser website.

Then in each tool project, we had a CI job that would loop over all the commands and spit the —-help output in Markdown format into that folder. That way when someone added commands or changed a command’s args or help text, the searchable docs were automatically updated.

We didn’t use this to generate a change log to show to users per se, but it did mean that you could run `git log .infra/docs` to see a log of every change to the tool’s interface.

that's cool, but shouldn't the best way to solve this would be to ship the changelog with the CLI itself? Something like `cli --changelog`? I mean I don't understand why its not a general practice with CLI authors

When I built developer tools at Airbnb, we had the tool show the new things in the changelog the first time you ran it after updating, and we had a subcommand to spit out the whole changelog as you suggest. But I think the general issue is that writing good, useful changelogs is manual work — you shouldn’t show end user something like `git log —-oneline`, which usually ends up pretty noisy and useless no matter what kind of grooming you apply to it. We only put things in the changelog if they affected the workflow of ~all our users, and spamming markdown in the terminal would materially reduce our support burden.

For something like AWS, I guess you could do a changelog per service command? A changelog for the whole thing would be too detailed and contain too much stuff I don’t care about, I’d think.

Reply on news.ycombinator.com