Hacker News Reader

Comment by bobbiechen | Hacker News Reader

Related: "If an AI agent can't figure out how your API works, neither can your users" (from my employer's blog)

https://stytch.com/blog/if-an-ai-agent-cant-figure-out-how-y...

Yeah, I've started to think AI smoke tests for cognitive complexity should be a fundamental part of API/schema design now. Even if you think the LLMs are dumb, Stupidity as a Service is genuinely useful.

Is this you have implemented in practice? Sounds like a great idea, but I have no idea how you would make it work it a structured way (or am I missing the point…?)

Can be easy depending on your setup - you can basically just write high level functional tests matching use cases of your API, but as prompts to a system with some sort of tool access, ideally MCP. You want to see those tests pass, but you want them to pass with the simplest possible prompt (a sort of regularization penalty, if you like). You can mutate the prompts using an LLM if you like to try different/shorter phrasings. The Pareto front of passing tests and prompt size/complexity is (arguably) how good a job you're doing structuring/documenting your API.

Lovely idea - thanks

It's a good tool to use for code reviewing, especially if you don't have peers with Strong Opinions on it.

Which is another issue, indifference. It's hard to find people that actually care about things like API design, let alone multiple that check each other's work. In my experience, a lot of the time people just get lazy and short-circuit the reviews to "oh he knows what he's doing, I'm sure he thought long and hard about this".

Reply on news.ycombinator.com