Hacker News Reader

Comment by itishappy | Hacker News Reader

My company ran a test and found that they reduce token output on flagship model by something like 2-4x, and that number has been increasing with newer models. I suspect the increased subagent usage is driving this trend, because this means we're relying on models to do their own prompt engineering.

Yes, they are just text, and can therefore be replaced with good prompting. However, this also means they confer a real benefit: a good set of skills creates a transferable baseline, raising the skill floor and offering a more consistent experience across the organization.

View on HN

I am somewhat confused by takes like this. Of course skills are just prompts, this is the whole point.

A skill is just a stored prompt you want to put more information into than you're likely to type out every time you intend to do that thing. Documentation of a business process.

What are you confused by? You're saying the same thing they said.

They added the additional claim that writing the skills down (apparently) prevents the models from having to self-prompt on the fly and therefore reduces token consumption.

I think skills also count towards the token counts. In the end, everything becomes a 1-D array of input text.

Yes, skills that are actually used count toward token consumption.

The question is whether the number of tokens required to achieve a certain behavior/intelligence/quality is equal between you manually providing those tokens via skills versus the model "deriving" the "skills" it needs on-the-fly in order to produce the outcome you want.

The claim above is that the former requires far fewer tokens.

Also skills only consume tokens when they are used, and part of the value is that the model will dynamically find and disclose only what's needed (assuming the skill is "well-designed").

Their claim is not about the prompt or skill tokens, it's about output tokens - skills can help the model bypass some thinking tokens or avoid reasoning deadends, and that way reduce output token usage. That's what they seem to have found empirically from their testing. (If it's truly 2x-4x, the time savings in waiting for the output is a pretty nice benefit too.)

It's not just a stored prompt, you can attach re-useable scripts to them to offer more determinism. ex: a script that validates that a PR follows exactly the template you want, with a max of N lines per entry.

The more determinism you have, the more consistent you can be and the more leverage you can build. (yes I understand that skill calls are non deterministic).

That's just a stored prompt that references a script :)

Yes, that's my point :)

We do that, but keep the scripts in the code and just tell them in the markdown where the scripts are, same with "references" (docs/) for us. It never made sense to me to put those in a skill dir, many are useful across skills and for humans (many written for humans before agents were a thing)

One of the more interesting benefits to skills is that many harnesses now run the inline command(s) in backticks, shortcutting the model needing to make a tool call. This is helpful for deterministically building up context content for the skill before the agent ever sees it.

We take this further in some instances and have workflows that (1) does deterministic context gathering (2) invokes an agent (3) processes a file the agent is told to produce. This has made our PR review agent much better and removed it's access to all credential files. We have a step that gathers the diff + existing pull request comments into a .review dir, let the agent process that and create a comments.jsonl, then run a script in a new step to apply the comments against the API

+1 on all of this, it is similar to the pattern I see emerging as well. For example I have open-sourced (let's be honest, it's just my personal tooling of taste) a repo on gh to try and formalize these things. I'd rather not share out of mostly imposter syndrome but essentially it's a scaffolding tool ala "create-react-app" for the stack I generally use.

Part of that is to create standardized "docs/{architecture, decisions, contracts, operations}" folders that my harness recognizes in my repos to have a better sense of where was it.

Glad others are converging towards similar ideas!

I was solving your problem.before LLMs with my personal developers Swiss army knife (hof gen/CUE+templates). I'm somewhat keen to see how I can capture the agent markdown I want in many places, like project init, in CUE via embed and modules I can import and version, skipping the need for a markdown dependency cycle management system.

https://github.com/hofstadter-io/hof

(sharing because it has sloppy parts and that/I.S. shouldn't stop you from sharing!)

Reply on news.ycombinator.com