Hacker News Reader

Hacker News

Comment by skissane | Hacker News Reader

Parent Full thread

skissane

How I use skills:

- Create a skill to perform some common task

- Watch the agent use it

- Wait until the agent makes a mistake

- Ask it to update the skill to warn it against making that mistake again

I think that, using that iterative process, I end up with something better than just asking it to perform the task plainly

One observation: ask an LLM to write a skill, it tends to make them overly verbose and prescriptive. Often, something briefer and human-written actually works better

Another: I generally let the LLM propose edits to the skill, but I review them carefully and often modify them, because I find it has a tendency to solve the current problem at the price of worsening the solution to a previous one

View on HN

> - Ask it to update the skill to warn it against making that mistake again

Yep! I have a canned prompt that basically boils down to "scroll up, read through and give me the top `n` things that were difficult..." and more often than not a small skill or change to `agents.md` comes out of that.

> "scroll up, read through and give me the top `n` things that were difficult..."

Why do you think it can answer that? It has zero clue.

It can easily detect repeated failed attempts at a non obvious problem such as incorrect documentation or missing run flags

Technically speaking: it can't, any more than it can actually answer any other thing you ask it to do.

But in practice it does produce fairly reasonable output fairly often, and if you've been there watching and correcting it you can probably validate the result quite easily. "Re-read everything before" and similar are definitely anthropomorphizing, but that doesn't mean they're ineffective.

Why wouldn’t it?

> One observation: ask an LLM to write a skill, it tends to make them overly verbose and prescriptive. Often, something briefer and human-written actually works better

Incredibly so. All too often a “hey, you shouldn’t have done that here” turns into “NEVER X”, without it actually understanding or making an effort to understand why the correction was made, though I would say usually the context to do so should suffice. I have a rule for it to reason this through and use nuanced language and would say the frontier GPTs (5.4 to 5.6 Sol) get it right around half of the time (very rough estimate). The conclusions are really dumb sometimes.

I’d be interested in an expertly crafted skill here, maybe there are things about it you can still improve. This is an area in which I still find LLMs to be quite lacking.

It also made me appreciate the complexity of the nuances and levels of indirection of what I want to teach it: “Upload to the remote host, which in this case is this but might be another one in another use case, only do this on the local system and this on the remote, do this to X unless Y …, prefer doing Z unless another rule overrides it …”

I sometimes feel like there could be a more expressive way to structure to structure these rules (bring the snark!).

Yep. That is what I do too. But a lot of times the changes it makes are bandaids rather than addressing root cause. I have to periodically evaluate the skills for consistency and opportunities for simplification.

Super valuable and actionable advice. :100:

Reply on news.ycombinator.com