Hacker News Reader

Comment by hubraumhugo | Hacker News Reader

> All models can handle correcting choosing a tool from 250+ tools provided the user query contains all necessary parameters for the intended tool with >90% accuracy.

This is pretty exciting news for everybody working with agentic systems. OpenAI has way lower recall.

I'm now migrating from GPT function calls to Claude tools and will report back on the evaluation results.

View on HN

Claude's [new] tool usage is pretty good. Unlike with GPT-4 where I had to really minimize the context and descriptions for each tool, Claude Opus does better when provided more details and context for each tool, much more nuanced.

I'm now using it with 9 different tools for https://olly.bot and it hits the nail on the head about 8/10 times. Anthropic says it can handle 250+ tools with 90% accuracy [1], but anecdotally from my production usage in the last 24 hours that seems a little too optimistic.

Annnd, it also comes with a few idiosyncracies like sometimes spitting out <thinking> or <answer> blocks, and has more constraints on the messages field, so don't expect a drop-in replacement for OpenAI.

[1] https://docs.anthropic.com/claude/docs/tool-use

Olly is really neat, I just set up a chat with it. How did you architect the web search (tools?) if you don't mind sharing?

You should do the new HF TGI server, it has both grammar & tool support now. Works fabulously with Mistral Instruct & Mixtral Instruct.

Whats grammar support?

It's basically guidelines to adhere in the output. Ex. Your prompt asks for a summary, the LLM won't necessarily just spit the summary, there might be phrases like here's the summary - then the actual summary or even basic JSON key/values can get messed up. So grammar allows for defining the expected output in a variety of ways, some via Pydantic class definitions, Microsoft's Guidance, Outlines along with llama.cpp's grammar are attempts at making structured output reliable.

Most of langchain is basically a specific prompt with exception handling for missing fields.

Even better than guidelines!

Grammars perfectly restrict LLM token output. They’re hard grammar rules.

You can give it a formal definition for valid JSON, and it will only generate output that matches.

I thought this too where it will usually pick stuff listed first rather than a more suitable tool down in the list.

Sometimes it will out right state it can't do that then after saying "use the browse_website tool"

It will magically remember it has the tool.

Reply on news.ycombinator.com