it's langchain, what did you expect
252 karma · joined February 18, 2022
I'd be forever ashamed if I wrote that.
100% agree on building the optimal context, just have not seen parallel agents do better at sequential tasks. the documentation agent may have better initial context about how to write documentation, but it doesn't have the context of the changes, apart from what is passed to it / can explore. if we don't spawn a new session, and instead throw it a /document command - that would still get all the guidelines and rules for writing documentation, and it should have the same weight since it's at the bottom of the context.
for me the highest 'model jump level' performance booster is externalizing context and controlling the process - having claude initialize a plan file with a pre-defined template that makes it use that as its to do list and documentation place, and getting it to use that as its primary working area
Agents have their own context and can be useful for tasks that can be parallelized, which is a minority of tasks. How are they critical to better performance for you?
i hate how LangChain has always tried to make things that are simple seem very complicated, and all the unnecessary new terminology and concepts they've pushed, but whatever sells LangSmith.
Mitata has added a ton of things since then and it's very mature, and it's built by evan so it's definitely the right choice over benchmate. I used benchmate because I can easily extend it to support the upcoming features in BenchJS.
BenchJS is for easily building shareable benchmarks, and I want to evolve it into a nice code analysis tool. It uses Babel and esbuild to transform and bundle the code before it gets evaluated, and I want to use this to extract more useful insights like the number of operations of a given type (ex. array methods), code path analysis (coverage-like), inline indicators of how much % time a line took, etc.
It might be that the new Structured Outputs feature is subpar when compared with "structured output through function calling", I'll try to compare them.
`/api` contains the openai/anthropic clients builder, using llm-api (on npm) as a wrapper. There are no external tools, the LLM is just forced to call a shim "returnResult" function to make it fill each field according to the schema.
I didn't like it overall and I think it can confuse people and it shouldn't mention best practices:
1. To ensure a reasonable amount of variation, its temperature was set to 1.0.
Why would you use any other temperature than 0 when you are asserting the correctness of the data extraction and the "reasoning" of the LLM? You don't want variation.
2. true_answer = (5029) + (1.7509)
Why are you using the LLM to do math? If the data was extracted correctly (with structured output or function calling) let it write the formula and evaluate it.
3. The new Structured Outputs in the API feature from OpenAI is a significant advancement, but it's not a silver bullet. / there's more to this story than meets the eye
The new API is just a nicer built-in way to extract structured data. Previously (still valid) you had to use function calling and pass it a "returnResult" function that had its payload typed to your expected schema. This is one of the most powerful and effective tools we have to work with LLMs, if used for what it's supposed to do, we shouldn't avoid this because it doesn't "reason" as well.
> John Doe is a freelance software engineer. He charges a > base rate of $50 per hour for the first 29 hours of work > each week. For any additional hours, he charges 1.7 > times his base hourly rate. This week, John worked on a > project for 38 hours.
And the sample scenario is something I wouldn't use LLMs for. Nevertheless, CoT can still be applied with structured output, I'd like to see your structured-output-reasoning-cot to figure out why it didn't work.
Structured outputs allow you to extend the prompts by providing descriptions for the fields you expect, making it much more effective than other solutions, and you can also implement features like self-healing loops (fx. to get rid of the 1% chance of gpt-4o to not reply with data following the schema), etc. The paper's authors used the plain "JSON mode" that is useless, glad to see you did it better.
---
Anyway, here's GPT-4o with function calling (not even structured output) solving the issue correctly every time: https://gist.ro/gpt-reason.mp4
As you can see it's super consistent with GPT-4o and temp 0, with a silly simple prompt. If someone worked on the prompts / split it into a "multi-step pipeline" (come on, is this what we call fn(fn(x)) now?) they would achieve the same result with 4o-mini.