Cost to Solve < Remaining LTV * Profit Margin
You might think that such a user doesn’t exist. Then you’d be getting the details wrong again! Example: Should ISPs disconnect users for piracy? Should Apple close your iCloud sub for pirating Apple TV? Should Amazon lose accounts for rejecting returns? Etc etc.
A business that makes CS more details oriented is 200% the wrong solution.
There are a whole class of problems that do not require low-latency. But not having consistency makes them pretty useless.
Frameworks don’t solve that. You’ll probably need some sort of ground-truth injection at every sub-agent level. Ie: you just need data.
Totally agree with you. Unreliability is the thing that needs solving first.
Sounds like management to me.
How does gpt o1 solve this?
I've been using the general agent to build specialised sub-agents. Here's an example search agent beating perplexity: https://x.com/xundecidability/status/1835059091506450493
I'm failing to see the point in the example, unless the agents can do things on multiple threads. For example let's say we have Boss Agent.
I can ask Boss agent to organize a trip for five people to the Netherlands.
Boss agent can ask some basic questions, about where my Friends are traveling from, and what our budget is .
Then travel agent can go and look up how we each can get there, hotel agent can search for hotel prices, weather agent can make sure it's nice out, sightseeing agent can suggest things for us to do. And I guess correspondence agent can send out emails to my actual friends.
If this is multi-threaded, you could get a ton of work done much faster. But if it's all running on a single thread anyway, then couldn't boss agent just switch functionality after completing each job ?
The prompt was: <prompt> Research claude pricing with caching and then review a conversation history to calculate the cost. First, search online for pricing for anthropic api with and without caching enabled for all of the models: claude-3-haiku, claude-3-opus and claude-3.5-sonnet (sonnet 3.5). Create a json file with ALL the pricing data.
from the llm history db, fetch the response.response_json.usage for each result under conversation_id=01j7jzcbxzrspg7qz9h8xbq1ww llm_db=$(llm logs path) schema=$(sqlite3 $llm_db '.schema') example usage: { "input_tokens": 1086, "output_tokens": 1154, "cache_creation_input_tokens": 2364, "cache_read_input_tokens": 0 }
Calculate the actual costs of each prompt by using the usage object for each response based the actual token usage cached or not. Also calculate/simulate what it would have cost if the tokens where not cached. Create interactive graphs of different kinds to show the real cost of conversation, the cache usage, and a comparison to what it would have costed without caching.
Write to intermediary files along the way.
Ask me if anything is unclear. </prompt>
I just gave it your task and I'll share the results tomorrow (I'm off to bed).
Given that there is (a fairly standard) API to interact with LLMs, the next question is, what abstractions and primitives help easily build applications on top of these, while giving enough flexibility for complex use cases.
The features in Langroid have evolved in response to the requirements of various use-cases that arose while building applications for clients, or companies that have requested them.
o1 (and likely sonnet 3.5) made chain of through and other complex prompt engineering irrelevant.
Realtime API (and others that will soon follow) will made the best VTT > LLM > TTV irrelevant.
VLMs will likely make LLMs irrelevant. Who knows what Google has planned for Gemini 2.
The point is building these complex agents has been proven a waste of time over and over again until, at least until we see a plateau in models. It's much easier to swap in a single API call and modify one or two prompts than to rework a convoluted agentic approach. Especially when it's very clear that the same prompts can't be reused reliably between different models.
I suppose my comment is reserved more for the documentation than the actual models in the wild?
I do worry that LLM service providers won't do any better than rest API providers in versioning their backend. Even if we specify the model in the call to the API, it feels like it will silently be upgraded behind the scenes. There are so many parameters that could be adjusted to "improve" the experience for users even if the weights don't change.
I prefer to use open weight models when possible. But so many agentic frameworks, like this one (to be fair, I would not expect OpenAI to offer a framework that work local first), treat the local LLM experience as second class, at best.
Inference speed is being rapidly optimized, especially for edge devices.
> too expensive,
The half-life of OpenAI's API pricing is a couple of months. While the bleeding edge model is always costly, the cost of API's are becoming rapidly available to the public.
> and too unreliable
Out of the 3 points raised, this is probably the most up in the air. Personally I chalk this up to sideeffects of OpenAI's rapid growth over the last few years. I think this gets solved, especially once price and latency have been figured out.
IMO, the biggest unknown here isn't a technical one, but rather a business one- I don't think it's certain that products built on multi-agent architectures will be addressing a need for end users. Most of the talk I see in this space are by people excited by building with LLM's, not by people who are asking to pay for these products.
I don’t think the tech is ready yet for other reasons, but absence of anyone publishing is not good evidence against.