Multi AI agent systems using OpenAI's assistants API
github.com
github.com
Here is how it works. When you upload attachments, in my case a very large PDF, it chunks that PDF into small parts and stores them in a vector database. It seems like the chunking part is not that great, as every time you make a call, the system loads a large chunk or many chunks and sends them to the model along with your prompt, which inflates your per request costs to 10 times more than the prompt + response tokens combined. So, be mindful of the hidden costs and monitor your usage.
Ideally if you want a model’s weights to include a credible representation of non-trivial data you want it somewhere in the training pipeline (usually earlier is better for important stuff but that’s a hubristic at best), but there’s transfer learning of various kinds, and joint losses of countless kinds (CLIP in SD-style diffusors come to mind), and fine tunes (if that doesn’t just count as transfer learning), and dimensionality reduction that is often remarkably effective, and multi-tower models like what evolved into DLRM, and I’m forgetting/omitting easily 100x the approaches I mentioned.
It’s possible I misunderstand you, so please elaborate if so?
This is how RAG works.
While you can come up with work-arounds like using lesser LLMs as a pre-filtering step the fact is that if you need GPT to read the doc you need GPT to read the doc.
Yes of course. The point remains: the LLM has to process the data somehow.
If you are concerned about costs and token usage then switch to a provider that works for your problem (Flash Gemini looks very interesting..)
Not sure what you mean by this.
And there are some serious people at Anthropic, they’ll get the typo thing if they haven’t already (been a busy week and change, they easily could have shipped a fix and I overlooked it).
I'd guestimate 99% of people using LLMs are using instruct-based message interfaces that have a variation of system/user/assistant. The top models mostly only come as a completion models, and even Anthropic has switched to a message based API
In some cases the lock in is what it is for now because a particular model in reality is so far ahead, or staying ahead.
It doesn't mean other options won't become available, but it does matter to relate your need to your actions.
Getting something working consistently for example might be the first goal, and then learning to implement it with multiple models might be secondary. The chances of that increase the later other models are explored in some cases.
It should be possible to tell pretty quickly if something works in a particular model that's the leader, how others compare to it and how to track the rate of change between them.
- Writing what I want in Python/other-lingo gives me much more customizability than these frameworks offer.
- No worries about the future plans of the repo and having to deal with abandonware.
- No vendor lock in. Currently most repos like this focus on OpenAI's models, but I prefer to work with local models of all kinds and any abstraction above llama.cpp or llama-cpp-python is a no-no for me.
The last point means I refuse to build on top of ollama's API as it's yet another wrapper around llama.cpp.
There is really no way to make the ensemble behave with an acceptable level of consistency.
Where we ended up is now having a frontier model generate a whole tree of possible execution plans, and then have the user select one of those path, and then we just run whatever the user chose in a plain sequence until the next decision point that needs user approval.
the other is memory for conversational retrieval. ai memory is still quite limited, especially if there needs to be a lot of token in context, and context too long impede the ability of llm of focus on the task itself, especially if the context is itself a conversation or a request, so spreading the context along a few agents, and propagating the user request among agent, and having those produce answer fragment for another llm to formulate an answer allows to not lose the conversational context without swamping the llm with noise.
the problem tho remains latency as son as you nest them latency explodes as you can only stream the last layer of llm output
“A lot of research has been doing in this are and we can expect a lot more in 2024 in this space. I promise to share some clarity around where I think this industry is headed. In personal talks I have warned that multi-agent systems are complex and hard to get right. I've seen little evidence of real-world use cases too”
These assistant systems fascinate me, but I just don’t have the time and energy to set something up. I was going to ask if anyone had a good experience with it, but the above makes it sound like there’s not much hope at the moment. Curious what other people’s experience are.
- Too many errors that just propogate on top of each other, if a single agent in the chain generates something even a little bit off then the whole system goes off the rails.
- You often end up having to pass a massive amount of shared context to every agent which just increases the cost dramatically.
Curiously enough we had an architect from OpenAI tell us the same thing about agent systems a few days ago (our company is a big spender so they serve a consulting function), so I don't think anybody is really finding success with multi-agent systems currently. IMO the core tech is nowhere near good enough yet.
LLMs are like the perfect improv comedy troupe, they virtually always say “yes, and…”
Check out Vtubers like CodeMiko, who improvs against LLM agents. Or 24/7 streaming LLM cartoon shows that take audience plot suggestions.
The ultimate answer is fairly short if you are a senior python data scientist, like 50loc. The agents will wander and iterate until they push through. You might correct & tweak if a bit off.
Importantly, this does agents opposite of the way Devin AI engineer replacements are presented. Here, you get it to do a few steps, and then move on to the next few steps. The agents still crank away a ton and do all sorts of clever things for you... to get you more reliably to the next step, vs something big & wrong.
We try to make the generated code and backing data explainable. Users are figuring out the scenario by having the AI go ahead for them, and automating much of the debug loop in typical coding and investigations, so folks can focus more on the analysis, less on syntax, schemas, libraries, and be more ambitious on each step.
Importantly, it is still kind of like making a much more accessible Jupyter notebook or editable excel/doc, vs a linear chat session. Instead of generating the whole notebook and it being buggy and you starting over (~= Devin, or notebook.io's ChatGPT plugin), you drive it forward only 1-3 cells at a time, and as it is an interactive document so you can edit those, go to the next, or non-destructively edit earlier ones. In contrast, ChatGPT's data assistant deletes cells below the current edit, which would stink in a normal data env.
There are other differences, but from a perspective of using genAI well, we budget 3-60s for genAI assisting in 1-3 steps, aiming for 10-100x productivity wins and a lot more peace of mind during it. Taking 1-3 steps forward may mean the AI takes 3-10 internally due to backtracking / CoT / etc
We could let the system take 100 turns, and have interesting experiments there such as around security investigations, but the use cases become more niche due to cascading errors => reliability.
- Agents need lots of manual tuning and guardrails to make them useful
- Agents with too many guardrails are not general-purpose enough to be worth the time and effort to build
I believe truly great agents will only come from models whose weights are dynamically updated. I hope I'm wrong.
Would be great to be able to ask it, 'have we completed the X process with contractor Y yet?'
> In my opinion, exploration of multi-agent systems is going to require a broader audience of engineers. For AI to become a true commodity, it needs to move out of the Python origins and into more popular languages like JavaScript , a major fact on why I wrote Experts.js.
I wholeheartedly agree
What is unfriendly about this?
import OpenAI from 'openai';
const openai = new OpenAI();
async function main() {
const stream = await openai.chat.completions.create({
model: 'gpt-4',
messages: [{ role: 'user', content: 'Say this is a test' }],
stream: true,
});
for await (const chunk of stream) {
process.stdout.write(chunk.choices[0]?.delta?.content || '');
}
}
main();
> assistant.on("textDelta”, () => …
Callbacks, which are not async and can’t be streamed that way directly without wrapping it in some helper function.
(Which does seem obvious; I’m also not sure why they called it out specifically as not being async friendly? I guess most callback style functions these days have async equivalents in popular libraries and these ones don’t)
Is this right? Aren’t you prematurely unwrapping the promise here?
It's like having good test coverage - you can make large changes and if the tests pass (the code compiles), you can be fairly confident that you didn't mess anything up.
I've written Ruby for years, so I'm used to dynamically typed languages. But JavaScript is it's own level of special, and there's so many ways you can accidentally mess things up.
Having tests cover every single path (especially failure paths) can be very time consuming, and often hard or messy to setup (how would you mock the OpenAI module returning an error when adding metadata to a thread?), where as using something like TypeScript can make sure your code handles all paths somewhat correctly (at least as well as the types you defined).
Your code looks clean, and you appear to have good test coverage, so you do you though :-)