- Do you want an LLM salesbot to close a deal your company isn't actually able to fulfill?
- Do you think your company will use AI more intelligently and reliably than the people who made a popular LLM coding system?
I want my AI support rep to create tasks to engage my team, with all the relevant data linked to it. It should be able to automatically schedule things and ask a human for confirmation.
It should be able to elevate communication to a human in the loop, using whatever mediums of communication makes sense given staff availability and workload.
At no point is it allowed to answer any questions unless the answers are constrained and probably directly quoted from a cited and linked section in our documentation.
In general, it should never confirm or deny things, it should never try to close things or acknowledge the content of any of the user's communication other than to call tools and surface public information which might be relevant to their request.
Most of this is a software architecture problem. The LLM is just there to provide an intuitive and extremely powerful natural language interface for search and tool calling. A little bit of glue between different systems, both internal and external.
If you were an end user of such a system, would you be happy?
It's an incremental stop along the way to truly reliable agentic systems which we can trust with important things.
I suspect Cursor told themselves that they are super-smart AI experts who would never make an amateur mistake like the airline, they will use prompt engineering + RAG. With this, it will be unpossible that the LLM could make a mistake.