The most crucial things missing in OpenAI's implementation for me were:
- Authentication for the API by the user rather than the developer.
- Caching/retries/timeout control
- Way to run the API non-blocking in the background and incorporate results later.
- Dynamic API tools (use an API to provide the tools for a conversation) and API revisions (for instance by hosting the API spec under a URL/git).
And even if they did, it would be poor UX to have the user have to visit our site first to connect their API accounts.
I also imagine many tools wouldn't run under the developers' control (of course you could relay over your server).
Requests you make to the service providers are made on your own buck, you are supposed to track user stuff on your end. It would make no difference chatgpt wise who the user is, that's not part of the abstraction provided.
Not a user auth SaaS, an LLM SaaS.
Like when you use an online PDF editor with Google Drive integration - paying for the storage etc is between Google and the user, the files belong to the user's Google Drive account, but the PDF editor gets read/write access to them.
This implementation of function calling works differently from those.
OpenAI/Anthropic don't make any API calls themselves at all here. You call their chat APIs with a list of your own available functions, then they may reply to you saying "run function X yourself with these parameters, and once you've run that tell us what the result was".
This is useful for more than just tool usage - it can help with structured data extraction too, where you don't execute functions at all: https://til.simonwillison.net/gpt3/openai-python-functions-d...
How? There is no execution model. The LLM simply responds, in JSON format, with the name of a function and its corresponding arguments in alignment with the JSON Schema spec you provided beforehand. It is entirely on you to do something with that information.
At the end of the day it is really not all that different to asking an LLM to respond with JSON in the prompt, but offers greater stability in the response as it will strictly adhere to the spec you defined and not sometimes go completely off the rails with unparseable gobbledygook as LLMs are known to do.
You have an http api, implement all of this yourself, the devs can't read your mind.
You should be able to issue a request and do stuff before reading the response, boom non-blocking. If you can't handle low level, just use threads plus your favourite abstraction?
User API auth. Never seen this by an api provider, you are in charge of user auth, what do you even expect here?
Do your job, openai isn't supposed to magically solve this for you, you are not a consumer of magical solutions, you are now a provider of them
About non-blocking: I am asking for their tools API to not block the user from continuing the conversation while my tool works. You seem to be thinking about something else.
To be fair, that was very ambiguous (talking about API’s and non-blocking IO) and their initial assumption was the same as mine (and quite reasonable).