Structured requests and responses are 100% the next evolution of LLMs. People are already getting tired of chatbots. Being able to plug in any backend without worrying about text parsing and prompts will be amazing.
Yup, a general desire of mine is to locally run an LLM which has actionable interfaces that i provide. Things like "check time", "check calendar", "send message to user" and etc.
TypeChat seems to be in the right area. I can imagine an extra layer of "fit this JSON input to a possible action, if any" and etc.
I see a neat hybrid future where a bot (LLM/etc) works to glue layers of real code together. Sometimes part of ingestion, tagging, etc - sometimes part of responding to input, etc.
All around this is a super interesting area to me but frankly, everything is moving so fast i haven't concerned myself with diving too deep in it yet. Lots of smart people are working on it so i feel the need to let the dust settle a bit. But i think we're already there to have my "dream home interface" working.
I have a working solution to exposing the toggles.
I’m integrating it into the bot I have in the other repo.
Goal is you point to an openapi spec and then GPT can run choose and run functions. Basically Siri but with access to any API.
Does this sound similar enough to what you were doing? Was there something difficult in this that you could explain?
Aside from being completely hand-wavey in my hypothetical guess-timated implementation, i had figured the most difficult part would be piping complex actions together. "Remind me tomorrow about any events i have on my calendar" would be a conditional action based on lookups, etc - so order of operations would also have to be parsed somehow. I suspect a looping "thinking" mechanism would be necessary, and while i know that's not a novel idea i am unsure if i would nonetheless have to reinvent it in my own tech for the way i wanted to deploy.
I'm guessing the solution looks like a model trained to take actions on the internet. Kinda sucks for those of us on the outside, because whatever we make is going to be the same, brittle, chewing-gum and duct tape approach as usual. Best to wait for the bleeding edge, like what that MinecraftGPT project was aiming at.
`useMakeCopilotActionable` = you pass the type of the input, and an arbitrary typescript function implementation.
https://github.com/RecursivelyAI/CopilotKit
Feedback welcome
For example, try to keep up with (frequent) API payload changes around a consumer in Java. We implemented a NodeJS layer just to stay sane. (Banking, huge JSON payloads, backends in Java)
Mapping is really something LLMs could shine.
Code/functionality archeology is already insanely hard in orgs with old codebases. Imagine the facepalming that Future You will have when you see that the way the system works is some sort of nondeterministic translation layer that magically connects two APIs where versions are allowed to fluctuate.