Comment by unshavedyak | Hacker News Reader

Comment by unshavedyak | Hacker News Reader

Parent Full thread

> Structured requests and responses are 100% the next evolution of LLMs. People are already getting tired of chatbots. Being able to plug in any backend without worrying about text parsing and prompts will be amazing.

Yup, a general desire of mine is to locally run an LLM which has actionable interfaces that i provide. Things like "check time", "check calendar", "send message to user" and etc.

TypeChat seems to be in the right area. I can imagine an extra layer of "fit this JSON input to a possible action, if any" and etc.

I see a neat hybrid future where a bot (LLM/etc) works to glue layers of real code together. Sometimes part of ingestion, tagging, etc - sometimes part of responding to input, etc.

All around this is a super interesting area to me but frankly, everything is moving so fast i haven't concerned myself with diving too deep in it yet. Lots of smart people are working on it so i feel the need to let the dust settle a bit. But i think we're already there to have my "dream home interface" working.

I just published CopilotKit, which lets you implement this exact functionality for any web app via react hooks.

`useMakeCopilotActionable` = you pass the type of the input, and an arbitrary typescript function implementation.

https://github.com/RecursivelyAI/CopilotKit

Feedback welcome

I was thinking about this yesterday. ChatGPT really is good enough to act as a proper virtual assistant / home manager, with enough toggles exposed.

ChatGPT isn’t the limiting factor here, a good way to expose the toggles is. I recently tried to expose our company CRM to employees by means of a Teams bot they could ask for stuff in natural language (like „send an invite link to newlead@example.org“ or „how many MAUs did customer Foo have in June“), but while I almost got there, communicating an ever-growing set of actionable commands (with an arbitrary number of arguments) to the model was more complex than I thought.

Care to share what made it complex? My comment above was most likely ignorant, but my general thought was to write some header prompt about available actions that the LLM could map to, and then ask it if a given input text matches to a pre-defined action. Much like what TypeChat does.

Does this sound similar enough to what you were doing? Was there something difficult in this that you could explain?

Aside from being completely hand-wavey in my hypothetical guess-timated implementation, i had figured the most difficult part would be piping complex actions together. "Remind me tomorrow about any events i have on my calendar" would be a conditional action based on lookups, etc - so order of operations would also have to be parsed somehow. I suspect a looping "thinking" mechanism would be necessary, and while i know that's not a novel idea i am unsure if i would nonetheless have to reinvent it in my own tech for the way i wanted to deploy.

https://github.com/ShelbyJenkins/LLM-OpenAPI-minifier

I have a working solution to exposing the toggles.

I’m integrating it into the bot I have in the other repo.

Goal is you point to an openapi spec and then GPT can run choose and run functions. Basically Siri but with access to any API.

Good shit!

Interacting with APIs is the old style. The magic of ChatGPT is the same magic as google had back in the day - you ask it in plain english and it has an answer.

I'm guessing the solution looks like a model trained to take actions on the internet. Kinda sucks for those of us on the outside, because whatever we make is going to be the same, brittle, chewing-gum and duct tape approach as usual. Best to wait for the bleeding edge, like what that MinecraftGPT project was aiming at.

How about unix's (and plan9's more extreme version of) "everything is a file" philosophy? The gift that won't stop giving..

(How) did you solve this?

Write me an email if you’re interested in details, probably have some code to share. Address is in my profile.

Reply on news.ycombinator.com