Second, giving access to APIs... gives you access to APIs, and that's it. Even the most advanced LLMs are dumb as shit when it comes to anything non-trivial in programming. What makes you think that a modern LLM will manage to translate "move tab X to a different window" into a dozen or more API calls with proper structures filled in, and in the correct order?
I don't sense that GPT4 is "dumb as shit". I sense that it's extremely capable and very close to changing everything if, for example, macOS completely integrates GPT4.
Of course it's not trivial if you think for more than a second about it.
LLMs produce output in exactly three ways:
- text
- images
- video
What you think is trivial is to convert that output into an arbitrary function call for any arbitrary OS-level API. And that is _before_ we start thinking about things like "what do we do with incomplete LLM output" and "what to do when LLM hallucinates".
You can literally try and implement this yourself, today, to see how trivial it is. You already have access to tens of thousands OS APIs, so you can try and implement a very small subset of what you're thinking about.
BTW, if your answer is "but function calls", they are not function calls. They are structured JSON responses that you have to manually convert to actual function calls.
You could in theory send ASTs as JSON.
Please show me how you will do that for the tens of thousands of OS APIs and data structures.
Edit: because it's not just "a function call" is it? It's often:
struct1 = set up/get hold of a complex struct 1
struct 2 = set up/get hold of a complex struct 2
struct 3 = set up/get hold of a complex struct 3 using one or both of the previous ones
call some specific function 1
call some specific function 2 using some or all of the structs above
free the structs above, often in a specific order
{ "function_name": "X", parameters: [...] }
Don't forget about failure modes. Where you have to check for the validity of some but not all structs passed around.
Repeat that for any combination of any of the 10k+ APIs and structures
It's the trivial part of all this. You're getting too much into the details of what is available to developers today. Instead, you should focus on what Apple and Google would do internally to make things as easy for an LLM as possible.
Up until now, the hardest part was always understanding what exactly users want. GPT4 is a thousand times better than what Siri and Google Assistant are currently doing.
Again, mapping OS APIs is the easiest part. By far.
You keep skipping the question of how.
> It's the trivial part of all this. You're getting too much into the details of what is available to developers today.
Somehow you have this mystical magical idea of "oh, it's just this small insignificant little thing".
Literally this:
LLMs understand human requests
* magic *
Things happen in the OS
And yet we haven't seen a single implementation that does what you want.
> Again, mapping OS APIs is the easiest part. By far.
If it is, it would make it trivially easy how to do this trivial and easy task for a small subset of those APIs, wouldn't it? Can you show me how you would do it?