The next iteration in LLM products is a 24/7 thinking loop where the claude-code like thing gets input continuously from your wearable, notifications, and newsfeeds and is constantly preparing things for you.
Then you have your cheap little MoE or ternary model just running in a loop, with an escalation pathway before it reaches the big expensive models.
Currently it's doing things like reminding me to take allergy medication when I wake up because it's checked AQI or whatever, reminding me to stop at the market when I'm on my way to pick up the kids to get the cherry tomatoes I forgot, giving me heads up of what folks are expecting from me in certain meetings based on cross correlating email and calendar, etc.
It's honestly the single most productive tool I've found for my ADHD.
People would do well to acquire at least a modest familiarity with what an LLM actually is. NLP is fascinating. So is entropy.
I’m sure you have reasons for your setup though, so I’m curious how you landed on it.
I'm not confident it's that most efficient way to do it, but it's quite a bit of fun.
Run the same exact prompt 100x in a single step test (one prompt, one response), hash the full responses, and you’ll see 10-50+ unique responses. The higher the unique the worse your prompt; focus on the system. A highly tuned system prompt will result in one response, even at a temperature of 1.0. Really. Once that’s done that’s the only thing it does and it’s the only one that does it and it never changes. Other LLMs that call ‘check_email()’ are unknowingly just passing a prompt to the specialized one.
I use the API directly, craft a small Python script for the API call and task interface, then hyper-optimize the system/user prompt using test scenarios and automated loops. My system prompts rarely/never contain complete sentences, yet include all the tools/functions and requirements.
Make your error messages user prompt instructions, not errors. That’s why “agent optimized” models exist. Chat models are primarily trained on conversational text, meaning the stackoverflow “How do I fix ‘too many levels of symbolic links’?” -> Explanation/resolution. It’s far less on “# ls broken_loop” -> “# ls: cannot access ‘broken_loop’: Too many levels of symbolic links” -> “# namei -l broken_loop”
It’s not that the good ones are bad, but you’re leaning on the million training documents rather than the trillion.
Anyway, go that route with your system. Think about it more like automating a factory floor rather than hiring interns.
The least efficient methods, by definition, have the most room for improvement, which means they have the greatest reward potential, but for that one “eureka” moment. The path less taken is often interesting, but the ill-advised path still has fruit on the trees.
This is a brilliant idea, thank you. Convergence as a metric for prompt robustness.
Couple tips:
A first pass is to blank out the system prompt, add only one tool, and work to reduce the thinking length for a direct function call prompt. “Read archive.log” should result in roughly 0 length thinking. If it’s thinking about anything, especially if it mentions {readfile tool}, rename the tool and minimize the description. Depending on the model it might always output thinking, so run it until you get a consistent outlier that’s far lower thinking length than the others. It’ll be obvious when you find it. Repeat the prompt dozens of times, modify it slightly, and focus on the lowest max length, not average.
You should really use a Claude with Python (or similar preferred) to make API calls to the LLM and have it iterate through hundreds of names/descriptions and return only len(thinking). Have it build a batch testing harness to run a dozen tests at a time, that helps keep it from ‘cheating’ to finish. laziness = count(messages), but frame it as an academic research project studying the effects of minimalist tool descriptions on thinking length. Don’t set the goal as minimal thinking length, Claude will short circuit it.
Remove all other tools until {readfile} is perfected, then add/test the next tool. Btw: you don’t need to describe readfile() when it’s named right.
The built-in tools[] makes that hard because it tacks a really dumb system prompt on at the server and requires some length of description, which is why I built my own function calling, but that’s still a good first pass. Focus almost entirely on the function name itself; readfile, readFile, read_file, readlines, file_get_contents, etc., and make the description just “Operational” or similar. Field description, if required by API, is literal “filepath”, same as field itself. Lowercase, nothing else said. Minimize your contribution to perplexity, use standard naming conventions.
When you add a second tool you need to still include the first tool prompt in the second tool testing. Adding {writefile} can absolutely break {readfile}. Have Claude run the tests and build it out into permanent testing module with file_read=[prompts], file_write=[prompts], making it easy to extend, and full_test() that runs them all to see if a new addition broke it.
Add your system prompt back in and probably watch the tests go to shit. <- THAT is likely your biggest problem. My system prompt for the main LLM has all of the tools it can use, which is ~30 lines of function names with no call syntax, and yet it has more tools than Claude Code and never messes them up.
Start with nothing and slowly work up. Focus on positive action framing, not negating: “Your responses are always..” and not “Do not…”
It sounds like a pain, but building the systems to automate the tests IS the infrastructure, everything you have it do afterwards is just the tasks.
That was longer than I planned, but I guess this’ll be a comment for future generations to find.
It seems like the purpose of humans isn't to keep machines busy. When our phone or laptop is idle, it's fine if it sleeps. And when we do want something, we'd rather not wait.
(Also, this new model seems to be designed to keep latency down, which is useful for interactive tasks.)
I'm not sure what the original commenter had in mind, but just because our machines are idle when we aren't using them doesn't mean that, that's how we will use computers in the future.
I think notifications are an example even now of the computer not really being idle when we aren't interacting with it.