github.com
We are working to apply the ideas of R1 to computer use. The primary struggle is creating reliable neural reward models since hard-verification rewards are not available at scale in GUI interactions.
Our team is currently deep in the weeds of collecting reasoning annotation data for GUI interfaces to train a reliable reward model.
We would love all thoughts, feedback, and collaborations!
Sorry to be a party-pooper, but does it really make sense to add a citation when you don't have fully working code yet, let alone a paper about it?
"We aim to replace hard-coded.."
Sad.
We are working to apply the ideas of R1 to computer use. The primary struggle is creating reliable neural reward models since hard-verification rewards are not available at scale in GUI interactions.
Our team is currently deep in the weeds of collecting reasoning annotation data for GUI interfaces to train a reliable reward model.
We would love all thoughts, feedback, and collaborations!
But if you still want to try out this path, Google has made the screenQA dataset(rico) available[2] along with bounding boxes.
1. A framework to use local/hosted models for android use/control - https://github.com/BandarLabs/clickclickclick
TL;DR:
- Turns out that if you do UXR, even if computer use is 100% successful in the action execution, and there's no latency, people don't use it. (interesting to me is, the core demo was buying airline tickets, and so is OpenAI's. no one would defer to a computer on that, for humanist / design reasons)
- You're not going to be able out-do model companies on building models, they have too much funds.
- Try writing GUI-based integration tests. Then imagine an LLM, miraculously, always chooses the right route. Does the UX look good?
- Note the reasoning models are worse at tool calling. It's very, very, VERY stark when you have Claude next to o1/4o. OpenAI also owns up to this in the o3-mini paper, though its not under a blaring red line headline or phrased that straightforwardly.
- Why is that? You're fighting against the current when you're trying to teach the next token predictor to throw a bunch of text out there to <think>, then generate perfectly correct JSON/python/whatever given N tools.
CLI, though....
It is possible for an human to inject features into notepad.exe without having access to the source code (I learn that in cracking tutorials from +HCU almost 30years ago...) it should be possible for an AI to do so.
I would never buy plane tickets that way, we built it because there are tons of things we couldn't automate and this was the only way to do it
> - You're not going to be able out-do model companies on building models, they have too much funds.
There are plenty of people edging out the model companies all over the place. We aren't powerless to them
> Note the reasoning models are worse at tool calling. It's very, very, VERY stark when you have Claude next to o1/4o. OpenAI also owns up to this in the o3-mini paper, though its not under a blaring red line headline or phrased that straightforwardly. Why is that? You're fighting against the current when you're trying to teach the next token predictor to throw a bunch of text out there to <think>, then generate perfectly correct JSON/python/whatever given N tools.
The reason why they are bad at tool calling is they aren't trained on it. The current reasoning models require hard-verification reward models, we don't have those for tool calling, whereas that data is easy to get for math/code. Reasoning will improve tool calling, OpenAI just talked about it recently as being the answer to autonomous agents
Yes they are.*
> OpenAI just talked about it recently as being the answer to autonomous agents
You nailed it.
It's key to autonomous agents, OpenAI says so out loud, and yet, we're 6 months into reasoning models and performance is regressing, which OpenAI also says out loud but in the fine print of a model card.
I know this is a flaming hot take because current thing is reasoning. But it's completely mistaken that reasoning models help with tool-use, both in theory and practice, which puts them in quite a situation.
I'm sure they'll figure it out, but I'm also sure on a long enough timeline the LLM is a computer.
* I have a bad habit of dismissing without evidence that which is asserted without evidence, and 'not even wrong', in the Pauli sense. It's a cheap way to avoid confrontation. But it makes me look petulant. We can observe, inter alia, every release post September 2024 (i.e. o1-mini and o1-preview) can call tools. (i.e. o1, o3-mini)
Reasoning will obviously improve agentic workflows, reasoning is the main problem we see today with autonomous agents. It seems to be likely the _last_ issue we have with them
- It's going to be difficult to find a source that says heavily on tool use. They don't talk about training at all anymore! :( And saying you trained heavily on tool use would imply your model isn't trained well for other stuff, which they'd want to avoid.
- Nothing implies they didn't do the exact same tool training they have been. Both model cards mention tool use.
- Both o1 and o3-mini have bog-standard tool calling via API. I'm not sure what gives us the sense that they weren't trained on them
- The claim shifted significantly:
From: "The reason why they are bad at tool calling is they aren't trained on it"
To: "[I don't think you can find a source that says] o1 [was] trained heavily on tool usage."
Google, Meta, Microsoft and other RTB firms send RTB data about people in the U.S. to Russia and China and anyone else who signs up...many people don't see the difference. In fact for many people, the CCP having your data is far less of a risk vector than the thousands of others who get your data every single time you hit a webpage, visit the local store etc...
When Google, Meta, Microsoft etc... are selling your data thousands of times a day, and companies aggregate that to sell even sensitive information completely based on even national security sensitive categories.
https://www.iccl.ie/wp-content/uploads/2023/11/Americas-hidd...
https://www.eff.org/deeplinks/2025/01/online-behavioral-ads-...
* it's not a "random third party". You know to whom the data is being sent, and at least according to service agreements, most services don't use your data for training. If you don't trust Claude, you could trust AWS hosted version, or GPT/Deepseek hosted on Azure. Well, if you think Amazon/Microsoft is not trustworthy and they may misuse your data in these cloud services (not some random consumer facing service where you are the product), you might as well give up your digital life.
This is a claim that really irks me (when companies make it). It’s a non-denial denial. “We don’t train on user data” is NOT the same as:
- We don’t retain user data
- We don’t share user data with business partners
- We don’t mine user data for business ideas or ways to compete with our users’ products
- Etc.
Running local models don’t protect you from prompt injection attacks or hallucinations.
There are some startups building capabilities apis to limit that but most websites/apps either don’t have the resources or aren’t willing to expose those capabilities.
And as some others have mentioned, users have a track record of giving up privacy for convenience. I’m not convinced educating non-technical users about the risks involved will ward them off.
Care to elaborate with example(s) of a startups doing this?
UI-TARS by bytedance also has a good amount of pretraining.
Molmo is also very good at coordinates.
also, how big of a gain to have reasoning for computer use? i feel like reasoning unlocks a lot when there is a single complex question but not so much better at taking actions in a long term plan.
We had previously created https://huggingface.co/datasets/agentsea/wave-ui but that was superseded by pixmo as it contains over a million data points.
Computer use is pretty exciting stuff in general though, good luck
People have tons of workflows that involve a lot of clicks and typing in response to data that are too difficult or one-off to automate with fragile macros.
But if my computer can quickly realize that I'm deleting every odd-numbered page of a PDF, or renaming every file to add a prefix, or following each link on a website and saving an image... and then just instantly automate the next 100 times... that's going to be huge!
The first two tasks could be easily done by asking ChatGPT to write a script for you. Scraping a website can be a bit more tricky. Still, I don't see why you have to rely on "computer use" for these tasks -- there are much more efficient and reliable approaches to the tasks.
There's a gigantic area of productivity improvement around repetitive actions that aren't easily scriptable or no scripting interface exists. But where an AI assistant that interfaces with your screen, pointer and keyboard would be a huge help.
That's about manually setting up agents, that run on a server, that seem to interact largely with the web (from the examples).
I'm talking about not manually setting up anything -- I'm talking about an AI that simply observes the repetitive actions you're taking on your computer, infers patterns from them, and then offers to take over and finish the job.
I think we should start with something simple, repeatable, and does little to no harm if/when things go wrong.
Edit: repetitive -> repeatable