Teaching GPT-5 to Use a Computer

Very cool. I've been thinking for a while that this is where things will end up. While custom AI integrations per service/product/whatever can be better and more efficient, there's always going to be stuff that doesn't have AI integrations but your workflow will need to use.

Without this, AI is going to be limited and kloodgy. Like if I wanted to have AI run a FEA simulation on some CAD model, I have to wait until the FEA software, the CAD software, the corporate models repo, etc., etc. all have AI integrations and then create some custom agent that glues them all together. Once AI can just control the computer effectively, then it can look up the instruction manuals for each of these pieces of software online, and then just have at it e2e like a human would. It can even ping you over slack if it gets stuck on something.

I think once stuff like this becomes possible, custom AI integrations will become less necessary. I'm sure they'll continue to exist for special cases, but the other nice thing about a generic computer-use agent is that you can record the stream and see exactly what it's doing, so a huge increase in observability. It can even demo to human workers how to do things because it works via the same interfaces.

I imagine in the future someone will make an Agent-First OS that is entirely built from the ground up to be run by AI and runs off the assumption that there are no human users or that their usage is limited. That will be interesting, imagine all the things you could do differently, the design choices you could make. You lose a lot by accommodating human ergonomics.

I really like this approach. Nice job!

> We also plan to compile solved steps into micro‑policies. If you're running something like a RPA task or similar workflow as before, you can simply run the execution locally (with archon-mini running locally) and not have to worry about the planning. Over time, the planner is a background teacher, not a crutch.

Conceptually, I really like this - why re-do the work of reasoning about an already solved task? Just do it again. For some plausibly large majority of things, this could speed things up considerably.

> In the future we hope to run a streaming capture pipeline similar to Gemma 3. Consuming frames at 20–30 fps, emitting actions at 5–10 Hz, and verifying state on each commit.

I love targets like this. It makes you tune the architecture and abstractions to push the boundary of whats possible with a traditional agent loop.

The salience heat map compression is a great idea. I think you could take this a step further and tune a model so that it compresses an image into a textual semantic/interactive element hierarchy. This is effectively what browser-use is doing, just using javascript instead of a vision model.

This seems like a task that would benefit from narrow focus. I'm aware of the "Bitter Lesson," but my intuition seems to tell me that chaining together fit to purpose classification as an input to an intelligent planning system is the way to go.

For someone who wants to do “archon‑mini is a 7B Qwen‑2.5‑VL–based executor (dynamic‑res ViT) fine‑tuned with GRPO for GUI grounding” part at home, is there a guide/post you would recommend?

It’s pretty interesting.

I see a ton of potential for testing. RPA can quickly get annoying because even simple change can break automation. The LLM ability to “reason” could really bridge the gap.

Coupled with agents who help turn specification/stories into a testing plan, I could really see automated end to end testing becoming far cheaper than it is nowadays in the near future.

That’s a very good piece of news for system reliability.

This is cool though wanted to share a couple of thoughts for reflection:

I feel like your demo video is not the greatest one to highlight the capability. A browsing use case likely does require a key press->planning loop, but a gaming use case, or a well known software (e.g., excel), may be able to think ahead 10-20 key presses before needing the next loop / verification. The current demo makes it seem slow / prototype-like.

Also, the X/Y approach is interesting when thinking about a generic approach to screen management. But for example for browsers, you're likely adding overhead relative to just marking the specific div/buttons that are on screen and having those be part of the reasoning (e.g., "Click button X at div with path XX"). It may be helpful to think about the workflows you are going after and what kind of accelerated management you have over them.

Nice writeup! A few years ago I proposed to a friend that he should try rendering accessibility trees and fine-tuning a model to issue tool calls over them; I don't know if this has been tried and failed or if nobody bothered trying because so few people know desktop APIs anymore. The main advantage would be accuracy/speed and avoiding the need for too many image tokens (you still need them for things that are actually images, though).

This is very interesting, my thinking has been along these same lines. This covers some of the more complicated engineering tasks required to pave the way for LLMs to automate diverse tasks with much higher accuracy and lower cost. Building the ultimate RPA as it were.

I can't help but feel like some sort of hybrid approach: use GPT5 for the strategy, then a more direct ML model for actually executing the strategy might work better than trying to use reasoning directly for input control, would work better than trying to reason your way through driving.

Can I use Archon? There is a use case I’ve wanted to explore and this tool can do most of it

Can we please squash the idea of turning this into a product early, so no one starts to take it seriously? Agentic AI shouldn't be hooked up to any environment that doesn't have a flawless "undo" or "rewind 5 minutes" button. Would you hook up randomized mouse/keyboard input into a computer that you need to get work done on?

At the very least, display a very strongly worded warning message when the tool is run in a non-VM environment. Internet connectivity is still dangerous, but at least VMs can be snapshot. And it should not be packaged as an end-user product with a bar at the top of the screen, period.

This is a risky product and those who developed it have every ability to know that, based on the history of AI hallucinations. Not because it will escape or self replicate or other things claimed by the various idiotic AI religions, but because one of the first things it will inevitably do is screw up someone's work and cause data loss.

This is great, once we get LLMs playing Doom, we'll all be free to touch grass.

Imagine getting beat by a bot and it also has the capability to talk trash to you.