Show HN: Structured Python control over AI computer use agents | Hacker News Reader

Show HN: Structured Python control over AI computer use agents

github.com

Most CUA frameworks are either a black box or raw tool calls with no structure. Orbit sits in between, natural language controls the screen, Orbit lets Python control the flow. Each step has its own model, budget, and typed output, but shares context across the session. Mix cheap and expensive models per step, extract structured data from any screen into Pydantic models, and steer the agent mid-task when it struggles. Built on the OS accessibility tree, not screenshots.

https://github.com/aadya940/orbit

Open articleView on HN

I'm curious what drove you to build a new agent framework from scratch? In your repo you say that existing frameworks are black boxes, but most of the mature ones are open source and you have access to full source? I'm just wondering because I'm also thinking of building an agent framework but then I wonder if it's worth the effort unless it's for educational purposes.

Most agents are prompt + a bunch of tool calls. Writing everything in one prompt is like writing your entire app in one function. It works until it doesn't, and when it doesn't, you have no idea where it broke. Orbit gives you steps and python control flow instead of prompts, so failures are local, models are swappable, and budgets are per-action. This makes it debuggable.

For example, Your `Read` step failed after 3 LLM calls on step 4 of 7. With a monolithic prompt, that's just... it hung.

But, it seems like that's a simple implementation of a particular framework? I mean, I think they do more than just take a prompt and execute tool calls? But, I appreciate the response and time. Thanks!

Fair, mature frameworks do a lot more. Orbit isn't trying to out-feature them. It's a thinner abstraction specifically for computer-use, where the bottleneck isn't orchestration complexity, it's controllability at the step level. Different problem.

Reply on news.ycombinator.com