Hacker News Reader

Comment by minimaxir | Hacker News Reader

Overall, Operator seems the same as Claude's Computer Use demo from a few months ago, including architecture requiring user to launch a VM, and a tendency to be incorrect: https://news.ycombinator.com/item?id=41914989

Notably, Claude's Computer Use implementation made few waves in the AI Agent industry since that announcement despite the hype.

View on HN

Big jumps in benchmarks from Claude's Computer Use though.

87% vs 56% on Webvoyager

58.1% vs 36.2% on WebArena

38.1% vs 22% on OsWorld

These are next gen improvements so the fact that Claude didn't make any waves doesn't really mean anything (Of course no guarantee this will either)

OpenAI is merely matching SOTA in browser tasks as compared to existing browser-use agents. It is a big improvement over Claude Computer Use, but it is more of the same in the specific domain of browser tasks when comparing against browser-use agents (which can use the DOM, browser-specific APIs, and so on.)

The truth is that while 87% on WebVoyager is impressive, most of the tasks are quite simple. I've played with some browse-use agents that are SOTA and they can still get very easily confused with more complex tasks or unfamiliar interfaces.

You can see some of the examples in OpenAI's blog post. They need to quite carefully write the prompts in some instances to get the thing to work. The truth is that needing to iterate to get the prompt just right really negates a lot of the value of delegating a one-off task to an agent.

Well that's fair. I wasn't saying that this was necessarily at a level of competence to be useful, simply that it seemed to be a lot better than Claude.

> OpenAI is merely matching SOTA in browser tasks as compared to existing browser-use agents.

No. It's not matching them, it's clearly exceeding them. The previous post provided the numbers.

Those numbers are not the full story. Note that GP specifically says: "Big jumps in benchmarks from _Claude's Computer Use_ though." Claude Computer Use was not SOTA for browser tasks at the time of its release (and is still not.)

In WebArena, Operator does 58.1%. Previous SOTA for browser-use agents is 57.1%. In WebVoyager, Operator does 87.0%. Previous SOTA for browser-use agents is the exact same.

See here for details: https://openai.com/index/computer-using-agent/

Those two were two different models (Kura and jace.ai), and one model being SOTA at one benchmark doesn't make it SOTA overall. Moreover, both are specific for browser use, so they don't operate only on raw pixels but can read HTML/DOM, unlike general computer use models which rely on raw screenshots only.

I think I hit all those points in my previous post, except for the fact that it's two different models, as you've noted. That said, neither of them seem to report scores for the other benchmark in each particular case.

Yeah, and Browser Use already has 89% on WebVoyager https://browser-use.com/posts/sota-technical-report

Gemini is 90.5% in Webvoyager[1] compared to 87% for OpenAI.

[1]: https://deepmind.google/technologies/project-mariner/

I thought Claude Computer Use is through API, and I remember hearing about high number of queries and charges.

This looks like its in browser through the standard $20 Pro fee, which is huge. (EDIT: $200 a month plan so less of a slam dunk but still might be worth it)

Is there any open source or cheap ways to automate things on your computer? For instance I was thinking about a workflow like:

1. Use web to search for [companies] with conditions

2. Use linked in sales navigator to identify people in specific companies and loose search on job title or summary / experience

3. Collect the names for review

Or linked in only: Look at leads provided, and identify any companies they had worked for previously and find similar people in that job title

It doesn't have to be computer use, but given that it relies on my LinkedIn login, it would have to be.

If you are worried about costs you can use Browser Use with deepseek which becomes super cheap! https://github.com/browser-use/browser-use

Exactly what I was look for. Thank you. I wish they had Gemini since the free tier is generous but I guess it's in the works. I'll take a look and see how bad it would be implement

Browser use supports gemini via langchain

> Is there any open source or cheap ways to automate things on your computer?

MacOS has had Automator since 2005. It's far more like "programming" to use than a 2024-tier ML based system, but it was designed for non-programmers, and lots of people do use it.

Personally, I hate it.

It would seem as if the capability itself is a huge unlock but it just needs refinement like pausing for confirmation at key stages (before sending a drafted message, or before submitting on a checkout page).

So the workflow for the human is ask the AI to do several things, then in the meantime between issuing new instructions, look at paused AI operator/agent flows stemming from prior instructions and unblock/approve them.

Like a general instructing an army.

Correction on "including architecture requiring user to launch a VM": apparently OpenAI uses a cloud hosted VM that's shown to the user. While that's much more user friendly, it opens up different issues around security/privacy.

38% on osworld vs 22% for Claude. That seems like a jump

But of course, after all the benchmark issues we've had thus far -- memorization, conflicts of interest, and just plainly low-quality questions -- I think it's fair to be suspicious of the extent to which these numbers will actually map to usability in the real world.

Claude's Computer Use API has been good for us and I'm surprised it isn't more popular. It can be slow, and definitely gets things wrong, but so far we've had thousands of people make and edit spreadsheets on "autopilot" so the value is already there today on simple tasks, even in an alpha state.

I do find it is best when combined with other capabilities so the internal reasoning is more "if Computer Use is the best for solving this stage of the question, use Computer Use. Otherwise, don't.", instead of full Computer Use reliance. So e.g. you might see it triggered for auto-formatting but not writing SQL.

Will report back how it compares vs Operator CUA once we get access!

This is mainly to reclaim mindshare from DeepSeek that has done incredible launches recently. R1 was particularly a strong demonstration of what cracked team of former quants can do. The demo of Operator was nice but I still feel like R1 is the big moment in the AI space so far. https://open.substack.com/pub/transitions/p/openai-launches-...

R1 is a fundamental blow to their value proposition right now, the uniqueness is gone, and forever open sourced. Unless o3 is the game changer of game changer, I am not seeing they are getting the narrative back soon.

You can use browser-use as open-source alternative for Operator

possible to use it with R1 for the reasoning part ?

Yeah I saw someone do this on Twitter recently... Can't seem to find it, as you do with twitter

Reply on news.ycombinator.com