I mean you could build the agent with a first principles understanding of the DOM instead of just hacking together with the accessibility tree
189 karma · joined December 19, 2024
There are plenty of zero day exploit patches that Google immediately rolls out and not to mention all the other features that Google doesn't push to Chromium. I wouldn't trust a random open source project for my day-to-day browser.
Check out rtrvr.ai for a working implementation, we are an AI Web Agent browser extension that meets you where your workflows already are.
The problem isn't a lack of tools; it's the painful and manual labor to setup, configure and connect to them.
Pre-defined MCP tool lists/directories, are inferior for several first-principle reasons:
- Reinventing the Auth Wheel: The key improvement of MCP's was supposed to be you get to package a bunch of tools together and solve the auth issue at this server level. But the user still has to configure and authenticate to the server using API key or OAuth.
- Massive Context Pollution: Every tool you add eats into the context window and risks context drift.
- Brittleness and Maintenance: If an API on the server-side changes, the MCP server must be updated.
- The Awkward Discovery Dance: It's a clunky user experience to discover the right server and then manually configure, defeating the purpose of seamless automation.
We propose a more elegant solution: Stop feeding agents tool lists. Let them build the tool they need, on the fly.
Our insight was simple: The browser is the authentication layer. Your logins, cookies, and active sessions are already there. An AI Web Agent can just reuse these credentials, find your API key and construct a tool to use. If you have an API key on your screen, you have an integration. It's that simple.
Our AI Web agent can now look at a webpage, find an API key, and be prompted to generate the necessary Javascript tool to call the desired endpoint at the moment it's needed. This approach:
- The user now just has to prompt an agent as opposed to the prior manual steps of finding a MCP Server, setting up, and authenticating to it.
- Keeps the context window clean and focused on the task at hand.
- No maintenance/debugging overhead. An API update? Just point the agent to the new docs and it will configure an updated version.
We wrote a blog post that goes deeper into this architectural take and shows a full demo of our agent creating a HubSpot tool from API key on page and using it in the same multi-step workflow of then loading contacts from LinkedIn with the new tool.
We think this is a more scalable and efficient path forward for agentic AI.
The full write up is here: https://www.rtrvr.ai/blog/on-the-fly-toolgen
The whole idea of cloud browser agents is a stupid paradigm. The agents are not only 7x slower but have the cost of hosting and proxying for that extra time!
Our own biggest cost is just LLM inference, thus we can just let our users bring their own API Key and use our service for free!
In fact, we built one, rtrvr.ai that has even better Web Agent performance than Open AI's Operator with human assistant and 7x faster than leading competitor: https://www.rtrvr.ai/blog/web-bench-results
Your Accessibility Tree requirement is a poor excuse, rather you should build up an agent from a first principles understanding of DOM interactions.
A browser is a SERIOUS security risk, you need a dedicated team to just pull in the latest security patches that Google pushes to Chromium or your users are sitting ducks to exploits and hacks...
But even these approaches face limitations as after a certain number of steps the AI Agent starts to lose context, repeat steps, or just go completely off the rails.
At rtrvr ai, we're building an AI Web Agent Chrome Extension that orchestrates complex workflows across multiple browser tabs. We followed the Manus approach of setting up a planner agent that calls abstracted sub-agents to handle browser actions, generating Sheets with scraped data, or crawling through pages of a website.
But we also hit this limit of the planner losing competence after 5 or so minutes.
After a lot of trial and error, we found a combination of three techniques that pushed our agent's independent execution time from ~5 minutes to over 30 minutes. I wanted to share them here to see what you all think.
We saw the key challenge of AI Agents is to efficiently encode/discretize the State-Action Space of an environment. Building on this insight, we setup:
Smarter Orchestration: Instead of a monolithic planning agent with all the context, we moved to a hierarchical model. The high-level "orchestrator" agent manages the overall goal but delegates execution and context to specialized sub-agents. It intelligently passes only the necessary context to each sub-agent preventing confusion for sub-agents, and the planning agent itself isn't dumped with the entire context of each step.
Abstracted Planning: We reworked our planner to generate as abstract as possible goal for a step and fully delegates to the specialized sub-agent. This necessarily involved making the sub-agents more generalized to handle ambiguity and additional possible actions. Minimizing the planning calls themselves seemed to be the most obvious way to get the agent to run longer.
Agentic Memory Management: In aiming to reduce context for the planner, we encoded the contexts for each step as variables that the planner can assign as parameters to subsequent steps. So instead of hoping the planner remembers a piece of data from step 2 to reuse in step 7, it will just assign step2.sheetOutput. This removes the need to dump outputs into the planners context thereby preventing context window bloat and confusion.
This is what we found useful but I'm super curious to hear:
How are you all tackling long-horizon planning and context drift?
Are you using similar hierarchical planning or memory management techniques?
What's the longest you've seen an agent run reliably, and what was the key breakthrough?
This would be more reliable than expecting the LLM to generate working code 100% of the time?
You request on app, and it sends you to a more central pickup/dropoff points
- you have app, and you enter destination
- optimal minibus reroutes itself to pick you up and take you there with mix of walking, while dropping off other passengers too
- minimizes the door to door time that makes cars so optimal
These tools they are guiding users to setup and execute are "inherently insecure" [https://issues.chromium.org/issues/40056642].
So if you go to a site that can take advantage of these loopholes then your browser is likely to be compromised and could escalate from their.
They do have a cloud offering that should not have these risks but then you have to enter your passwords into their cloud browser environment, presenting a different set of risks. Their cloud offering is basically similar to SkyVerne or even a higher cost tier subscription we have at rtrvr.ai
Perhaps we are talking past each other, your literally giving instructions to your users to connect to their actual browsers: https://docs.browser-use.com/customize/real-browser Where under the hood your launching Chrome with debugging mode but with the user's credentials and passwords. This browser is then controlled via CDP by a highly insecure browser-use binary running in a container. Your users are bound to get pwned with this setup! https://github.com/browser-use/browser-use/blob/70ae758a3bfa...
Your claim is analogous to saying that Apple's app store is not secure. We had to go through stringent vetting and testing by Google to list in the Chrome Store. Any basis or reasoning you can provide for your claim?
Regardless, its a wild leap to claim a Chrome Store Chrome Extension is more insecure than this arbitrary binary?
I meant by in production in the sense how you are advising your users to setup the local installation. Even if you launch browser use locally within a container but your restarting the user's Chrome in debug mode and controlling it with CDP from within the container, then the door is wide open to exploits and the container doesn't do anything?!
You are using debugger tools such as CDP, launching playwright without a sandbox, and guiding users to launch Chrome in debugger mode to connect to browser-use on their main browser.
The debugging tools you use have active exploits that Google doesn't fix because they are supposed to be for debugging and not for production/general use. This combined with your other two design choices let an exploit to escalate and infect their main machine.
Have you considered not using all these debugging permissions to productionize your service?
Say I figured a prompt, recording and other input to get an AI Agent to work exceptionally at say filling a job applications on LinkedIn, others can then reuse this exact input for their own job apps with the extensibility of our AI Web Agent Chrome Extension.
On the dataset side, we are targeting the long tail of datasets that people don't even bother to create because its expensive or tedious, like master list of Yemeni Coffee Houses in SF or all municipal audit files for municipalities in California. So yes for personal but more for prosumer or small and medium business use case!
I gave a sample artifact of over a hundred credit card APR's and other offer details extracted: https://www.rtrvr.ai/exchange/artifacts/L2U0jTbkwHTJ73nzSFgO
But let me know your thougths on rtrvr.ai, looks like we are targeting the same use cases of automation, scraping, research?