LaVague: Open-source Large Action Model to automate Selenium browsing
github.com
github.com
So far all of these are… not working except for trivial cases. This one is also choking on basic saas sites; especially the ones with spinners while getting content. Notice that this type of tool would be great for the millions of enterprise ‘internal app’ garbage ‘integration’ that is now done manually by copy/pasting data from pdf to email to excel to app1 to app2 to app3 to excel to email to app4 to app5 to word to email etc etc. But, because before the latest ssr fad, everything was client side loading SPAs with a billion spinners so many of those departmental/enterprise apps/saas are that. None of the solutions named here can handle that properly, so in the end it’s a frustrating experience of repeating yourself 10 times with maybe one success.
The cases with static or fully ssr sites was not really needing much automation (although it could fix breaking changes to the site automagically); those are trivial with existing tools already. Just a little bit of manual setup (the right selectors).
But I know it's early days! There is a reason I test ALL of these every few months.
My thought was to put the results of this in a vector store, with any errors that resulted as opposed to wasting time training a model.
1. https://brajeshwar.com/2021/how-to-delete-all-photos-and-get...
1. https://www.amazonforum.com/s/question/0D56Q0000BMJvWOSQ1/do...
2. https://chromewebstore.google.com/detail/amazon-order-histor...
We are thinking of developing an extension that would connect the browser to LaVague so that actions can be sent to the extension and be executed locally, thus bypassing their barriers
Apple also makes it nearly impossible to get full purchase history from app stores. The only place left is Music, Account, Purchases, Custom range, All Year, checkmark all types, checkmark all family members -- then it's in a tiny vertical scroll pop up with no cut and paste. With everything as IAPs, extracting 500 of these a year at 3 - 4 at a time is tedious.
Were competitive intelligence apps or browser extensions using user browser creds to surveil or surreptitiously steal entire purchase histories?
[1] AI Employe: https://github.com/vignshwarar/AI-Employe
[2] BrowserPilot: https://github.com/handrew/browserpilot
HN is awesome!
I'm kinda surprised why you chose to open source this instead of slapping AGPLv3 like all the YC funded github projects are doing
how can i contact you
The “problem” with AGPL is companies who want to use open source software to build proprietary stuff on top without contributing anything back. AGPL is purposely designed to avoid this kind of parasitic behavior, but that doesn't make it “not open source” quite the opposite: it's “forced open-source”.
It is indeed restricting companies' freedom though: their freedom to restrict their user's freedom.
It seems like quite the intuitive approach, but we quickly discovered that due to differing implementations, and the reliance on the apps actually cooperating with you, it's actually so much more reliable and much faster to use OpenCV to physically detect UI elements by appearance.
I'd love to tell it to just log in to my own website, click on certain pieces of functionality and repeat that. Especially with more casual day to day tasks.
Heck, we could even auto-generate tests from a bug report (where the steps to reproduce are written in plain english by non-technical testers).
That means less time for a dev to actually reproduce those steps, right?
Every time we detect, for instance with a vision model, that the interface changed, we ask the Large Action Model to recompute the appropriate code and have it be executed.
Regarding generating tests from bug report totally possible! For now we focus on having a good mapping from low level instructions ("click on X") -> code, but once we solve that, we can have another AI take bug reports -> low level instructions, and use the previously trained LLM!
Really like your use case and would love to chat more about it if you are open. Could you come on our Discord and ping me? https://discord.gg/SDxn9KpqX9
Do yourself a favour, use Playwright instead.
It's a headless browser that's both faster and less flaky than Selenium.
I don't think it's caught any real bugs yet because I haven't actually broken anything but the playwright script keeps running reliably it includes a login and fills out a big long complicated form. Works great. Very quick. Selenium was slow and unreliable.
UI front-end tests are often brittle because people try to test things through them that should have been tested in earlier stages. Either on API level or unit level.
Just to give a simple example. Say you have a login screen. It has a username input, password input, login button and finally a div to show any messages.
The only things you actually want to test here are:
1. A succes login action 2. A action that leads to a message being shown in the message div. 3. *If* there are multiple categories of messages (error, warning, etc) possibly one of each message.
What you don't want to test here are all sorts of login variations that ultimately test input validation (API level) or some other mechanism surrounding password (possibly unit testing).
The problem is that often, and certainly the decade earlier you are talking about, is that companies often take their manual regression tests and just throw them into automation. Forgetting that those manual regressions tests are equally brittle but that this is often overlooked due to the way manual tests are done and reported on.
Having said all that. Selenium is still a solid option in a java environment. But as others have pointed out, there are other very solid options out there like Playwright. But these also can be equally as brittle if the tests are not setup properly.
Last I heard, Y Combinator is seeking startups that can automate "Back Office" work.
However, the net gains, in my humble opinion, could be phenomenal. Imagine all the time, mental energy and money spent on navigating through the legacy of today's society? From the legacy legal systems that is super complex, to legacy websites, I believe there is much time to be saved so we can dedicate resources to what truly matters, intellectual pursuits or quality time with friends and family
And historically, have always been phenomenal.
If 100 years ago, you told people that only 1.5% of people in USA/Canada would work in agriculture, politicians would have been horrified and in fear of mass unemployment. They would have been similarly horrified if you told them that virtually nobody would work in textile manufacturing in the Western World.
But in reality, the jobs in the former are considered so dismal that they are heavily staffed by desperate people who have no other legal work options and migrant workers from poor countries and jobs in the latter pay so poorly globally that you would be better off running a lemonade stand in a Western country.
We are far better off for the combine harvester freeing us from harvesting wheat by hand. We are far better off for the sewing machine.
Who's "we"? It's not like the people who aren't working with a scythe have moved up to be un-employed computer programmers, they're just picking fruit now.
People who were sewing by hand as a professional don't generally get the afternoon off now to chill with their homies, they just use the sewing machine all damn day.
The only "we" who is better off are consumers and business operators, because they pay less or nothing for that labour. Nobody is talking about the comfy lives of fast fashion makers or the people who assemble our $7000 MacBook pros.
Doesn't seem like a very humble opinion, every time people lose work they need to find income somewhere else or end up working more anyway. Productivity gains equalling more free time has only really ever worked for people who end up or who were already unemployed or self-employed, otherwise it's propaganda spread by people who stand to gain. Even in cases where someone's job became only less manual, it's not like they suddenly got the rest of the day off to spend with their family, they just ended up operating the machine all day anyway, and often getting paid less to do it, to a point where eventually families and friends as a concept started becoming more rare.
I can see the business perspective for sure. But I really don’t think humanity have the luxury to consume even more energy to run billions of GPUs to do what a programmer team could do and in the meantime having an excuse to not fix its legacy.
That sounds like either totally cyberpunk or very late stage capitalism.
We need to reduce global energy consumption and fix the society as much as we can, not going full throttle in the current direction.
Maybe now we can get closer to completely eliminating some jobs? But I think this challenge will still present itself.
I used to be an innovation analyst at a bank and we looked at automating tasks quite frequently and found that many could be automated. But you are right on the money for why it did not happen.
Tasks are straightforward to automate. Entire job roles are not. If you want to save headcount, you need to automate some tasks and then rethink one, if not several, job roles. That is a lot messier to do.
In most cases, we decided not to bother as we didn't think there would be a net savings.
Robotic process automation has been on the scene for a number of years doing exactly this, and is quite a bit more mature.
I agree that this kind of tool has the potential to take more jobs, but companies looking to do this kind of thing have had a number of options available for awhile now. New tech like this will accelerate the trend.
I think one of they pieces to do that is actually being able to explain, not just silently watch your screen, and ask questions,make it a dialogue, even once that you might get pinged on later if they hit a snag or a situation changes and they need confirmation of something.
RPA today is really nothing like that.
Some RPA products have improved this using computer vision so they can more reliably click on the right things.
But I agree that the introduction of natural language is new. But I see that as primarily a change in interface, not outcome. i.e. eliminating tasks that involve systematically doing the same things over and over already has options. This new generation of tech just makes it far easier. I’m sure RPA tools will incorporate it.
I’ll also be curious to see how this kind of thing translates to legacy thick clients where access to the DOM can’t be used to “understand” the interface.
I just did so on one of the more terrible ones yesterday - and the dark pattern was it would put you in captcha-loops... and youd have to reload/retry several times before stopped asking you firehydrant bus traffic motorcycle crosswalk over and over.
but to save unsub/delete me scripts with this would be nifty.
A recipe bounty would be neat - for example - Optery found me in more PII dbs than I expected - and it would be cool for people to see which brokers they are found in and there is a bounty list for all the brokers people are finding for someone to create a Delete-Me for each thing, so that one hopefully has the help of many to navigate the minefield of dark patterns in such.
Lots of paper show that fine-tuning only helps with steerability and form (https://arxiv.org/abs/2402.05119), therefore I thought it would be sufficient to provide just the right examples and it did work!
We do intend to create a decentralized dataset to further train models and have maybe a 2b or 7b model working well
Much more sophisticated activity than anonymous polling, like “political temperature” on social media, has also been broken for probably a decade, if not more.
If you’re building a public facing product today, you most certainly have to account for incentives of malicious (or just rational self-interested) actors. A bit of rudimentary game theory and adversarial thinking goes a long way.
Once we solve browser automation, we intend to support other integrations to further facilitate automation of workflows
Granularity and explicitness is often written off as expensive in this space but throwing a large model at a largely solved problem with existing tools and techniques seems spirit of the times.
Your goal is to write Selenium code to answer queries.
Your answer must be a Python markdown only. You can have access to external websites and libraries.
You can assume the following code has been executed: ```python from selenium import webdriver from selenium.webdriver.common.by import By
driver = webdriver.Firefox() ```
---
HTML: <!DOCTYPE html> <html> <head> <title>Mock Search Page</title> </head> <body> <h1>Search Page Example</h1> <input id="searchBar" type="text" placeholder="Type here to search..."> <button id="searchButton">Search</button> <script> document.getElementById('searchButton').onclick = function() {{ var searchText = document.getElementById('searchBar').value; alert("Searching for: " + searchText); }}; </script> </body> </html>
Query: Click on the search bar 'Type here to search...', type 'selenium', and press the 'Enter' key
Completion: ```python # Let's proceed step by step. # First we need to identify the component first, then we can click on it.
# Based on the HTML, the link can be uniquely identified using the ID "searchBar" # Let's use this ID with Selenium to identify the link search_bar = driver.find_element(By.XPATH, "//[@id='searchBar']")
search_bar.click()
# Now we can type the asked input search_bar.send_keys("selenium")
# Finally we can press the 'Enter' key search_bar.send_keys(Keys.ENTER) ```
---
HTML: <!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8"> <title>Mock Page for Selenium</title> </head> <body> <h1>Welcome to the Mock Page</h1> <div id="links"> <a href="#link1" id="link1">Link 1</a> <br> <a href="#link2" class="link">Link 2</a> <br> </div> </body> </html>
Query: Click on the title Link 1 and then click on the title Link 2
Completion: ```python # Let's proceed step by step. # First we need to identify the first component, then we can click on it. Then we can identify the second component and click on it.
# Based on the HTML, the first link the link can be uniquely identified using the ID "link1" # Let's use this ID with Selenium to identify the link link_to_click = driver.find_element(By.XPATH, "//[@id='link1']")
# Then we click on the link link_to_click.click()
It then goes on with several other examples.
Basically its few shot learning, with a few examples of instructions, HTML and selenium code to execute those instructions. Then you give it your new instruction and it generates selenium code for those.