Hacker News Reader

Comment by adeelk93 | Hacker News Reader

I really like this, but I wonder if a better approach would be to take that simplified DOM and instead generate playwright or puppeteer code instead of direct DOM manipulation. That way it’s reproducible browser automation.

View on HN

That's the direction I'd like to take this longer-term. The major advantages are: lower latency, lower costs and better auditability/reproducibility each time you run a workflow.

The major disadvantage is less resiliency if the structure of the page changes, eg. if a site adds a "subscribe now!" modal or something you have to click through to get to the content. But that's solvable (if desired) by falling back to the LLM if the scripted steps don't produced the expected output.

We actually started our implementation in this direction but decided to change course and focus on direct LLM manipulation to iterate and launch faster. But we'll likely get back there at some point, especially as we build support for scheduled/automated workflows.

Awesome, thanks for the reply! The fallback approach sounds like it could be really great, hardening against modals and popups can be really annoying.

While your example of "site adds a modal" is true, with browser based regression tests, the brittleness I find much more often is minor changes to the DOM, which don't have a large impact on what the user sees. (Sometimes this is due to refactoring the DOM, sometimes it's because the DOM doesn't have good class names to use as handles, and sometimes it's because I want to reuse a step on 2 different pages, but the rendered DOM is slightly different).

My mental model of how I think about drift (which might be totally worthless from a LLM perspective):

1) I give a prompt

2) snapshot of dom is taken

3) gpt looks at that snapshot DOM and implements some solution which works

4) that solution is transformed into a more concrete implementation

4i) whether this "transformation" is any of the following isn't SUPER important, and to be honest I'd love to see all 3: a) selenium/playwright code writen by Taxy, b) a hybrid of explicit code and a gpt prompt, or c) developer can override with fully custom code

4) it runs correctly for X amount of time

5) application dom changes

6) taxy notices the step is failing

7) taxy takes a new snapshot of the dom

8) taxy runs a (reinforcement?) algorithm against the new snapshot and confirms it finds the "same" dom element as the one from old snapshot.

unrelated: the other thing which I've found very hard to program into my browser tests (and makes the code hard to interpret), so I'm curious how gpt/taxy could help:

Given I have this dom:

<ul>

  <li>

    <div class="tweet">

      <h1>Check out my sandwhich!</h1>

      <button>Retweet</button>

    </div>

  </li>

  <li>

    <div class="tweet">

      <h1>Check out my shoes!</h1>

      <button>Retweet</button>

    </div>

  </li>

</ul>

I want to write test code which is:

1) When I load the page, I should see a tweet "Check out my sandwhich!"

2) I can retweet that tweet.

Currently, I either need todo:

a) a dom traversal: find(text: "Check out my sandwhich", css: ".tweet h1").parents(".tweet").find(text: "retweet"). It's that `parents(".tweet")` part which becomes awkward at scale and incentives developers to only create 1 tweet in the test database....

b) use Page Objects, which I love, but adds overhead/training for the team

I would love if gpt could figure out, these 2 elements are "related". :)

Long-term, generating reproducible code will be important for speed/economics. However, even then the LLM will still have to calculate its actions one at a time when creating the reproducible workflow, since Taxy will have to deal with html elements being added and removed from the DOM after each action, not to mention navigation to other webpages.

Reply on news.ycombinator.com