Comment by sprobertson | Hacker News Reader

Comment by sprobertson | Hacker News Reader

Parent Full thread

Great demo, straight to the point. It might be nice to have some kind of feedback mechanism if it can't find the element on the page, or if it's partially cut off. For example, I changed the GitHub profile to my own (https://github.com/spro) in the example and it doesn't scroll down far enough for the whole image. I imagine in general it would be nice to scroll to an element ID (or even element description using the vision models) instead of a hardcoded value.

Side-note: The comment for the frequency graph is wrong, it mentions stars instead.

RE: feedback mechanism -- yep, a feedback mechanism is definitely something we're thinking of adding. Since we use VLMs that are trained to always output coordinates (i.e., they don't have a way to say "not on the page"), we're probably going to try fine tuning with some negative examples to see if we can build that feedback mechanism in.

One way to hack the scrolling to an element is to first run extract_bbox on a natural language description (in your case for GitHub it might be "follow button") then take the Y coordinate of that element and scroll that number of pixels. I just wrote this bit of code that I tested and it brings the contribution graph into full view:

    simplex.goto("github.com/spro")
    coords = simplex.extract_bbox("follow button")
    simplex.scroll(coords[1])
    simplex.wait(2000)
    image = simplex.extract_image("green tile graph")

But then it incorrectly picks the code review/submissions/etc. graph as the green tile graph -- we'll look into it!

re: frequency graph typo -- just pushed a fix, thanks!

Reply on news.ycombinator.com