It might help to consider how stable an arbitrary human would be at this task.

User interfaces are like pieces of artwork. You really don't know until it's finished, and even then it's not for everyone.

LLMs are most effective when working with human parsed representations of websites. They are terrible at doing the actual parsing.

They're also good at working on a relative basis. One png screenshot of the viewport is a difficult hill to climb. The difference between two screenshots is much more informative. Deltas usually tell a much bigger story than dead reckoning against the base world model.