Anerli wrote: “When the interface does eventually change, that's where it becomes non-deterministic again by necessity, since the planner will need to generatively update the test and continue building the new cache from there.”

But what determines that the UI has changed for a specific URL? Your software independent of the planner LLM or do you require the visual LLM to make a determination of change?

You should also stop saying 100% open source when test plan generation and execution depend on non-open source AI components. It just doesn’t make sense.