Yes, great point! We have an 'Assistant' feature where you can perform the flow on the device, and we automatically generate the test case as you navigate the app. As you mentioned, it’s a great starting point to quickly automate the functional flow. Afterwards, you can add more detailed assertions as needed. Technically we do this by using both the UI hierarchy from the app as well as vision models to generate the test prompt.

This comes up all the time. It seems like it would be possible, but imagine the case where you want to verify that a menu shows on hover. Was the hover on the menu intentional?

Another example, imagine an error box shows up. Was that correct or incorrect?

So you need to build a "meta" layer, which includes UI, to start marking up the video and end up in the same state.

Our approach has been to let the AI explore the app and come up with ideas. Less interaction from the user.