Thanks! We are releasing an API soon along with some common CI/CD pipeline integrations. We're working hard on speeding tests up so that we don't slow down your pipeline.
The issue with this approach is that for all but the most simple apps it is not possible to deduce the runtime element information needed to write traditional UI tests given just the source code. This can only be done reliably at runtime which is what we do. We run your app and iteratively build UI tests that can be reused later.
1. The AI agent writes an automation script (similar to Appium) that we can replay after the first successful run. If there are issues the AI agent gets pulled back into the loop.
2. You can define acceptance criteria in natural language with camel.
Thanks! Yes, we experimented with that! I think because of the way that GPT sees images in patches it has a hard time with absolute positioning but that's just a guess.
Great questions.
1. Yes, running tests in parallel helps. We also cache actions so subsequent runs are much faster (this is disabled in the demo).
2. I agree that testing can be much more reliable and pleasant in some codebases than others. I have not been blessed with these types of codebases in my career. Flakiness is from personal experience automating UI tests specifically and having them break when a new nondeterministic popup modal is added or another engineer breaks an identifier/locator strategy.
That being said, if you like writing UI tests and your codebase supports easily maintaining them, there are some really cool DSLs like Maestro!
GPT-4V is great for reasoning about what is on the screen. However, it struggles with precision. For example, it is not able to specify the coordinates to tap when it decides to tap an icon. That's where the object detection and accessibility elements help. We can precisely locate interactive elements.
Good questions. Execution cost is indeed higher than traditional testing automation scripts but much lower than the human cost of writing and maintaining the scripts. We're starting at $500/month and our plans go up from there depending on how many devices you want to test across. We do have customers running across multiple devices and OS versions today.
Thanks for the feedback! We'll add some styling to the demo page. We're processing the image with an object detection model and classification model and also using some accessibility element data to get a better understanding of what is interactive on the screen.
Cool idea. Is there any way to steer the image generation? I couldn't find any way to. (p.s. site is pretty slow right now. not sure if that's due to high load or something)
Oh this is really cool! Since the repl is just executing wasm, could this not work in the browser? I realize you'll probably need to create a new wasm module every time you run a statement but could still be neat!