We used to use a similar approach for testing page objects in stb-tester. It was called stbt auto-selftest (it has since been removed). It would run your page objects against canned screenshots and generate a Python doctest capturing the behaviour of this page objects. You’d then commit the doctest to git and have ci set up to check correctness. See https://david.rothlis.net/pageobjects/characterisation-tests

This was really helpful - we could then see in or pull requests how changes to our page object code would affect the behaviour of that code - just look at the diffs.

It worked ok for a while, but we soon hit limitations with this approach. The big one is the interaction with git. We would get many merge conflicts in these files, particularly when rebasing to reorder commits. Sometimes we’d forget to regenerate the files and have to rerun and rebase after submitting to ci. It worked okish for us as we are sophisticated git users comfortable with rebase and filter-branch, and we’d designed the system so could debug issues effectively, but it wasn’t suitable for wider use.

We’ve moved away from this model to a slightly different one. We still generate this data[^1], but we store the data outside of git in a database indexed by git commit sha. We then display this data and associated diffs in a web-ui. We use GitHub checks to wire it back up to pull requests, so it’s just as visible as when it was committed to git, but more convenient and displayed in a nicer format. You can use the same web-ui to approve changes to this data.

[^1]: though in more structured form than doctests