Thanks for the response Matt. I leave the link here for others to look into [1].

Their effort seems currently limited to iOS based Phone screens. iOS is perhaps easier to solve given the strong Apple design guidelines for apps to pass the App Store review process.

Perhaps a community supported distributed approach to help build the database of annotated screens for the model to learn from, combined with open source models for all kinds of screens and applications( not just Apple) would be interesting project to work on.

[1] https://machinelearning.apple.com/research/creating-accessib...