I'd like to have the names of all programs visible in the screenshot (easy), possibly application specific metadata like the opened filename or a URL (more difficult) and more generally full OCR of the visible text (pretty easy). You'd need a PDF to get the most out of this, but presumably most other image formats have generic metadata storage.
Regarding open programs: I once led a project where we developed something that keeps track of the software you're running (no screen recording) in order to conduct research into attention and distraction. We didn't have the resources to support many platform versions, so we wrote only a Windows client (most used OS on the floor). It was similar to RescueTime https://www.rescuetime.com/ but more respecting one's privacy and absolutely avoiding the cloud, as we deployed the experiment in a lawyer-intensive environment; for instance we logged running program names but not titles of open windows because file names often reveal sensitive matter. We questions occasionally how productive they felt in the last half hour, and they could comment.
Why can't we le the ML models figure classification themselves and then give them the human data to adjust to be readable by the human.
You can set the File name using various parameters in combination like - time date - program name - window title
This can also be configured to do complex workflow like
- for each screenshot add a border then copy image to clipboard and then upload to Imgur and copy url finally run and ocr and copy results as well.