How are you planning to train a model based on annotated actions done on a page? Wouldn’t it be more practical to have a model like PIX2ACT that understands the action to be performed and does it? Or are you planning to compose future actions from decomposing flows?