I’m not sure how unique or a new problem this is first individually to me and then generally.
Automation technologies to handle things like UI automation have existed long before LLMs and work quite fine.
Having an intentionally imprecise and non deterministic software try to behave in a deterministic manner like all software we’re used to is something else.