Non-practitioner in either field ...
If you don't focus on the more or less "defined" ML subset, and focus on the AI in the question, whatever "AI" has meant over the years, then ...
The reason why AI is hard and physics simple may be that physics already "is," while we may not yet have defined what AI should be and what to do with it .
Physics has guardrails of reality, however subtle and hidden they may be. What is described by physics is what it is, and will be there as it is, whether we correctly describe it or not, and whether we observe it or not. (Depending on your view of an observed universe's dependant relationship with its observer.)
AI is ... what? In any particular decade? We aren't describing anything that already is, and that comes with constraints. We're making it up.
Specification is hard.
The universe as described by physics, on the other hand, is well beyond specification and requirements gathering. We're just writing the missing manual.