It’s almost like we need some deterministic set of instructions that can be fed to a machine and followed reliably? Like… I don’t know… a “programming language”?
You need something general and flexible, dare I say "intelligent", or you'll be babysitting the automation, slowly adding the thousand little corner cases that you find, to your hard coded decision tree.
This is also why every company with a home service robot, that can do anything even remotely complex as a sandwich, are doing it via teleoperation.