I feel like the real reason we don't have robots doing all of our physical grunt work, is that it is just way too expensive to develop robots that do advanced and custom things. Currently only big players with deep pockets can consider developing software for a robot that automates a task that requires some level of fine grained control or visual feedback.
The computer vision part is challenging as you need deep subject knowledge, you need to implement complicated algorithms and assemble huge datasets. We are working on a way to improve the efficiency of creating datasets, standard algorithm implementations and standardizing interfaces to make things as composable as possible.
It ain't much, but it's a start. https://www.strayrobots.io