That actually brings up an interesting concept that never occurred to me.
We could get automation of complex tasks far earlier than I had thought. We're at this divide where we're getting capable with our understanding of motion and mechanics, but intelligently figuring things out is a massive roadblock.
What if, as you say, we make very dumb robots where their interaction with the world has been entirely laid out by humans. Ie, a humanoid robot making toast seems difficult. Yet if we give it the exact dimensions of the room, rf sensors for locations of shelving, cabinets, bread, etc, it seems possible with our current tech that the robot could be programmed to handle dynamic environments as long as the inputs were laid out ahead of time.
There's limited value in something like toast making, defining the inputs would be a hassle. Yet, for house building, or factory automation, if every piece, shape, location and dimension was known it seems a fleet of robots could build a house with quickness and ease, no "intelligence" needed.
Humans would need to lay a foundation to a spec of dimensions, but then the bots could go nuts. Neat!