Are you sure about the physical labor part? Because I'm yet to see a machine make fruit pickers, plumbers, construction workers, oil & gas workers, landscapers, or caregivers even remotely not viable anymore in the workplace. Meanwhile I'm seeing machines put various graphic designers out of jobs right now.
Who's jobs do you think the companies will go after first in the race to replace expensive labor with cheaper automation? The minimum wage burger flippers or migrant fruit pickers barely paid 10$/hour who gets no vacation and sick leave, or, the web developer/graphic designers paid $60+/hour who gets healthcare, paid time off, free meals and massages at work?
What is an actual reality is that more and more physical work is actually being completed by fewer and fewer people.
In my lifetime I've seen first hand, since the 1970s, mining and agriculture expand production dramatically with smaller work forces being required through the multiplying magic of SBFM (Seriously Big Flippin' Machines) - Haulpacks, Tractors, Ships, Trucking, Trains, have grown is size; blast patterns for fracturing rock have increased their field sizes, continuous conveyors moving 100s of tonnes per hour have multiplied in parallel.
As for fruit picking, maybe look at orchard automation - there are many examples of tight geometric spacings for trees that are robo-mowed to supress weeds and harvested by laying down ground sheets and using hydralic tree shakers to bring down and gather up fruit, etc.
Maybe watch more Landline - https://en.wikipedia.org/wiki/Landline_(TV_series)
The companies go after the jobs that are culturally hip to go after. I initially wrote something here trying to describe which jobs should really be automated but it was so vague I now think we are clueless until we try.
You could [say] hardcode the asshole employee manager and he/it would no doubt be worse than the human asshole in more aspects than one but he/it would also have many advantages that we cant pretend to enumerate without doing the experiment.
Who is to say we cant feed the world if we build enough autonomous chicken farm guard bots with high speed solar powered egg drones?
Who is to say what a 60/hour web dev can do with the help of some robots?
Many countries are starting to see a critical lack of farmers, fishermen, hunters, and other people working in agriculture. Healthcare is also starting to suffer.
The robot automation will happen, probably either in our or the next generation's lifetimes, because there just aren't enough workers either due to lack of will (poor pay) or lack of literal humans.