Yet! Seriously, lets just focus on building robots to do things like build roads and buildings, tend farms, drive vehicles, etc. etc. We could almost definitely pass human efficiency in the next 50 years, and nobody would ever have to do manual work again (although if that's something somebody likes doing they should totally have the option if that's what they want).
Also, I'm not convinced a job in construction is "similarly skilled" to office and admin work. But that's kind of beside the point, it's possible to earn a good living from either manual or sedentary work. Sure, the extremely high earners are exclusively sedentary workers (if they work in the traditional sense at all), but that speaks to the wider problem of wealth distribution, where the fabled top 1% have 20% of the countries total wealth.