We've seen that people behave worse when you introduce indirection. People act worse on the internet. Soldiers have an easier time killing with drones than in person. The ethical issue is in both directions: its inhumane to the operator, but I also don't want to feel like a fake person on a video screen to them.
This is then exacerbated when you realize that the people operating this machine are almost certainly not being paid well, creating obvious and legitimate negative incentives. Then you plop them into the households of people with the insane wealth required to afford this. You might think that I have just described the situation with maids (and to some extent, I agree! I have never really felt comfortable that dynamic either), but this is actually different, because you are adding in the indirection and making actions and interactions feel less "real" to both parties: the clients are likely to treat the robots worse than they would a human helper, and the operators may feel these rude clients they see on their monitors aren't as real as the people around them.