Yeah, add to that the famous Elon Musk's troll quote, "working from home is unethical, because other people can't all work from home - think about all the people growing your food etc". Yeah, Am I also supposed to feel bad because I work normal working hours and others work at night? What about the people that have to lift heavy objects all day and do back breaking labour, and I "sit or stand at my desk all day". How about those that have a 3h commute while I walked to work from my city center apartment (back when I did work on site).
People will always find something to beat you over the head with. The most important thing is not to let them infect you with their negativity.