I'd argue it's quite the opposite!
If I stay at home, I'm one less person compressing them in the metro, or helping gridlock the highways or taking up parking spaces (I live in the city where parking is scarce for pretty much everyone). If I can work remotely every day, I'm likely to leave my apartment in the city center and go live somewhere far away. That's one less person competing for housing.
So in some ways (i.e. housing prices), you working from home does directly make these people lives worse, as you are now their competition for real-estate.
But if enough people like me move out from the city, there's a possibility that prices will go down in the cities, right? So that should help the people who need to be physically present at work.
I think it's fair to expect employees to exert the mental effort to empathize with the companies they work for when the companies show effort to do likewise.
Early in my career, a senior colleague used to reply to complaints with the simple statement "you only get the company/boss that you deserve".
By that, he meant that if you complained, you were always free to leave and find a better fit elsewhere. I learnt to appreciate that saying as I grew older.
If you think you deserve a better office space / coworkers / manager / CEO, then go look for it elsewhere. If you don't find it, then you are probably in the place you deserve, like it or not.
I didn't have to go outside that day, but I'll never forget how lucky I am and that other people still have to.
But what difference does it make? I can feel sorry for people all day but I can't stop the rain.
On the other hand, nobody listened to people like me who were saying open offices were bad for us and we can't concentrate properly in them. So much for empathy.