Does my eating of an apple actively hurt people who are hungry in Africa? No, it doesn't. In fact, over the long run it will help them, because I would live longer to buy more to buy more of their goods and support them economically in the future with trade, etc.
You know what would actively hurt the poor in Africa? Me eating bread and water and dying and being unable to help the world in any future way ever because I'm dead. Me choosing to have apples and peanut butter and other vegetables and proteins is so that I can live longer and improve the world more, with either open source software, general economic contributions, studies of the weather and climate to reduce overall risk, etc.
Choosing to live on a diet that would kill me is not helpful to the poor in any part of the world.
But does this building's existence with enormous effort put into cheating the rules, skirting the system, and catering to the exclusively epicrich actively harm the middle-class and poor living nearby?
Yes it does.
The rich who live in this building are not doing so purely to help others, they are not there to improve society, they are there and are specifically wasting resources with no benefit but their own view of NYC.
It's obviously a very different situation.