Safety particularly. In 1977 on the heliport of the Pan Am building (aka the MetLife building), a helicopter's landing gear broke which caused the helicopter to tip over and the rotor to break. Pieces of the rotor killed four people on the roof, fell to the street below, and killed another person on the ground. It took emergency responders about an hour to get onto the roof because the elevators in the building shut down. The public was already wary of the danger of helicopters (as well as being annoyed by the noise), and this incident confirmed their fears. Since then, heliports on buildings in NYC have been severely restricted, for a time banned completely, I believe, but 3 rooftop heliports are still active in NYC.

Another helicopter crash in 2019 suggests another danger. A helicopter crash landed on the roof of a skyscraper and caught fire. The pilot was the only person killed that time, but it took firefighters about an hour to put the fire out. They apparently got it under control without too much difficulty and were lauded for a quick response.. but what if that helicopter had been packed full of lithium batteries? Lithium battery car fires are notoriously hard to put out. Such a fire on the roof of a tall building seems like a serious concern to me.