I have no basis of comparison, but this city really kicks ass. The restaurants, the activities, the opportunities for work. If something is holding you or your startup back, that something is not NYC.
Why would I choose to live and start a business in NYC when places like Seattle, Portland, Austin, Chicago, and DC all have drastically lower costs of living?
If you enjoy certain characteristics of urban living -- dense, progressive, diverse, bustling -- NYC is peerless.
Plenty of cities have these characteristics that have a much lower cost of living than NYC.
I love NYC. It's a great city, probably the greatest city on earth, but there are plenty of other options out there.
And then think about the tremendous amount of culture here -- plays, restaurants, music, groups of people, and so on, on a larger scale and with more diversity than anywhere else in the world except perhaps London. This might not all make a big difference for you, but it's the reason a lot of people choose to live in NYC.