If you can't figure out why the market still has brokers, you are destined to make the same mistakes that everyone who sunk time and money into this before has made.
100%, this is a societal (political or otherwise) problem to solve - it’s not for lack of an app to facilitate the tx.
The result is that there is a shortage of housing in NYC and this system pops up to try and extract economic rents from the situation.
Unfortunately, the economics of building new housing on extremely expensive real estate, targeting an "affordable" price point just don't incentivize as much construction as the city needs.
Also know that the oft-shared window we get on it from individual newcomers to NYC isn’t one the market particularly cares about. Brokers hustling 3br in Sunset Park to quartets of Oberlin grads are the starving underclass of New York real estate industry. They know it and, in their overworked misery, condescend to their market.
Avoiding a brokers fee is a popular tip given to new renters…but it’s availability is largely a function of privilege. If you have the right background and connect with a landlord who appreciates that, you have a fighting chance. (Finding a building you like and asking existing tenants if they know of anything opening is the recommended path to this.) It’s still a low probability event without connections.
The standard fee is 1 months rent and this is almost always paid by the tenant.
The arguments here tend to cluster into two groups: 1) Laws should be passed that the landlord needs to pay the realtor's fee and not the tenant. 2) It doesn't matter if #1 happens as the landlord would just bake it into the rent anyways.
EDIT: I should say it's not just this reason as what you cite does happen. But it's just de rigueur here regardless of if you're renting the upstairs unit of a homeowner or going through a property management company.