For almost all the smaller buildings in New York (non-professionally managed), which is where you see these middlemen brokers:
* The brokers are given an exclusive by the landlords because the landlords are not professionals and often not in New York City. They don't want to do any work, they want the brokers to do the work of listing the apartment, showing the apartment, and checking the qualifications of the tenant.
* The landlord has to give the broker an exclusive otherwise the broker won't do the work, because the broker needs to get paid.
* Although the broker fee (normally 15% of first year rent) seems expensive and even extortionate to the typical fresh college grad renting their first apartment, the brokers as a whole are not making very much money, because they have to do an awful lot of work to get the exclusives (marketing themselves to landlords, so to speak) and an awful lot of work to coordinate keys, show apartments, etc. It is easy to see this because there is massive turnover in the broker business and almost all of the rental brokers you see in New York charging "extortionate" fees are, for the most part, broke.
* Which is why they seem so slimy: they're kinda desperate.
* A ton of apartments in New York are still under some form of rent control, especially the cheap ones in small buildings that kids want. That means that the maximum rent is set by law. So even if a tenant MIGHT be willing to pay a higher rent if they didn't have to pay the brokerage fee, they can't. The landlord of a $1800 rent stabilized apartment is going to get $1800/month whether they use a broker or not. So for this landlord, the broker is paid for by the tenant, and might as well be free. Considering how much work they do, and how much they charge, zero, it's a great deal for the landlord.
So every time one of these fresh-eyed college grads shows up in New York, decides that the rental brokers are making bank and need to be disintermediated, and writes an App that will Connect the Landlord Directly to the Tenant, no landlords sign up for it. The bottom line is that the brokerage function is not ripe for disruption because, slimy as it seems, there's just not that much money being made.