I thought the median had to be significantly lower with those ultra luxurious rentals distorting the average, but I was wrong. The median is $4000 as of a few months ago: https://www.elliman.com/corporate-resources/market-reports
Cheap apartments are illegal and the legal ones don't get financed/constructed. There's a cutoff much higher than zero for housing costs everywhere in the US and I'd imagine it's pretty high in NYC.
Where did you see the $4k number? This looks like condo and co-op numbers mostly?