I'm not sure you're reading it right. What it says is that 20% of the vacancies (as of that time) were those set aside for seasonal or occasional use. But that has more to do with the very low vacancy rates that Manhattan has had essentially forever, than with the absolute number of such apartments.
While the knock on effect you mention could have a legitimate effect, with only 1% of the apartments affected it's hard to believe it's terribly significant. Particularly since at the stratospheric levels we are talking about (high level bankers and lawyers) the substitutes for UES/UWS spreads include Long Island and Westchester rather than just other parts of the city.
Queens & Brooklyn dwarf Manhattan in size. Queens has only one or one and half neighborhoods with Manhattan exiles and even though Brooklyn has more, the non-trendy ares are still much larger.
This whole perception of a crisis is fomented by young, hip, creatives who think they have a good given right to live in the West Village. There's still plenty of safe, affordable rentals in places like Flatbush, Kew Gardens, Pelham Bay, and so forth.