Penn Station, the way it is now is just something you sort of shrug at, given the way Midtown is, in general.
Everything on 8th Avenue from 23rd Street to 59th Street is seedy and weird, but the creeps and transients really occupy a strip between 34th Street and 42nd Street.
I've always figured it was a legacy of The Old 42nd Street, from the 1970's. That place ancient New Yorkers still talk about, from a time before Giulliani's Disnified Times Square took root in the 90's, but honestly who knows.
Somtimes I wonder, though, if that part of town would be different now, if Madison Square Garden weren't there, and the original, grander Penn Station had still remained. Throughout the 20th century many other parts of Manhattan endured similar transformations, and even Grand Central went through a period of neglect, but to compare Grand Central to Penn Station now, is like night and day.
I figure there's something to that.