The weather is a drawback (in probably 200+ days in NYC over 10 years I can remember a single week of what I would consider nice weather). I have realized that crappy climate is a poor reason to avoid the move.
My one main concern is finding a job with similar pay and interesting work. 90+% of the job listings I see are for Java enterpise-type work paying a third of what my (admittedly more than comfortable) current total pay is.