Why the snow forecast for New York City was so bad

More to the point, who cares? The storm was still pretty bad: this isn't a case of a "bust" forecast. Long Island got hit, as well as much of New England.

So the line on the map of where the heaviest snow would be was a bit off: the only reason this is "news" at all is because that line happened to affect New York City, and there are a lot of journalists who live there whose view of the world matches that New Yorker cover[1]. If the line had been anywhere else we wouldn't have heard about it at all.

[1] https://en.wikipedia.org/wiki/View_of_the_World_from_9th_Ave...

While this article mentions that the GFS model was more accurate than the others (ECMWF & NAM), it neglects to give the backstory as to why it was more accurate and less trusted. For years the GFS has been made fun of because of its poor performance when compared to the Euro - and it just got the major upgrades it needed this month! New supercomputers, new high-res models, all went online just before the storm arrived. So, very few trusted the model even though it was in the best spot to get everything right.

In an unexpected move, The Weather Channel was much more accurate than the US NWS / AccuWeather / etc because of their use of the GFS as a trusted model. More info in this article: http://www.alternet.org/environment/why-almost-everyone-got-...

Edit: And of course, it's my opinion that all the models are currently suffering from a lack of input data. The "...and what should be done" part of the headline, in my opinion, should be answered with, "everyone download PressureNet and contribute your phone's sensor data to weather models", so that we can better predict difficult quantities like heavy snowfall and severe thunderstorms: https://play.google.com/store/apps/details?id=ca.cumulonimbu... ;)

Alternative hypothesis that also fits the evidence: You look bad if reality turns out worse than your forecast. You look less bad if reality turns out better than your forecast, thus there is a built in incentive to make forecasts more pessimistic than is justified.

Imagine if 10in were forecast and then NYC actually got 2 feet. Then everyone would be complaining that the city wasn't adequately prepared for the 14 extra inches they received and would blame the forecasters for all the economic consequences of not being prepared.

After watching the same ridiculous news reporting touting "historic", "storm of the century", "unprecedented", etc., year after year, the more cynical side of me suggests that there's a small supplemental reason besides flawed models: there's probably good money to be made from overhyping storms. At the very least news stations must get great ratings the day or two beforehand. Note that this doesn't preclude accurate estimating...you just have to pull a bunch of cheap psychological tricks to get people to focus on the worst case scenario. The end result of people freaking out is still largely the same.

One reason this comes to mind is that I can't ever remember a storm reaching its worst-case snowfall predictions, and usually they barely make the minimums. A rule of thumb I use is to treat the minimum prediction as the actual maximum and zero as the actual minimum. Anecdotal, but seems to work well.

Several years ago in Australia, a blogger called Possum Comitatus, aka Scott Steel, used to mock journalists for breathlessly reporting every twitch of the polls, even though most such movements were within the margin of error.

So successful was this mockery that the way polls are reported actually changed. It become normal to report a change in polls with a remark about whether the change was within the margin. Political race-calling has moved onto less abuses of statistics.

Better reporting can be done, and done with surprising ease, using a simple tool.

Mockery of journalists.

To gain status and prestige over their peers, some will adopt the better method. In a few months they'll all do it.

Three problems:

1. Uncertainty isn't actionable. A statement like, "There's an 85% chance of snow with accumulation of 1–24" with peak probabilities of 28% and 17% at 2" and 18", respectively" is nonsense to most people. And moreover, it doesn't mean anything to me. Are we going to get snow that will affect my morning commute? Are we going to get so much snow that I need to take emergency action? A probability distribution doesn't answer those questions, even if that's the best that exists.

2. Weather is news. The cable news channels were dominated by snow predictions yesterday, and the Weather Channel exists as an "entertainment" venue, not a source of scientific information. If it doesn't fit into a soundbite, the public can't absorb it. And "Weather is hard to predict" doesn't garner viewership.

3. Prepare for the worst. While small changes in the environment can lead to large variability in snow accumulation, the fact of the matter is that there was a not-unlikely chance (according to the models) that NYC, Boston, and other cities could have seen crippling snowfall. They were able to prepare for that eventuality. That it didn't come to pass is almost a non-issue. Can you imagine the outcry—not to mention the impact—if the situation was reversed (i.e., surprise 3' of snow)? Fairfax County Public Schools outside of DC got slammed in the media for not closing a few weeks back.

The real question is: at what point of probability do you prepare for a potential outcome, and are you prepared to back that up (either way)? Do we prepare for record snowfall on 80% likelihood? 50%? 10%? (And as a broader point of discussion, this same calculation applies to other areas: TSA spend vs. terrorism; flood insurance cost vs. coverage; etc.)

People don't like it when 14 inches is forecast and the actual snowfall is 2 inches, but they aren't going to like hearing forecasts of 2-16 inches.

That sort of wide confidence interval is more accurate, but it's also basically useless. How does a 2-16 inch forecast help a parent or school district or mayor plan for anything?

We should always strive to improve our predictions, but I see a fundamental dichotomy: do you prefer false-positives, or false-negatives? We don't get to pick no wrong results.

When it comes to predictions like snow storms, hurricanes and tornadoes, you obviously want to favor false-positives. It's better to predict something bad will happen, pay the preparation costs and have nothing happen, than to have something bad happen when you're unprepared.

So, yes, we should try to improve our predictions. And perhaps we should be better at communicating our uncertainty. But I don't blame the meteorologists, or the people in government who shut things down and told everyone to prepare.

I was delighted to notice a while ago that the Finnish Meteorological Institute had added 50% and 80% confidence intervals to their 10-day online forecasts, for instance: http://en.ilmatieteenlaitos.fi/weather/turku?forecast=long

Note that this article is written by the Capital Weather Gang, a group of DC-area meteorologists who consistently put out accurate and nuanced short-range forecasts of upcoming weather in the area. They give out great info and they aren't afraid to state their uncertainties up front, nor are they afraid to admit it when they're wrong (which is not too frequent).

They give an implicit answer to the question of "how could everybody do better?" which is "be more like us." And I think that would be great.

I was curious about what kind of bump weather sites get in ad revenue as a result of storm hype.

That's an interesting angle when wondering where the hype is driven from ... if you find the mental image of Murdoch twirling his mustache (this is my imagination, so why not add a mustache?) while making quiet phone calls from a back room a correct version of reality.

How is it possible that millions of New Yorkers accurately discounted the probability of it being a 500-year storm and yet the mayor didn't? The responsibility must lie with either the Mayor or the meteorologists. And there is something to own up to -- shutting down the city is not without real cost to real people. The mayor and governor will stay it was for our own good and better safe than sorry. But put a few million people under house arrest (no using roads -- under threat of arrest, no public transport) for basically no reason.

But a few things to consider: 1) The meteorologists with the most dire predictions are the one who are going to be picked up by the media -- the media needs to sell ads. 2) The bulk of the population accurately discounted the most dire predictions, because of well-calibrated but subconscious Bayesian calculation. 3) Bureaucrats, and to some extent politicians of both parties, have a well-intentioned, but costly and wrong, need to subvert the judgement of millions and replace it with that of a handful of experts. When the judgement proves wrong, they rest on their good intentions without assuming responsibility. 4) In subverting these judgements bureaucrats and politicians can and do rely on the states monopoly on force.

It seems like the costs are asymmetric on misses. If a bunch of people die, the mayor gets voted out of office. If they prematurely close the city for a day, the mayor just apologizes afterwards.

We are in the year of 2015, and yet we still cannot accurately predict the weather...