Hacker News Reader

>The appropriate amount of precision to conclude with, given all of the fuzzy inputs, is something like 'well, we can't tell who's going to win - it's close, although this one candidate's chances look slightly better'. It's not 'Candidate A's chances improved from 74.6% yesterday to 76.5% today'.

Your post shows a horrible misunderstanding of how this all works. When you have sketchy data, you don't just disregard it -- it's still telling you something and the role of statistics here is to extract, from amidst all the noise, that information.

Just because something is hard doesn't mean it's not worth the attempt.

>His results are also completely unverifiable.

Kind of. But how his methodology performs over all, over a wide range of elections, is verifiable. Wikipeida mentions this:

>The accuracy of his November 2008 presidential election predictions—he correctly predicted the winner of 49 of the 50 states—won Silver further attention and commendation. The only state he missed was Indiana, which went for Barack Obama by 1%. He also correctly predicted the winner of all 35 Senate races that year.

Whether his models perform like this consistently is a very real way that the methodology can be verified.

A couple of points:

1) His model is easy to verify... wait for the elections to be over, and see how much his predictions correlate with reality. He predicts a great many races, and the results speak for themselves.

"The accuracy of his November 2008 presidential election predictions—he correctly predicted the winner of 49 of the 50 states—won Silver further attention and commendation. The only state he missed was Indiana, which went for Barack Obama by 1%. He also correctly predicted the winner of all 35 Senate races that year."

http://en.wikipedia.org/wiki/Nate_Silver

2) Also, his model accounts for the response rate figures you cite - that was actually the point of his most recent blog post. The reason Nate gives Romney any chance at all to win is because his model predicts that there is less than 1/5 chance that the polls are systematically biased (based on data since 1968). He thinks that if he's wrong, it's exactly for the reason you say - that those who respond to the polls are a small group that don't represent the population. He has a large enough sample size to eliminate sampling error, and we're close enough to election day to discount error due to polls being a snapshot in time. So there is only systematic bias left to discuss...

"So why, then, do we have Mr. Obama as “only” an 83.7 percent favorite to win the Electoral College, and not close to 100 percent? This is because of the other potential sources of error in polling."

And that error is simply that polls don't reflect reality, and he thinks that is about a 15% chance.

http://fivethirtyeight.blogs.nytimes.com/2012/11/03/nov-2-fo...

His model is easy to verify... wait for the elections to be over, and see how much his predictions correlate with reality.

FWIW, he got a bunch of congressional races wrong in 2010.

What you just posted is misleading. Silver's model gave the GOP a 2 in 3 chance of winning the House in 2010, predicting a net gain of 45-50 House seats, and the majority - http://fivethirtyeight.blogs.nytimes.com/2010/09/10/g-o-p-ha...

Also, his modeling is MUCH more accurate for presidential races, where there is more polling data, and thus a lower sampling error.

He's not the Oracle at Delphi; he gives probability estimates, not inescapable prophecy. It's inevitable that some elections will go to the candidate who he predicts has a lower chance of winning. The real question is, how often? If it happens more or less often than he predicts, then his predictions are biased and should update on that fact.

One factor that messes with any model is lack of consistent data. House races aren't publicly polled nearly as often as statewide polls. And furthermore, it's not necessarily that he "got races wrong." When he says Obama wins 80% of the time, he's also saying that Romney wins 20% of the time. 1 out of 5 times, Romney will win and Obama will lose, despite being a favorite to win.

It is important to note that the cited accuracy was for the predictions made the day before (or perhaps even later) the election. It would be interesting to know how accurate the predictions were X weeks out.

I don't think Silver claims to have data or knowledge that is better than what is available to the public.

What he does different - and his isn't the only model built off of polling data, nor are 538's probabilities that different than other models - is just trying to decipher all the available polling data to reduce the noise and focus on the signal.

I also don't think he sees himself as being in the business of picking a winner, as in saying "I believe X will win tomorrow".

One thing you might be missing is that his predictions are verifiable at the state level - a result where he predicted the overall winner but only got 30 of 50 states right versus getting 49 out of 50 states right are pretty different.

Polling aggregation and models are as much "science" as the practice of polling is in the first place.

One thing you might be missing is that his predictions are verifiable at the state level - a result where he predicted the overall winner but only got 30 of 50 states right versus getting 49 out of 50 states right are pretty different.

This really isn't true, because something like 35-40 of the states aren't even remotely up for grabs and anyone intelligent could predict them after 15 mins of looking at recent polls and previous election results. So he's trying to pick winners for 10-15 states where the race is closer, but for probably half of those, it's really not that close and you'd have a pretty good chance of getting them right if you just eyeballed it. So now we're talking about 5-7 really tight swing states, and you're basically saying that predicting 44/50 states is very different than predicting 49/50. Not so sure that it is, especially given that it was a 50/50 choice for each of those states, and he did it for one election. Also, lots of pundits and bloggers made predictions in 2008; you'd expect someone to be mostly right. Fooled by Randomness and all that.

For what it's worth, I think Silver probably is right and Obama will win (though I'd prefer he didn't), but his methods do lean towards being relatively unverifiable on a short timeline.

but his methods do lean towards being relatively unverifiable on a short timeline.

Perhaps. But I think a lot of his critics fall into this trap of thinking that what Silver thinks is that he is offering some sort of revolutionary and perfect prediction, or that he thinks he is doing some sort of groundbreaking science.

He's just trying to make a prediction based on all of the available information. If election predictions are inherently "relatively unverifiable", Nate Silver isn't capable of changing that - just working with what he is given (and doing a great job of explaining all of the ins and outs of this stuff to the layman).

In fact he says a good amount of this himself. I think some of his readers impute a greater level of "scientificness" to his numbers than he himself claims. He's had many posts throughout the fall explaining where his model is based on some assumptions that could turn out to be incorrect, and key parameters fit based on relatively limited data. For example, an important one is how you translate current poll leads to likelihood of winning on election day, i.e. why does an x% lead a week before the election give you y% chance of winning? His method is to look at the empirical distribution of poll misses in the 11 elections 1968-2008, make some normality assumptions, and use that to estimate poll->results mapping, which serves as a single estimate of a whole bunch of miscellaneous sources of error (likelihood the polls are systematically biased this year, likelihood of a last-minute change, etc.). But of course that's a small number of data points, and not IID ones, either, all of which he acknowledges. All he really claims is that this model is a reasonable attempt to integrate the available data.

The headline predictions aren't his only predictions.

Silver also publishes projected margins of victory (with margins of error), and he does so for every state and every senate race, not just the battleground ones. As such, it's possible to do a fairly substantial review of his model against actual results.

Statistics are never perfect, but his predictions are far more transparent and verifiable than anything else on the market.

It seems like you are confusing predictions about this election with overall model accuracy.

The state by state predictions are the primary method by which you can judge the accuracy of his model. If his model forecasts a state as 60% for one candidate, you can assess that level of accuracy by looking at all 60% predictions.

Let's take some examples from todays date on FiveThirtyEight.

* Florida 54.8% chance of Romney win

* Virginia 67.0% chance of Obama win

* Nevada 67.9% chance of Obama win

* North Carolina 79.6% chance of Romney win

* New Hampshire 80.4% chance of Obama win

* Iowa 80.7% chance of Obama win

* Nevada 88.7% chance of Obama win

Other states like Texas, Utah, Idaho, Wyoming are projected at 100% for Romney and New York, California, Oregon, and Illinois at 100% for Obama.

If any of the 100% states go to the opposite candidate, that is a model problem. If some of the ones specified with extremely high percentages 95% go against his predictions with a high margin of victory, again that is a model problem.

Finally, some of those close races should go against the models prediction.

Let's take the 7 states listed above. 5 of the 7 are projected for Obama and 2 for Romney. However there is only about a 37.68% chance of that exact distribution happening. I break it down as follows:

Obama-Romney * 0-7 0.02%

* 1-7 0.14%

* 2-5 2.26%

* 3-4 12.14%

* 4-3 31.48%

* 5-2 37.68%

* 6-1 16.00%

* 7-0 0.28%

Each of these numbers are probabilistic statements about the likelihood of the overall event occurring based on the probabilities. They are from a single simulation of the 7 state probabilities run 10,000 times. Each of them has it's own distribution, eg 5-2 was 37.68% in the first run, 37.38% in the next, then 38.03%, then 36.85%, etc.

This is a very simple model prediction, but by taking all 50 states into account you can get a very clear assessment of how well his model is actually predicting the outcome of the election. Complicating matters is the time series nature of the predictions.

This type of model is precisely the way you get away from "Fooled by Randomness and all that". His model is clearly articulating the amount of uncertainty in the forecast.

Electoral-Vote.com used a much simpler method (averaging polls) and got 48 out of 50 states right.

EV.com had Missouri as a tie (McCain ultimately won by 0.1%, 538 got the state right) and had Indiana wrong (as did 538).

The guy made a model that he (and the nytimes) think is reasonable. The model takes inputs and spits out a probability. As the inputs change slightly so does the output. He's well within his right to publish these changes. What's the problem? That you just can't accept the validity of day-to-day basis-point changes in 'chances of something happening' when the inputs might be dirty? Can't you just say 'oh, that's nice' and take it as a directional signal without allowing it to irritate you?

I'll take a thoughtful statistical analysis any day over the words of a single television journalist giving me second-hand information from a 'source within the campaign' especially when any campaign insider has an absurdly strong incentive to manufacture public perception in his candidates favor.

>His results are also completely unverifiable. Let's say his final prediction is 80-20 in favor of one candidate, but the other candidate wins. Well, his model did say that'll happen one time in five, so you can't really criticize it.

That's a good point. He is using percentage chances everywhere but then running only one trial (election).

Clearly a wager is the way to solve this problem (Bayesian interpretation of probability). Is it legal to gamble on an election result? They could use the following payouts - J represents a regular journalist using 50/50 odds, N represents Nate Silver using 75/25 Obama/Romney:

Romney win:

J: +1000

N: -1000

Obama win:

J: -600

N: +600

With those payouts each side has an expected value of +200 dollars for this wager, using the specified odds they believe to be true, so both should be willing to take it.

Imtrade and trade sports do this. It is also trivial to simulate with play money over many elections and races.

This is, from my recollection, an incorrect assessment of Silver’s methodology.

He runs (continuous?) Monte Carlo simulations over his inputs, and when he says that there's an 80-20 chance in favour of candidate O, he's saying that 80% of the simulations resulted in a win for candidate O, not candidate R.

You're forgetting that he predicts the specific numbers of the outcome, not just the result. When he predicts that candidate A will win with X electoral votes, and the result is very close to X, that's a substantial verification of his prediction, especially if it repeats several elections in a row.