How Can We Tell Which Election Forecasts Are True?

Yes, my hope was to point out the absurdity of this situation for an event that only happens once.

It would be nice if all polls calculated a CI or probability distribution of their results (i know 538 does). But even 538 doesn't display their priors.

But this would confuse the public even more.. e... asking them to interpret a 95% chance to win with low confidence

The distribution for the roll of a die doesn't change, but the individual outcomes do.

It all boils down to how this random variable is defined.

> if the election were repeated an infinity number of times, wouldn't the same candidate win each time?

It depends on what you mean by "repeated an infinite number of times".

First, there's a problem with the framing of that statement, because you can't literally repeat something an infinite number of times. You can repeat something indefinitely, and observe the behavior as the number of repetitions increases without bound, but that's not quite the same thing. (I realize this sounds pedantic, but if you want to understand the underlying interpretation of probability, it's an important distinction to be aware of).

So, let's say you asked, "if the election were repeated over and over again, without bound, what would the outcomes be?" And the answer to that is... well, it depends on who you ask. Setting aside the "fate vs. free will" philosophical question, different statisticians would answer that question differently, depending on which interpretation of probability they use. There are three main interpretations of probability, though these days, people pretty much only ever use two (frequentist and Bayesian). And you could construct either a frequentist or a Bayesian argument that the same candidate would always win, or that a different candidate could win each time.

Think of the election as a function - some unknown inputs go in, and an outcome goes out. What are the inputs to that function? (That's a rhetorical question - we're not really sure what makes every individual voter vote, and what disruptions could happen to nudge each person to vote a different way, or to not make it to the polls that day, or for their ballot to get lost in the mail, etc.). Congratulations, you have a random variable[0]! Now all you have to do is decompose it to determine the underlying components and the true distribution they draw from[1].

I realize that's not a very satisfying answer, but the underlying issue ("what does probability mean") is not a simple question, and statisticians have been debating it for over a century.

[0] which is, confusingly, neither random nor a variable.

[1] Which is itself controversial way of putting it, because some statisticians would argue that there is no "true" distribution, or that, if there is, we can never actually know it.

A (very) slightly more practical way to look at: what happens over a hundred elections where Nate Silver reports a 80% lead for one candidate three weeks out? If that candidate wins about 80 times then the model is good. What you want is infinite (or many) actual historical elections and then just look at the ones where the model gives 80%. A good weekend project if you have a universe simulator lying around.

Edit: this is what "under a stationary distribution" means. It means holding everything that goes into the model, such as the polls, constant, while allowing everything else to vary, then you run the experiment many times. The easy way to do that is just let Nate Silver keep doing his thing for a few centuries.

it's all a distraction to give the illusion of offering participation. people can go home and feel "i contributed" having fulfilled their civic duty. everyone paying any amount of attention knows its a rigged game ten ways to sunday.

democracy in general is a feedback loop between the press and politicians with the voting population as pawns/fodder. journalists and politicians have one shared concern: staying relevant and needed.

That is what was thinking. But not really. The most difficult part is to assign probabilities to the individual state races. The final probability is done using a bootstrapping technique(a simulation). So you really have 50 predictions that you can evaluate after the election.

Let's say if some forecaster says a candidate has a 60% of winning in each state, and in fact he wins all of them. Does that means that the forecaster was 40% wrong?

"So if Romney had won Florida, Silver would have been wrong, but only a little wrong."

No, Silver would not have been wrong at all, or at least we'll never know. He doesn't predict a winner; he gives probabilities of victory. It would have been perfectly consistent with Obama having a 50.3% chance of winning (or 63.5% or 32.8% or 99.9% or whatever probability you want to assign) and then for Romney to have won. Even after the election was over, a Florida victory for Romney would not have meant that Silver's pre-election probability assignment of 50.3% to Obama was wrong. That's not how probabilities work.

Think of sporting events. A huge underdog with only, say, a 2% chance of winning may beat the favorite. This does not mean that it had better than a 2% chance of winning before the event. Depending on the reason for the underdog's victory, it may not even mean that the underdog has better than a 2% chance of winning if they play again. One in a million events happen, and it doesn't mean that their chances of happening were greater than one in a million.

Right, but a lot of other poll aggregators who don't have that baggage are also predicting a Clinton win, many with higher confidence than Nate Silver.

Just read the posts and comments there - lots of CT + right-leaning discussion that seems curiously aligned with alt-right ideology.

ie, more heat, less light.