By doing the experiment and letting the election happen.
Experiment is the arbiter of truth.
Experiment is the arbiter of truth.
"The reason for this is that pundits (the better ones, anyway) don’t just predict election outcomes, they also tell you how confident they are in each of their predictions. For example, Silver gave Obama a 50.3% chance of winning in Florida. That’s pretty damn close to 50/50 or “even odds.” So if Romney had won Florida, Silver would have been wrong, but only a little wrong. In contrast, Silver’s forecast was 92% confident that Rick Berg would win a Senate seat in North Dakota, but Berg lost. For that prediction, Silver was a lot wrong. Still, predictions with 92% confidence should be wrong 8% of the time (otherwise, that 92% confidence is underconfident), and Silver made a lot of correct predictions. So how can we tell who did best? We need a method that accounts not just for the predicted outcomes, but also for the confidence of each prediction. There are many ways you can score predictions to reward accuracy and punish arrogance, but most of them are gameable, for example by overstating one’s true beliefs. The methods which aren’t cheatable — where you score best if you are honest — are all called “proper scoring rules.” One of the most common proper scoring rules is the Brier score. A Brier score is simply a number between 0 and 1, and as with golf, a lower score is better."
http://rationality.org/2012/11/09/was-nate-silver-the-most-a...
"Also note that Wang & Ferguson got a better Brier score than Silver despite getting Florida (barely) wrong while Silver got Florida right. The Atlantic Wire gave Wang & Ferguson only a “Silver Star” for this reason, but our more detailed analysis shows that Wang & Ferguson probably should have gotten a “Gold Star.”"
No, Silver would not have been wrong at all, or at least we'll never know. He doesn't predict a winner; he gives probabilities of victory. It would have been perfectly consistent with Obama having a 50.3% chance of winning (or 63.5% or 32.8% or 99.9% or whatever probability you want to assign) and then for Romney to have won. Even after the election was over, a Florida victory for Romney would not have meant that Silver's pre-election probability assignment of 50.3% to Obama was wrong. That's not how probabilities work.
Think of sporting events. A huge underdog with only, say, a 2% chance of winning may beat the favorite. This does not mean that it had better than a 2% chance of winning before the event. Depending on the reason for the underdog's victory, it may not even mean that the underdog has better than a 2% chance of winning if they play again. One in a million events happen, and it doesn't mean that their chances of happening were greater than one in a million.
This is consistent with Silver's own explanations of what he's doing, and (I'm pretty sure ) he is careful to say he doesn't make "predictions". (Actually, not sure, it may be merely that he often clarifies that a different result than predicted does not make the probability assignment that prompted the prediction "wrong". The prediction and the probability analysis it's based on are different things.) I'm not saying that the Brier Score can't differentiate between better and worse probability assignments, but I am saying it's confusing the issue if it calls probability assignments "right" or "wrong". They may be "better" or "worse" than each other, but not "right" or "wrong". This may be using language in more technical manner, but it seems to me (and to Silver) to be an important distinction.
I think being concerned about this authors usage of the words "right and wrong" misses the point. The media will say "you were wrong" if you were on the wrong side of a 51/49% prediction, even if you shouldn't be penalized for it. Wang & Ferguson got dinged by The Atlantic Wire for being wrong, when they shouldnt have. The last quote in my post, supports your point. The person who is least wrong is the person with the lowest aggregate Brier score. The author was using right and wrong, because the articles they were referencing used those terms. The quote was written with the colloquial language of what it addressed.
The probability analysis that the prediction is based on does admit of degrees; one probability assignment may be better or worse than another. But saying a prediction is "only a little wrong" implicitly refers to the probability assignment it was based on; it confuses the concepts of "prediction" and "probability". The probability assignment was not necessarily "wrong" to assign a higher probability to the event that didn't occur. It may have been a bad probability assignment, but discerning that would involve considering factors other than the outcome of the event.
https://terrytao.wordpress.com/2016/06/01/how-to-assign-part...