>In that sense, the best tip-off that the forecasts are different is they have Biden at 97% to win the popular vote, and have sometimes been as high as 98% (and 99% before they revised their model) whereas we are at 82%. You're getting up to 97-98%, you're getting VERY confident.
Most of the linked thread isn't about his own work or the Economist model, but about the false description of their forecasts as being pretty close, made as part of the (true) description that fairly small changes in intermediate results in the Economist model would lead to the same bottom line forecast as Silver's model.
Or I guess another way to put it is, given the highly polarised two party system in the US, elections tend to be fairly balanced and small differences have an inflated impact on the results.
But whether you see the difference as small (1% difference in popular vote) or large (3x difference in chance of victory), the models are different, and one of them must be better, although it’s probably impossible to tell which it is!
Comparing models across multiple elections and calculating the Bayesian regret is one way to do it. The models get tweaked each election so this isn’t exact, but it could give a sense of the skill of each forecaster.
Does anyone have links for previous election forecasts from Morris and/or Gelman?
As long as the same inputs aren’t fundamentally unavailable (and even then if the model has systematic handling of missing data, though the validity of that comparison is less clear) you can run the tweaked model on past elections (the main problem there is since those are probably the data used to generate them, it rewards overfitting; you might do better by running them against past elections with differing sets of random dropout of input data points [individual instances of particular polls, etc.] to mitigate that.)
Silver made the same point about probability before the 2016 election, and got into a similar Twitter argument (https://www.politico.com/story/2016/11/nate-silver-huffingto...). One guess on who proved to be correct.
This isn't actually true. The probability of winning is based on aggregation of regional probabilities. The same national vote probability distribution can lead to very, very, different election probability distributions due to regional variation. The electoral college essentially guarantees that the national probability distribution is worthless for actually predicting who will be President.
If candidate B then wins, it does not mean that our analysis was "proved to be [in]correct". By itself, it doesn't actually say anything about the quality of our analysis. After all, we explicitly pointed out that this was a possibility, and it would be strange to argue "your analysis said this might happen, and then it did, so your analysis was incorrect". There's just not enough information to draw any conclusions.
How about 100 times? 10 times? 2 times?
At what point does evidence cease to 'say anything about the quality of our analysis'? The answer is never. Every datapoint can be used to update your priors according to bayesian statistics.
Especially for someone who titles them-self "Chief Scientist"
Aren't regional probabilities just conditions on the national distribution? Building the national Presidential election distribution from the conditioned distributions versus just using the sampled national distribution is bayesian statistics.
>The probability of winning is based on the popular support for a candidate
This was the argument you made. And I don't see how this is Bayesian vs frequency when all I am saying is that the same national popular vote distribution can have large variance in regional conditional distributions which leads to large variance in election outcome distribution due to the electoral college.
Sounds pretty bayesian to me.
But in 538's case they use the similar methods to forecast many individual contests like presidential primaries, presidential elections, senate elections, house elections, governor elections, club soccer, college football, March Madness men and women, MLB-NBA-NFL games and playoffs, etc.
Multiple-year track records over many events so you can compare their forecasts to actual results over time.
Get the data and compute your own error rate here: https://github.com/fivethirtyeight/checking-our-work-data