Do you have an example of a time where Nate Silver has shown a misunderstanding of probability? Everything I've read by him has seemed informed.
This seems to be one of Taleb's (and others') criticisms of their predictions: that they don't consider the probability of major unexpected events occurring between now and the event (like an athlete getting injured in the future, or the FBI announcing an investigation of a political candidate). But as I understand it, 538 is explicitly not attempting to consider those things, and is simply publishing probabilities of outcomes if the event were to happen now.
I don't know of a good way to try and capture these sort of events meaningfully, do you?
Nate publishes predictions for events that will never occur pr(x wins | election today) when there is no election. It would be like me placing bets on a sports game that is not scheduled to be played.
> Why are they publishing probabilities for if the event were to happen today? The event isn't happening today!
My guess is that it's extremely out of scope (and probably ultimately infeasible) to try to incorporate these things into their models. Sure, there's a non-zero chance of a candidate being killed by a meteorite, but I think it's fair to ignore that and, if it happens, admit that you didn't even attempt to include the probability of that into your model.
The putative “average reader” who does so will see the default view, which is a forecast, not the nowcast, which you must actively shift to in order to see.
> and 538 does have some responsibility to actively explain things upfront in their UI,
Like, on the view selector that you have to use to choose the nowcast instead of the default forecast, where the nowcast is described as “Now-cast / Who would win the election if it were held today”.
I think they have that covered.
Exit polls are of course asking for voting preference today, because no person in their right mind would ever answer differently for "who would you vote for today?" versus "who will you vote for on November 4?" so it doesn't matter.
The problem is, we don't care about what would happen if the election were to be run today. We care about what will happen when the election actually occurs. What we want is a prediction that takes the polls into account but also factors in all the other information we have that could be used to try and predict the future election (e.g. trying to identify whether changes in a candidates polling numbers are likely to be temporary or reflect a permanent shift, identifying historical trends that can be used to predict future behavior, etc). I realize that predicting the future is much harder than predicting the hypothetical of what would happen today, but predicting the future is the whole point of forecasting.
There is value in being able to tease out the "accurate" voter sentiments today from the various polls and accounting for uncertainty and error, but that value is primarily in how we can then turn around and use that information to aid in future forecasting, and how candidates can use that data to gauge the effects of what they're doing. But without an attempt at forecasting the future, this is largely useless to the general population.
The only thing these people need is to learn what he's really doing, and then he will be immediately correct again! To me this seems like the simple and obvious solution to this debate.
538's forecasts, especially far from election day, are for the newspaper consumer. They provide insight into current political dynamics, and a quantitative look at how people view policy, the economy, where current fundraising is at, etc. in a single concise number that can serve as the basis for further discussion and analysis. They however aren't useful for the sportsbook consumer, because they don't account for the practically innumerable "unknown unknowns" that will occur between now and election day (scandals, terrorist attacks, fluctuations in the economy, flubs on the campaign trail, external activist efforts gaining traction). If they did account for all that uncertainty, then the predictions consistently be near 50-50 for the majority of the race, which not only is still uninteresting for the sportsbook types, but also means you no longer are providing value to the newspaper types.
Its important to point out that, over time, the predictions become applicable to both readers. As election day nears, the number of "unknown unknowns" decreases, and a prediction for "what would happen if an election were held today" and "what will happen on election day" converges. So up until the few months and weeks leading up to election day, the predictions should be taken more as commentary, and then perceived as serious indications of what will occur closer to election day.
"p.s. You continue to cite a paper in which you critique our now-cast (a deliberately volatile hypothetical of what would happen in an election held today) and mistake it for our forecast but because you're such a stubborn idiot you won't admit to this very basic mistake."
I don't think that's true. While 538 does such predictions (the “nowcast”), they also predict actual results (the “polls-plus forecast” and the “polls-only forecast”), the default view is a forecast (I think the polls-plus usually, but I vaguely recall a chance to polls-only as the default during the 2016 cycle with concerns that there were reasons to suspect that some of the other signals on the polls-plus model had become less useful.) The “nowcast” is not the the default/headline number, so it would be very odd for a someone to mistakenly fixate on it thinking it was the forward-looking prediction.
Previously, the reported predictions were the "nowcast", "polls-only", and "polls-plus". But despite their names, I believe those too, whatever 538 may have described them as, were "predictions for outcomes as if the event occurred today". In fact, were they not the case, then Taleb's argument would actually have merit, despite his obtuse attempts to explain it.
Essentially, his argument is that, if you are predicting a particular outcome to an event (say, Clinton winning the election) a year from now with a certainty of 90%, and 3 months from now, your prediction is 75%, and 3 months after that is 85%, and then right before the election, is 70%, is evidence of a flawed model (keep in mind, I'm making these numbers up).
If your prediction is 90% a year out, it should fluctuate much as time goes on, because 90% is very strong certainty. One may counter that as time goes on, public opinion may change, there are changes in news coverage, and external events like the economy or scandal can make a big impact, and all of that is absolutely true. But the point is that, if in light of new information coming forward, your model predictions repeatedly exhibit volatility, than your model failed to adequately account for the uncertainty as a result of future events, and you were never actually 90% sure in the first place. By necessity, a model predicting a distant event in which lots of unknown factors can cause it to fluctuate before it occurs will either be fairly stable as time goes on (meaning you indeed were correct that there was a high chance of a Clinton win), or should give much more hedged estimates of a win (meaning your probabilities a year out would be much closer to 50-50).
Taleb's point is that 538's models are too volatile to actually be accurate when they say there is an X% chance someone wins an election a year, or 6 months from now. However, his argument does not apply in the case where the prediction is that there is an X% chance someone wins an election, were the election held today. Instead he seems to dismiss that that type of prediction is worthless, which is simply wrong and foolish in my opinion. There is lots to be learned about the current political climate from those predictions.
So its entirely possible both parties are partially wrong and right. Taleb is correct that 538's models are too volatile to be accurate predictions far out from the election, but it assumes that the model is a prediction of what will happen on election day, rather than what would happen today. He also is throwing out the baby with the bathwater by insisting that that is the only kind of political prediction that has any value. On the other hand, 538 has not been very clear on if their forecasts are predictions of what happens on election day or what happens were election day today. If its the former, then their models have issues and Taleb has at least some validity to his argument. If its the latter, then Taleb's argument is no longer applicable.