Nassim Talebs case against Nate Silver is bad math

It's worth noting that you are completely wrong: not only do they keep such a record, they also provide a handy chart of it on the latest prediction page so you can easily see exactly how the prediction changed over time.

Assigning a probability to a one-off event is enumerating all the ways it could happen, all the ways it could not happen and assigning a probability to each of those ways. Obviously there is a lot of guesswork; but if you need to make a decision based on the future that approach gives you a much better chance of making a good decision. In practice an event will be made up of components that are more predictable than the whole, and some real uncertainties. There is a lot to be gained by thinking hard about the situation, and assigning a probability will do that.

Silly example - how do I estimate my risk of falling climbing up a set of stairs that I've never climbed before?

* Baseline risk of tripping - I have a lifetime of data.

* Increase of risk being on a staircase - I have an area I want to put my foot on (a specific step) that is about 1/3 of the area that my foot usually falls in, so that increases the risk by an amount that can be reasonably estimated.

* I will watch my foot - maybe an order of magnitude improvement in precision.

This is enough to let me estimate the risk of carrying a bulky object (that obscures my view of my feet) up a staircase. I've isolated the uncertainty (how much does visual observation change the odds) from the certainties (areas, background rate).

Now I can take that to several experts who will identify new mechanisms and tighten up my estimations on how big a deal the components are. In this way - even though the final % I come to would still be a bit arbitrary - it is starting to become a summary of what a large number of people think about the inputs to the problem and their relative magnitudes. Being able to communicate all that thinking with a single number is a miracle in its own way.

The data might favour one candidate, but even assuming it is unbiased and representative, it is only a random sample, and there is a chance it could be randomly wrong.

Kidding aside, your observation is correct, you need to perform repeated predictions. 538 does just that. They keep predicting a whole lot of outcomes, so you can check their track record. They even have a challenge for the audience, where you can record your own predictions and compare them with their corresponding predictions (see for example [1] for NFL games). The scoring of the predictions uses the Brier score [2], which is just a version of R-squared for classification problems.

In the case of the November 2016 election, other prediction sites were giving the Trump team a 1-2% chance, 538 was giving them a 20% change. Considering that, 538 comes out as the clear winner.

Which brings us to what I consider to be the best measure of prediction accuracy (my own invention, it doesn't have a scientific name yet): you can compare two sequences of predictions (let's say yours agains 538) by simulating bets at some mid-odds level. For example if I say Trump's probability of winning 2020 is 45% and you say it's 35%, then I'm willing to give you 2-to-3 odds, while you're willing to take that. We then see who stays ahead most of the time in this betting simulation.

For what is worth, here's a recent survey of different scoring types for predictions: "Assessing the performance of prediction models: a framework for traditional and novel measures" [3]

[1] https://projects.fivethirtyeight.com/nfl-predictions-game/ [2] https://en.wikipedia.org/wiki/Brier_score [3] https://www.ncbi.nlm.nih.gov/pubmed/20010215

But the betting odds depend on how much knowledge you have - for someone with perfect knowledge the odds may be 0 or 1, if you know nothing you might guess 50:50 for practical purposes.

So, for example, when 538 said that Trump had a 30% chance of winning, its hard to say how accurate that was given it was a one time event. But if you look over all of their election predictions, and look at those they predicted to win 30% of the time, about 30% of them should have been victories (with some margin for error, of course). If 50% were victories, or only 10%, then maybe that would indicate their models aren't doing so hot.

But complaining that a model is incorrect because it enables arbitrage is a standard complaint, it's how you can prove all non-Bayesian models are irrational. That doesn't mean all Bayesian ones automatically pass the test though.

Error bars and spreads are already outside the Bayesian paradigm, which demands a single number.

And this is precisely the complaint Taleb is making.

An arbitrage is just a strategy that's guaranteed to make money. If you know for sure that something will go up and down at different points you could make money risk free. Simply buy at any point below the current price and sell at any point above.

The nowcast does, but the nowcast isn't the main output, the two (polls-plus and polls-only) forecast models are.

Dear @aubreyclayton, I'm genuinely interested in seeing how you arrived at this conclusion. Would you kindly share the proof, or at least explain the logic behind it. Thank you.

Probably a lot more real value than a typical news outlet these days which pump out garbage at high speeds.

> Why are they publishing probabilities for if the event were to happen today? The event isn't happening today!

My guess is that it's extremely out of scope (and probably ultimately infeasible) to try to incorporate these things into their models. Sure, there's a non-zero chance of a candidate being killed by a meteorite, but I think it's fair to ignore that and, if it happens, admit that you didn't even attempt to include the probability of that into your model.

538's forecasts, especially far from election day, are for the newspaper consumer. They provide insight into current political dynamics, and a quantitative look at how people view policy, the economy, where current fundraising is at, etc. in a single concise number that can serve as the basis for further discussion and analysis. They however aren't useful for the sportsbook consumer, because they don't account for the practically innumerable "unknown unknowns" that will occur between now and election day (scandals, terrorist attacks, fluctuations in the economy, flubs on the campaign trail, external activist efforts gaining traction). If they did account for all that uncertainty, then the predictions consistently be near 50-50 for the majority of the race, which not only is still uninteresting for the sportsbook types, but also means you no longer are providing value to the newspaper types.

Its important to point out that, over time, the predictions become applicable to both readers. As election day nears, the number of "unknown unknowns" decreases, and a prediction for "what would happen if an election were held today" and "what will happen on election day" converges. So up until the few months and weeks leading up to election day, the predictions should be taken more as commentary, and then perceived as serious indications of what will occur closer to election day.

Previously, the reported predictions were the "nowcast", "polls-only", and "polls-plus". But despite their names, I believe those too, whatever 538 may have described them as, were "predictions for outcomes as if the event occurred today". In fact, were they not the case, then Taleb's argument would actually have merit, despite his obtuse attempts to explain it.

Essentially, his argument is that, if you are predicting a particular outcome to an event (say, Clinton winning the election) a year from now with a certainty of 90%, and 3 months from now, your prediction is 75%, and 3 months after that is 85%, and then right before the election, is 70%, is evidence of a flawed model (keep in mind, I'm making these numbers up).

If your prediction is 90% a year out, it should fluctuate much as time goes on, because 90% is very strong certainty. One may counter that as time goes on, public opinion may change, there are changes in news coverage, and external events like the economy or scandal can make a big impact, and all of that is absolutely true. But the point is that, if in light of new information coming forward, your model predictions repeatedly exhibit volatility, than your model failed to adequately account for the uncertainty as a result of future events, and you were never actually 90% sure in the first place. By necessity, a model predicting a distant event in which lots of unknown factors can cause it to fluctuate before it occurs will either be fairly stable as time goes on (meaning you indeed were correct that there was a high chance of a Clinton win), or should give much more hedged estimates of a win (meaning your probabilities a year out would be much closer to 50-50).

Taleb's point is that 538's models are too volatile to actually be accurate when they say there is an X% chance someone wins an election a year, or 6 months from now. However, his argument does not apply in the case where the prediction is that there is an X% chance someone wins an election, were the election held today. Instead he seems to dismiss that that type of prediction is worthless, which is simply wrong and foolish in my opinion. There is lots to be learned about the current political climate from those predictions.

So its entirely possible both parties are partially wrong and right. Taleb is correct that 538's models are too volatile to be accurate predictions far out from the election, but it assumes that the model is a prediction of what will happen on election day, rather than what would happen today. He also is throwing out the baby with the bathwater by insisting that that is the only kind of political prediction that has any value. On the other hand, 538 has not been very clear on if their forecasts are predictions of what happens on election day or what happens were election day today. If its the former, then their models have issues and Taleb has at least some validity to his argument. If its the latter, then Taleb's argument is no longer applicable.

It makes more sense philosophically to make claims about one's model than about the real world, anyway.

[0] https://en.wikipedia.org/wiki/Arcsine_distribution

To me, a 20% probability is not "extremely likely", but it is "extremely possible". A 10% probability is "quite possible" or "not terribly surprising". An 80% probability is "very likely", but perhaps not "extremely likely".