Comment by pizza | Hacker News Reader

Comment by pizza | Hacker News Reader

Parent Full thread

If 538 is trying to do aleatory probabilities, then it is kinda like a broken clock - right sometimes, but as a tool for answers about future events of any complexity (i.e. the real world, where actions have consequences, and errors can be grave..), I'd avoid putting one's confidence in something that claims it has a simple/'linear' relationship to the real future.

It is also a cop out to claim "everybody who thinks probabilities based on initial poll responses are not representative of eventual votes" must clearly not understand probability. Maybe I am straw-manning or just don't understand the actual disagreement.

As a poll aggregator? I'm sure it's probably a fabulous tool for past, historical, observed data. The brand itself, probably even more valuable, as a symbol with some sort of social authority in certain circles, I guess.

Anybody who reads my comment history will see I have obviously become a Taleb stan, but his approach seems right to me: "show me the money". Which in his case means mathematical papers of proofs primarily, actual money, second.

Taleb is just mixing up the nowcast and the forecast, even after being corrected many times.

I think he would say that the "nowcast" is so useless and deceptive to be malpractice, since everyone will interpret it as a forecast, and even if they didn't it's not in any way useful. Which I agree with (unsurprising since I've semi-invented it as an opinion for another person).

It's only useless and deceptive if you willfully do absolutely nothing to understand what it is, or how it differs from the other models 538 has, which apparently is the case for Taleb.

Or in other words, you're 99% of people who will ever hear the number

It explains on the website what it is. And if you are going to rail against something, you should understand what you are railing against.

Silver has said before, at least in the podcast, that everyone at FiveThirtyEight hates the nowcast and its various incarnations but I think there's some pressure from the business side of things (for views, likes, shares, etc) to have something volatile for people to eat see day in and out.

I mean obviously its hard to make predictions of complex things as the data from every angle is difficult to get and sometimes incorrect. I used to love Taleb but he is more of a broken cloc, many say if he stopped writing after black swan he would have gone down as great but some of his other points and politics have undermined him.

I think Silver is actually pretty legit. He was over-hyped, then he was largely panned as a chrlatan but through and through he has been pretty consistent. The Obama elections he rose to prominance but he also did well in the most recent election. The absolute only news outlet saying that Trump had a statistical chance of winning. He hovered around 20-33% at peak while Huffington post told him he was an idiot. Many major outlets didn’t personally attack him obviously but had Trump in low single digits throughout the last weeks

Taleb actually makes a great point here and published a pretty cool paper about it. The point is that if a probability of the event changes too much, you can arbitrage it. E.g. assume these are payoff odds and you can sell your position before the event. He then found some nice no-arbitrage conditions that "real" probabilities must meet and showed that Nate's predictive timeline allowed arbitrage. Unfortunately Nate never responded to this claim with anything other than "math is hard" and other nonsense.

You can read the paper here:

https://arxiv.org/pdf/1703.06351.pdf

I don’t have the background knowledge to properly understand the math in that paper. But I do know that you can’t simply set bounds a priori on how much a probability can change after additional information has been gathered; no matter how pathological the swings, you can design a system, and a series of observations of that system, that would make all the forecasts (i.e. conditional probabilities) correct. Thus the paper must be making additional assumptions. As far as I can tell, those include at least that the electoral process can be modeled as Brownian motion, which is a martingale, but is not the only type of martingale; in particular, I’d expect random motion to be a good model of typical polling drift, but a poor model of sudden polling swings caused by news events (which in reality are a large change caused by a single random event, not the sum of a series of small changes caused by independent events that just happen to mostly point in the same direction). I am not sure whether the assumptions also include an estimated value of `s` or volatility; the paper doesn’t seem to explain how it’s calculated, but maybe it can be derived from the raw polling results plus the assumption of Brownian motion? In any case, the paper says nothing explicitly about what data was used to produce the “rigorous updating” graph. I’d love if someone could explain this to me…

Subjectively, it’s hard for me to believe that an unbiased forecast would truly be so utterly noncommittal until just before Election Day, or indeed that there’s enough data to answer that question, especially seemingly from just one election result. But my subjective impressions, of course, could be utterly wrong. I’m very curious whether or not this is the case.

I believe he's making a slightly more subtle point. It's not just that the forecasts are swinging too much, it's that they're swinging too much too early.

Consider an option on a stock (which is the analogy Taleb is making here). If you buy a 1 year call option on AAPL and tomorrow they announce that they beat earnings by 10%, that's not a huge deal for you. If on the other hand, you owned a 1-week expiration call, it is a big deal for you. That is, your prediction should be less sensitive to changes in environment the further out it is.

I believe he is somehow formalizing this statement, and then showing that Nate Silver's forecasts violate it, but I too don't fully understand his formalism.

I think that Taleb's problem is that he thinks all the uncertainty in Silver's model comes from the fact that people might change their minds about who to vote for.

This is why Taleb can't make a model that fits Silver's forecasts. Silver could only be confident early on if he knew that people weren't going to change their minds much. But if people don't change their minds much then Silver's forecast shouldn't fluctuate much as time passes. Alternatively, if the forecast fluctuates a lot, it must be because lots of people are changing their minds. But then Silver shouldn't have been so confident to begin with!

But in fact the uncertainty in Silver's model isn't (wholy) caused by the possibility that people change their minds. It's mostly caused by the possibility of polling error. As we approach the election people have less time to change their minds, but the possibility of polling error doesn't change. This why Taleb can't create a model under which Silver's forecasts are rational; he's not taking into account polling error.

No, it's mostly caused by people changing their minds on who to vote for (or whether to vote). That's why the estimates move around a lot when there is news affecting the election, like the given example of Comey re-opening the Clinton investigation.

There's still also a lot of uncertainty caused by polling error. You can tell by comparing the final polls to the actual results.

Those things may be true...but how does that relate to the issue Taleb is highlighting? That the forecast is overly volatile with respect to arbitrage pricing theory.

That would be only true if the thing you are forecasting becomes more sensitive to changes in the environment as time passes. It is true in your option example but not entirely true for elections.

Why is it not true for elections?

I would say most people make up their mind more and more with information release over time. Towards the end you know the candidates very well and its hard to change your mind. How many Trump supporters will change their mind even now?

Sam Wang of the Princeton Election Consortium did commit to a prediction, in August 2016, that the 2016 would have some of the lowest polling variability on record. (I think that he declared something like 2.6% uncertainty.)

http://election.princeton.edu/2016/08/21/sharpening-the-fore...

And I think that he was very wrong on that count.

Is it a great point?

He seems to miss the point of what 538 is doing. Their nowcasts are attempts to say what would happen if the election was today, which throws away the time-dependent uncertainty.

But 538 was giving predictions for different events each day in the moving "now cast" so there was no arbitrage.

The forecast still fluctuated way too much to be correct, as can be seen in Taleb's paper, which cites the "forecast" and not the "now cast."

Reply on news.ycombinator.com