Hacker News Reader

Yeah, "extremely possible" is perhaps a poor choice of words, and could be interpreted uncharitably as a hedge against any outcome, but I think there's an obvious charitable interpretation too: that things that have a 20% chance of occurring turn out to occur around 20% of the time. No one would be blown away by surprise if I proclaimed "SIX!" then rolled a six-sided die and got a six, but that's even less likely than 20%.

So it's reasonable to say that, if 538 claims that an event has a 20% probability of happening, and that event happens, no one should be astonished. Of course, such events ought to happen about 20% of the time, if 538's models are good. And that evaluation is precisely what 538 published a few days ago: https://projects.fivethirtyeight.com/checking-our-work/

Reminds me of the old essay on "Words of Estimative Probability" from the CIA

https://www.cia.gov/library/center-for-the-study-of-intellig...

That's a great table they have with associated words and numerical probability associated with them.

Loads of people criticized Silver for not having "predicted" Trump's electoral victory. I imagine Silver's language was trying to hand-hold people to an understanding that even 20% probability events happen 1 out of 5 times. With so many people trying to twist and spin every headline, prediction, number, I appreciate Silver's attempt to make a reasonable guess with transparent methodologies.

The same people were criticizing him a day before because everybody else had Hillary at a 99% chance to win. Nate tweets that Trump is one standard polling error away from winning and people scoffed.

It is worth noting that 538 doesn't keep a historical record of their predictions, making it difficult to really see how they predict outcomes coming up to a conclusion. You can only see the latest/final prediction.

538 also predicted lower chances of victory in the weeks and months leading up to that, then adjusted it on election day vanishing away their earlier guesses. This isn't wrong on the face of it, but people infer from the final datapoint to the conclusion later that same day about 538's accuracy.

What I am saying is: 538 needs to keep a historical record of their predictions.

538 published this last week: https://fivethirtyeight.com/features/when-we-say-70-percent-...

Maybe this was true of past elections, but their 2018 house forecast had a “how this forecast has changed” section that showed their historical predictions throughout election season. It’s still visible here: https://projects.fivethirtyeight.com/2018-midterm-election-f...

> Maybe this was true of past elections, but their 2018 house forecast had a “how this forecast has changed” section that showed their historical predictions throughout election season.

It has not changed. They did this the whole way back when it was an independent blog before the NY Times (later ESPN then ABC News) picked them up.

Where are they? Five different posters have claimed this, but zero have been able to link to the historical predictions.

> Where are they? Five different posters have claimed this, but zero have been able to link to the historical predictions.

They are literally charted on the page with the current prediction, e.g., for the 2016 Presidential race (which is locked on the last forecast):

https://projects.fivethirtyeight.com/2016-election-forecast/ , under “How the forecast has changed”, it charts the forecast (by default, the win probability) for the entire time the race was being forecast.

The 2016 election predictions are still posted. https://projects.fivethirtyeight.com/2016-election-forecast/...

Yes the newest one. Not the historical ones.

Maybe I misunderstand your argument, but doesn't that linked page have a section "How the forecast has changed"[1]?

[1] https://projects.fivethirtyeight.com/2016-election-forecast/...

> It is worth noting that 538 doesn't keep a historical record of their predictions, making it difficult to really see how they predict outcomes coming up to a conclusion. You can only see the latest/final prediction.

It's worth noting that you are completely wrong: not only do they keep such a record, they also provide a handy chart of it on the latest prediction page so you can easily see exactly how the prediction changed over time.

Where? Go ahead and link to it.

Hey I don't have a dog in this fight, but I think this is the link you're looking for.

https://github.com/fivethirtyeight/checking-our-work-data

Nope. I'm looking for 538's predictions. You cannot go from the raw data they used back to how they interpreted it.

Please look at the actual data on the linked page. Each of the csv files on that page has thousands of predictions 538 made going back as far as 2008. This isn't the raw data they used to make the predictions, these are the actual predications that they made.

Edit: For example, here are the headers on the file named presidential_elections.csv

year,office,state,district,election_date,forecast_date,forecast_type,party,candidate,projected_voteshare,actual_voteshare,probwin,probwin_outcome

These are the actual forecasts they made for presidential elections going all the way back to 2008.

Here is the historical data for the 2018 House forecast:

- Web: https://projects.fivethirtyeight.com/2018-midterm-election-f...

- JSON: https://projects.fivethirtyeight.com/2018-midterm-election-f...

You are correct they do not publish the exact math of the forecasts, but you can see how the forecast changed over time.

The link I provided does actually show every single forecast 538 has ever made. OP was wrong when they said it was the raw data they use to make the predictions. Check out the link I provided, it has everything OP was asking for.

Nice! I didn’t poke into it much further cause I knew about this one.

I have a hard time understanding this: what does it mean to assign probabilities to one-off events?? The only way to verify probabilities predictions is to test it a great number of times. Otherwise you can say anything and never be wrong (except for 0 or 100%).

Those questions underpin big chunks of philosophy.

Assigning a probability to a one-off event is enumerating all the ways it could happen, all the ways it could not happen and assigning a probability to each of those ways. Obviously there is a lot of guesswork; but if you need to make a decision based on the future that approach gives you a much better chance of making a good decision. In practice an event will be made up of components that are more predictable than the whole, and some real uncertainties. There is a lot to be gained by thinking hard about the situation, and assigning a probability will do that.

Silly example - how do I estimate my risk of falling climbing up a set of stairs that I've never climbed before?

* Baseline risk of tripping - I have a lifetime of data.

* Increase of risk being on a staircase - I have an area I want to put my foot on (a specific step) that is about 1/3 of the area that my foot usually falls in, so that increases the risk by an amount that can be reasonably estimated.

* I will watch my foot - maybe an order of magnitude improvement in precision.

This is enough to let me estimate the risk of carrying a bulky object (that obscures my view of my feet) up a staircase. I've isolated the uncertainty (how much does visual observation change the odds) from the certainties (areas, background rate).

Now I can take that to several experts who will identify new mechanisms and tighten up my estimations on how big a deal the components are. In this way - even though the final % I come to would still be a bit arbitrary - it is starting to become a summary of what a large number of people think about the inputs to the problem and their relative magnitudes. Being able to communicate all that thinking with a single number is a miracle in its own way.

It’s not that the election itself is subject to random uncertainty so much as the data that they are using to make the prediction is.

The data might favour one candidate, but even assuming it is unbiased and representative, it is only a random sample, and there is a chance it could be randomly wrong.

Every event is a one-off event. In particular the Trump election was the most one-off event in recent memory :)

Kidding aside, your observation is correct, you need to perform repeated predictions. 538 does just that. They keep predicting a whole lot of outcomes, so you can check their track record. They even have a challenge for the audience, where you can record your own predictions and compare them with their corresponding predictions (see for example [1] for NFL games). The scoring of the predictions uses the Brier score [2], which is just a version of R-squared for classification problems.

In the case of the November 2016 election, other prediction sites were giving the Trump team a 1-2% chance, 538 was giving them a 20% change. Considering that, 538 comes out as the clear winner.

Which brings us to what I consider to be the best measure of prediction accuracy (my own invention, it doesn't have a scientific name yet): you can compare two sequences of predictions (let's say yours agains 538) by simulating bets at some mid-odds level. For example if I say Trump's probability of winning 2020 is 45% and you say it's 35%, then I'm willing to give you 2-to-3 odds, while you're willing to take that. We then see who stays ahead most of the time in this betting simulation.

For what is worth, here's a recent survey of different scoring types for predictions: "Assessing the performance of prediction models: a framework for traditional and novel measures" [3]

[1] https://projects.fivethirtyeight.com/nfl-predictions-game/ [2] https://en.wikipedia.org/wiki/Brier_score [3] https://www.ncbi.nlm.nih.gov/pubmed/20010215

> In the case of the November 2016 election, other prediction sites were giving the Trump team a 1-2% chance, 538 was giving them a 20% change. Considering that, 538 comes out as the clear winner.

One common mistake I've seen is people mistaking that 20% as the expected vote ratio. It definitely wasn't; that was hidden behind an apparently too-well-hidden toggle (buttons above the graph), and was floating around 45% Trump and 48% Clinton, with overlapping error ranges.

You can bet money on one off events. People and bookies do this frequently and you can see if they loose or win.

But the betting odds depend on how much knowledge you have - for someone with perfect knowledge the odds may be 0 or 1, if you know nothing you might guess 50:50 for practical purposes.

You can't verify probabilities for any single given event, but you can look at the outcomes for all events for which predicted probabilities were given, and use that to estimate the accuracy of the methodology/models used.

So, for example, when 538 said that Trump had a 30% chance of winning, its hard to say how accurate that was given it was a one time event. But if you look over all of their election predictions, and look at those they predicted to win 30% of the time, about 30% of them should have been victories (with some margin for error, of course). If 50% were victories, or only 10%, then maybe that would indicate their models aren't doing so hot.

I remember him saying before the election that the odds of Trump winning we're roughly the odds of a randomly picked day being a Saturday. Which always seemed pretty clear to me - Saturdays definitely happen, Trump could definitely happen. And then it did, and it was nothing like a Saturday, not at all.

That's not specifically a Brooklyn thing, it supposedly has more to do with the fact that most Italian families in the tri-state area came to the U.S. before Italy was a country with a standardized language. That's why e.g. in Connecticut they pronounce pizza as abeetz. For whatever reason knowing how to pronounce and spell Italian words "incorrectly" has become a point of local pride, even for people who aren't at all ethnically Italian.

Is this related to the characters in the Sopranos dropping the final vowels in a bunch of Italian words? Madonn, gabagool, prozhut, etc.

For sure the pronunciation thing is real, either due to regional accents or linguistic divergence; I've heard it firsthand used unironically by Brooklyn and Philly Italian-Americans, as well as ample examples from television and movies. But writing it that way I haven't seen outside of Taleb (and Google seems to bear me out to some extent), and using it that way isn't wrong, per se, it's just an eye-rolling moment; someone trying too hard to show that it isn't book-learning, it's "street-learning".

Yes. My favorite example of that is risott, since that's something that people actually commonly order in restaurants.

Then there's also things like tirami-sue vs tirami-zoo, which don't even make sense since the dessert wasn't invented until recently.

> But in this case I'm having trouble following what his criticism even is

A piece by Isaac Faber [1] from December last year clearly described what Taleb's position is. A summary paragraph towards the end:

> Here we can say, with some confidence that FiveThirtyEight predictions are not always reliable probabilities. However, they masquerade as one, being between 0 and 1 and all. This is Taleb’s primary argument; FiveThirtyEight’s predictions do not behave like probabilities that incorporate all uncertainty and should not be passed off as them.

Faber's piece was discussed on HN [2] soon after it appeared.

[1] https://towardsdatascience.com/why-you-should-care-about-the...

[2] https://news.ycombinator.com/item?id=18704376

By far a much better write up of the issue.

I confess, I still don't follow the critique. Let's take the output of a skin-in-the-game model, like a binary futures market. This also predicts numbers between 0 and 1, that look a lot like probabilities, but by the argument presented here, they are not probabilities, and if you look at the output of a futures market, then you are being scammed because the market itself is not giving a decision boundary; it's just outputting the last trade price.

Yes, this allows a forecaster to say that even if the <50% candidate wins, that they weren't wrong, but so what? The forecasts should be reasonably well-calibrated -- that is, of all 20% predictions, 20% of them go in the 20% direction. 538 recently published a report of their calibration, which bears out their results (and kicked off the latest round of name-calling between Taleb and Silver).

Is the criticism just that we shouldn't look at Silver as a person with anything interesting to say because he won't commit 100% to one side of an argument? I honestly don't understand what the critique is, and this link I don't find helpful.

Here's another article analyzing their disagreement: https://towardsdatascience.com/why-you-should-care-about-the...

>I've often found Taleb's ideas to be profound and interesting

Really? Because financial crashes happen a lot more regularly than black swans. This seems like par for the course for Taleb; I've never read him in depth but he always seemed like a memer.

Taleb is simultaneously both worth reading as well as one of the most pretentious, unpleasant asses I've ever read.

The Black Swan was mostly fantastic, but Antifragile was mostly regurgitation.

The biggest problem with him is he thinks he's much, much smarter than he is, like a kind of inverse-Dunning-Kruger effect. It's odd because he is quite a smart man in the first place; there's no need for him to be such a pretentious asshole.

I've never read Black Swan. I found Antifragile hugely thought-provoking. Knowing both, would you imagine I had anything to gain from also reading Black Swan, or do you really think they basically cover the same ground?

I read both, and Fooled By Randomness. I would say you stand to gain a lot reading about the older ones as well.

There is a lot of overlap in the Venn Diagram between the two. I would say read the first 80-100 pages of Black Swan and decide if you would like to complete the book.

Taleb's forte is applying various insights from practitioners of financial options to other domains.