Hacker News Reader

Comment by encoderer | Hacker News Reader

There is a strange cast of right-once hangers-on from a decade ago like burry that people love to quote in headlines. Nate silver is another example.

Why are these people so interesting to bloggers/journalists?

View on HN

Not a Nate Silver fan, here, but I don't think the "right once" sentiment about him captures his record accurately. What really happened is that millions of people who have an underdeveloped understanding of what it means for a lower-probability event to happen in a single trial are telling on themselves whenever they refer to him as having been "wrong". I mean, if you play only one round of russian roulette — but you lose — it's not really a surprise, is it. There are only six chambers.

Yeah, Silver simply tells us the odds. Someone did the analysis on it at one point and 538 was "right" at about the rate of their percentages.

i.e. For all the times they called an event 65%, it happened about 65% of the time.

So they weren't "wrong" the other 35% of the time, it's just those are the times the 35% chance bore out.

People have a habit of assuming "is likely" is the same as "guaranteed". Or that that's what the caller believe will happen. And then using the event once passed to discredit the initial analysis.

This is exactly why the 2016 election was interesting. Silver called it about 33% chance Trump would win. And then he did, and people said "AHA! You were WRONG!"

But he gave Trump a higher probability of winning than just about anyone, other than Trump himself. One of Silver's detractors, Sam Wang, even said it was so impossible Trump would win that he'd eat a bug if he got more than 240 electoral votes. (he chose a gourmet cricket https://www.politico.com/story/2016/11/sam-wang-poll-expert-...)

Nate and team do modeling well. Fat tails, correlated populations, and they deal with the crap datasets they get from pollsters. I always tune into his podcast when an election rolls around.

> This is exactly why the 2016 election was interesting. Silver called it about 33% chance Trump would win. And then he did, and people said "AHA! You were WRONG!"

What was funny about that one was he also got a bunch of "LOL no way you lunatic, he can't possibly have that large a chance and no-one else is saying so, so you're dumb, or maybe you're secretly a pro-Trump shill".

Caught it going both ways.

That “someone” includes 538 themselves. They’re very transparent about their track record and methodology.

That logic means you can never prove them wrong (or judge accuracy) unless they give a 100% probability to an outcome which they never would.

Sounds kinda useless?

I mean, if they said the probabilities were 99 to 1 or 1 to 99, regardless the outcome they can say “well, we never said it couldnt happen”

No, we know how to deal with statistics. Sounds like you need a course in it.

Who is "we"?

In that case, please "educate" me on how to understand the quality of a prediction (was it accurate or not), of a binary event (say a Presidential candidate winning election), when both outcomes are both "probable" to a non-zero degree.

I'm eagerly looking forward to my "education".

For 538 specifically, can start by reading their own evaluations, as written in their helpfully titled "How Good Are FiveThirtyEight Forecasts?"[1]. You can then look up keywords describing their methodology to understand strengths/limitations of those methods.

On a broader scale, what you are asking about is called a "Scoring Rule". Wikipedia, as usual, provides an overview[2]. You can take the mean of a forecaster's score, which allows you to compare forecaster methodologies.

I'm not sure why your comment needed to be so aggressive.

[1] https://projects.fivethirtyeight.com/checking-our-work/

[2] https://en.m.wikipedia.org/wiki/Scoring_rule

A single prediction? You can't. Multiple predictions? Compare the predicted and the actual frequencies, that is, check whether, for example, (roughly) a third of the events that were predicted to happen with 33% certainty to happen actually happened. It is as bad to be not confident enough as it is to be too confident. The standard term for this is calibration[1,2].

Note that this is the easy part: a sports model that predicted a 50% win for the first team, a weather model that predicted historical averages, or a language model that predicted letter frequencies, would have near-perfect calibration, but would at the same time be pretty useless. The other part is discrimination: how educated your guesses are. That is not so simple to quantify, although the 538 articles above mention some of the possible measures.

That's it for what an uncertain binary prediction means; but why do we want one? Well, if you're betting (literally or figuratively) on an outcome, it probably makes a difference to you whether the "losing" possibility will come up 1% of the time or 40% of the time; but that does not seem that easy to formalize and may feel unsatisfactory.

In that case, here's a formal result.

An always-certain prediction service is obviously equivalent to a deterministic decision rule, which churns some data about the situation and says yes or no based on that. (They are the same thing.) An uncertain prediction service is (less obviously) equivalent to a randomized decision rule, which churns some data about the situation, tosses some (known) coins, and says yes or no based on both. (Take the service's result, output yes or no with the probabilities it gave.) Of course there's always a probabilistic decision rule that performs at least as well as any given deterministic one (run the deterministic rule, choose not to toss any coins, output its result).

It turns out (see e.g. the introduction[3] to Chentsov's monograph[4]) there can be randomized decision rules that are strictly better on average than any possible deterministic rule.

[1] https://fivethirtyeight.com/features/when-we-say-70-percent-...

[2] https://projects.fivethirtyeight.com/checking-our-work/

[3] https://books.google.lv/books?id=iqMluWtSFdoC&pg=PA8

[4] https://openlibrary.org/books/OL26831120M/, https://bookstore.ams.org/mmono-53

> A single prediction? You can't.

Thank you. I thought I was going slightly insane having someone argue otherwise.

I understand you could evaluate quality over many deterministic predictions, but I'd also presume that those have to all be similar in nature (i.e. all be election outcomes) otherwise there are too many confounding factors if you try and evaluate quality over vastly different prediction calculations.

It's cover for the increasing editorialization of 'news.'

Reporter wants to say something about 'I bet WFH will have negative impacts on white collar workers."

Their options are:

1) Do data-driven research and come up with a dry, sort-of-compelling article about recent data releases that doesn't get traction

2) Use a 'famous for being right in a big way' celebrity to 'launder' their idea.

We may live in an irreligious society, but humanity's innate desire to anoint prophets to warn us of a coming reckoning continues unabated.

I don't really understand why people are so obsessed with Nate Silver (particularly negatively). I listen to the 538 podcast and I'm interested in what he has to say about polling (and the discussions are always interesting, even if I don't always agree with them), but that's about it. I also listen to dozens of other podcasts where they talk about other things they know a lot about. It's also not like Nate is the word of god on polling. He's just another source of information. You can take or leave it. It doesn't matter. But why complain about him?

Burry was actually just right again, and in a major way.

He predicted the market would crash this year, and liquidated most of his portfolio in the first half of the year to get out of harm's way. Most of the rest of us are now nursing massive losses.

https://markets.businessinsider.com/news/stocks/big-short-mi...

If you are in finance/hedge funds you would need to be blind to not know that based on all the indicators around the world in the first quarter of 2022 - inflation, war, energy prices, lockdowns in China, data about trade and maritime transport etc.

I also liquidated most of my stock positions based on that data and I'm not in finance.

He deletes the takes and tweets where he's obviously wrong.

Actuslly he deletes all his takes and tweets.

I don't know about Burry, but Nate Silver has been right more than just once! https://projects.fivethirtyeight.com/checking-our-work/ is a statistical analysis of 538's track record in forecasting elections going back to 2008. The analysis shows that 538's predictions are generally pretty well-calibrated, in the sense that e.g. if there were 100 races where 538 predicted "Democrats have a 70% chance of winning this race", then the Democrats did actually win about 70/100 of those races.

I do agree with you, but if 2008 primary hadn’t happened I don’t think anything he’s done since is notable enough that we’d know who he is.

Just a guess though of course.

How about right 4 times, is that enough for you?

https://www.livewiremarkets.com/wires/grantham-this-is-a-bub...

Grantham is a notorious perma-bear, and given that article was posted right before a 20% rise in the market, I'm inclined to say that's another wrong call for him.

Generally I agree with you, but Nate Silver has yet to perform poorly.

There was a movie etc, and people know his name, and how hw knew about the crash, when nobody else did. So it attracts clicks because people want to know this time. Bloggers/journalists are interested in clicks.

people know the names (or movie name in this case) and clicking on a familiar name is probably more likely than a name no one has ever ever heard about.

there's also something inherently compelling about the outsider who gets something more right than the experts.

Does Nassim Nicholas Taleb qualify under this umbrella?

I don’t see talked about as much these days, curious what the sentiment is on him now?

(I really enjoyed his books)

Simple - they get clicks.

Pop culture knows the movie 'Big Short', but they probably have never heard of a Nobel prized economist.

Milton Friedman had a television show on PBS in the 1980s. It's not as though economists don't sell well on television.

That was the 1980s and it was PBS, one of the biggest of the few major channels on American airwaves. It wouldn't work today because economists cannot boil down complex ideas into 15 second videos.

Paul Krugman writes a column in the New York Times.

Paul Krugman is under qualified to write about economics for a hishschool newspaper.

> but they probably have never heard of a Nobel prized economist.

I think paul krugman is an pop culture figure too.

Probably for all the wrong reasons. He was pretty widely mocked at LSE when I attended there a decade ago and that’s an institution I’d expect him to still be well regarded at. He traded his academic bonafides years ago.

Reply on news.ycombinator.com