Hacker News Reader

Comment by jemmyw | Hacker News Reader

I see a particular problem with this kind of prediction which maybe someone else can explain. If you are predicting one off events like China invading Taiwan, or London being hit by a nuclear weapon, as a percentage, how can you ever know if the prediction was accurate? Whether the event happens or not does not validate or invalidate the percent chance of it happening. Is there some kind of aggregate that happens to all the predictions to validate them? If you give invasion an 8% chance and invasion happens it doesn't mean you're wrong, just that the world fell into that 8%.

View on HN

Probabilities turn out to be surprisingly meaningless sometimes. Especially for rare unrepeatable events.

A good way to make them a bit more concrete to put some money on it, but even then it's hard to be sure. You can however use this to measure how good someone is in predicting. Take sports for example, by betting according to someone's prediction you can see how much money you earn (if their probabilities are 'true' you are almost guaranteed to earn money, eventually). But really in that case you're hoping that someone's ability to predict one sports match is indicative of their ability to predict the next, which is a good bet, but obviously doesn't hold for these kinds of miscellaneous rare events.

In this case you can imagine a betting system, and you can imagine some people might turn out to be better at it than others (though the relevant measure is not the one used in this article), so I suppose you can talk about someone's ability to predict these events. Though in the end the bookie always wins, so make of that what you will.

Of course this being probability theory all of these methods only work with high probability, so they may not work at all.

You look across unrelated events that you scored similarly. In aggregate, across all things you predict as having an 8% chance to happen, 8% should actually happen.

That sounds reasonable. The article doesn't dig into it. Does this group actually make enough predictions to have this kind of validation?

Yes, serious forecasters predict on hundreds of questions & the relevant websites prominently feature "binary calibration" plots.

This is mine on Metaculus (mentioned in the article):

https://imgur.com/a/d3s67xk

There's still a bit of noise at 205 predictions but you can see a pattern emerging!

But if you can pick what you’re predicting on and pick the percentage you assign to an event, I fail to see how that gives you a calibration.

Because you make lots of predictions, and smooth out the numbers. If the events you give 8% probability actually happen 35% of the time, you aren't as calibrated as you are if they happen 10% or 5% of the time. The calibration numbers measure how much you are off by.

This makes no sense. If I have 1000 predictions and vary my estimates from 0-10%, that’s only 100 samples assuming you round to the nearest whole percent. And there’s no correlation between any of those samples. For example, I could say the probability of a lightning storm tomorrow is 1% and the probability of a war with China in the next year is 1% - calibrating amount those 1% events is clearly non sensical. You could try to calibrate among similar events but then you have no way of estimating that a 10% war between US and China vs a 1% of war between Russia and China.

Basically this is an exercise of garbage in and garbage out.

I think youre kind of right, its not really meaningful to know “im right about things that happen 5% of the time!”. But if a lot of things you bet on structurally happen to be low probability and of the same type (eg you bet on one type of catastrophic event like pandemic, crop disease, or war etc) then its useful

Or you’ve just learned how to optimize your score on the game you’re playing but the game isn’t actually about predicting the future.

Proper scoring rules like Brier mean that any attempt to optimize your score by definition mean that you are improving your calibration.

The article does dig into it. They have Brier scores.

See https://samotsvety.org/track-record/

That sounds a bit like the gambler's fallacy to me.

Doesn't seem right, if a state capital city base chance for a nuclear strike is 1% and you have 195 capitals or so 2 of them would be hit by now.

Maybe your base rate is way off then?

Nuclear strikes on N different capital cities aren't statistically independent events, and that kind of computation only makes sense for statistically independent events.

You're correct it mostly only works for a spread of predictions that don't have such a large influence on each other.

If one capital gets nuked the odds of another one suffering the same fate skyrocket.

Yeah, it's just about possible to imagine Kyiv or Tehran gets nuked in a "limited engagement" that never escalates further. But if, say, Lisbon gets nuked then the whole world is at war.

That might give you some confidence. But if you're looking at your performance on unrelated predictions, it won't necessarily detect problems with a given prediction.

I predict there's an 8% chance of one drive in my four-hard-drive array failing in the next year.

I predict there's an 8% chance the next UK election ends without any party holding a clear majority, leading to a labour-lib dem coalition government.

For the first prediction to be accurate, I just need to read the backblaze hard drive stats and multiply.

For the second prediction to be accurate, though? That depends on a lot of factors that are a lot harder to know. For example, would the lib dems be likely to enter a coalition, given how the last one went for them in 2015?

Sure, you cannot expect to reliably detect problems with a given prediction, since it could always be right or wrong due to chance. But also, what would you do if you had the ability to detect problems with a prediction?

If you just want to use a different prediction method that fixes the problem and makes better predictions, looking at aggregate performance over many unrelated predictions does help. Compare how well different methods do, go with the best.

But you could come up with a probability prediction on whether a random prediction from the predictor is accurate.

You don't know for that event.

If you predict 100s of events, then use a method like Brier scores, you can get a good idea of how good you are. Even then it's probabilistic, but with enough samples it becomes incredibly unlikely you are just the worlds luckiest guesser.

  > it becomes incredibly unlikely you are just the worlds luckiest guesser

And yet that is still more likely than the idea that you can predict the future.

In any case, we are not talking about _you_ becoming the world's best guesser, we are talking about _somebody_ becoming the world's best guesser. That is going to be someone, so no need to be surprised when they emerge.

you're right, but people predict the future all the time, if there's a pre-established model for that prediction. I just threw a ball in the air and predicted how long it'd stay there </akshually>

so I think this comes down to how the predictions were arrived. One way to do this is to ask individuals who both have similarly high predictive scores and bet on the same types of events to explain some of their past predictions, and if their methods are similar then you've learned a new predictive tool.

How is the difficulty of a certain prediction scored and weighted?

In these competitions it generally isn't. So, you can't compare, for example, my brier scores for predicting events a, b, c and yours for events x, y, z.

There aren't actually that many people sitting around predicting things and keeping track of it. The whole monkeys tossing coins argument sounds clever, except that there aren't many monkeys in the real world.

The standard solution in ML for this is cross entropy loss. You can think about it like this: from the probability estimate you derive two codewords, one saying it will happen and one say it won't happen. Then when the event happens (or doesn't) you write down the corresponding code in your log file. Then after a large number of predictions you can compare the size of people's log files. Someone having a shorter logfile means they used a more efficient encoding which means their probability estimates were better.

But yeah it's not possible to do this after just one event, you need some track record to be able to say that someone is statistically significantly better.

In a good prediction market the bets function as binary options, with a payout at 100 for being correct within the time frame.

so its more about what price between 0 and 100 would you bet on that belief

at 20 you have a 500% gain if it resolves at 100

and this willingness to bet is telegraphed to the crowd, it doesn't really mean conviction but maps to the crowds aggregate understanding of the probability occurring or not

> Is there some kind of aggregate that happens to all the predictions to validate them?

Most likely an application of the law of large numbers

https://en.wikipedia.org/wiki/Law_of_large_numbers

I think this is essentially the divide that separates frequentists from bayesians. I'm sure someone can chime in and correct my summary if it's wrong, but broadly speaking, my understanding is that frequentists consider probabilities to only make sense in the context of a repeatable event where you can measure how often something occurred, whereas bayesians consider probabilities to represent the "likelihood" of something happening even in a one-off event, which often is conveyed in terms of whether you'd be willing to take a given bet with certain odds; if a bayesian thinks something has an 80% chance of happening, they'd consider it a fair bet to offer someone $4 if it does happen but get $1 from them if it doesn't, since 80% is four times more likely than 20%.

The downside of bayesian probability that you point out is that isn't not straightforward to evaluate whether odds were correct or not. An example of this would be the 2016 US presidential elections; even the highest estimations of the odds of Trump beating Clinton were 30%, which after the fact was often cited as the predictions being "wrong". It's not easy to falsify this though, because a 30% chance doesn't mean something is _guaranteed_ not to happen; maybe it really was 70% likely not to happen, but we happened to end up in the 30%!

Frequentism doesn't suffer from this issue, but it also makes it impossible to talk about certain types of events (like presidential elections). In practice, bayesian probability gets used a lot when people want to talk about those sorts of events because there's not really any obvious alternative.

No that’s not the divide. Bayesian requires you to have an estimate of the prior and a reasonable way to update that estimate by validating against ground truth. Here there’s no ground truth and no principled way that the prior is being picked. It’s numerology masking itself as statistical rigor.

I have no problem with 538 saying that politician A has a 30% chance of winning because it’s a blended weighted average of several independent measurements of the ground truth (polling). There’s no independent measurements of the ground truth happening here. Indeed, things regarding war plans would be classified documents random people wouldn’t have to come up with a better estimate.

Previous responses have given you some great food for thought, but my two cents is that what's being discussed in the article is no different from horoscopes or the old lady in the tent with the crystal ball.

I'm inclined to agree with you. Events like China invading Taiwan and a nuclear strike on London cannot be aggregated with other predictions because they are unique. There's no correlation between predictions to allow for it. Showing that you have some accuracy predicting any number of other things does nothing to prove you have accuracy predicting a % chance of a unique event like those ones. Further, if those events happen or not there's no way to validate the prediction. There aren't 100 Chinas ready to invade 100 Taiwans. Even expanding to other invasions of a very broadly similar nature doesn't give enough data points. Looking at historical data is bum too, there's too much geopolitical and technological change since. And that makes the base rates meaningless. All my opinion of course.

> Showing that you have some accuracy predicting any number of other things does nothing to prove you have accuracy predicting a % chance of a unique event like those ones.

When someone has a track record of consistently predicting these "unique" events better than average, then clearly there must be some pattern there that they're picking up on and the events aren't as unique as one would think.

At the end of the day, someone who financially invests based on these predictions will eventually end up richer than someone who doesn't, whether you believe that should be possible or not.

They are mixing predictions that do allow for accuracy on base rates with ones that don't. There is clearly a bias that this particular group has leveraged rather than them being better at prediction.

Proving that this isn't pure chance is impossible.

This is the great advantage of financial markets, where a prediction is not expressed in a percentage, but betting a certain amount of money on an outcome.

Taleb's works and often posts cover exactly this.

Reply on news.ycombinator.com