A ragtag band of internet friends became the best at forecasting world events | Hacker News Reader

A ragtag band of internet friends became the best at forecasting world events

vox.com

229 pointsthyrsus132 comments

This is very interesting. A discussion of the process of estimating and predicting future events, specifically going into their focus on "base rates". Apparently they are significantly better than other groups that enter most predicting competitions. (who knew there was such a thing?)

I particularly liked the idea of base rates and averaging them. [1] I can see here that the idea of them is not to be on their own accurate (some of the methodologies individually are dumb), but do get an idea for the scales of numbers to talk about (the difference between a 5% event and a 10% event is very hard to notice as a human)

[1] "One was the rate at which provinces claimed by China (like Hong Kong, Macau, and Tibet) have eventually been absorbed, peacefully or by force; another was how often control of Taiwan has changed over the last few hundred years (twice; once when Japan took over from the Qing Empire in 1895 and once when the Chinese Nationalists did in 1945); the third base rate used Laplace’s rule. Laplace’s rule states that the probability of something that hasn’t happened before happening is 1 divided by N+2, where N is the number of times it hasn’t happened in the past. So the odds of the People’s Republic of China invading Taiwan this year is 1 divided by 75 (the number of years since 1949 when this has not happened) plus 2, or 1/77, or 1.3 percent.

Sempere averaged his three base rates to get his initial prediction: 8 percent"

Thanks!

Out of interest - why don't you guys do way more of this for financial markets?

Presumably you don’t publish your strats there.

I'm not sure it's quite the same thing though. It's not a strat, it's a judgement call. And, they can put a bet on at time t and then publish at time t + 1 min. This is done by hedge funds, they do publish/pitch their ideas sometimes.

Yeah, good question. A few years ago the answer may have been that we thought we could influence more money than we could make.

Fwiw, when I've tried it, it has tended to suck in a whole lot of my time.

For me, other projects seem more meaningful. E.g., right now I'm setting up this: https://alert-team.org, another member is behind https://theaidigest.org/progress-and-dangers, a third works at https://forecastingresearch.org/, etc. Maybe we're an idealistic bunch, but also maybe we're making a bad judgment call, and still yet it's possible we couldn't cut it.

> Maybe we're an idealistic bunch

I mean, it makes me quite happy that y'all apparently use your skills for wholesome, humanitarian things instead of just focusing on making $$$.

Aren't you just itching to see if you can do it against the hottest competition on the planet? Ie, and I mean this with respect - financial markets are the major leagues of prediction - it feels a little bit like you are playing in the minor leagues. And it's not just a chest beating exercise - you guys appear genuine and to want to push forward the science/art of prediction. But if you can't beat the current best, you aren't state of the art, let alone able to push things forward. You might be pushing things backwards.

fwiw, it appears you'd have a decent chance. You guys seem to have an extreme level of objectivity and rationality. There is a lot of domain knowledge required to bet on say, healthcare stocks or whatever, but maybe too much domain knowledge harms rather than hurts.

this

Naively, if any of the predictions could be made +EV accounting for time preference, the boffins' predictions should have been mirrored into the market to someone's profit.

I remain unconvinced that they're not merely very lucky, the same way that some hedge fund operators or day traders are lucky.

Has your group made predictions on nuclear fusion timelines, or mean temperature rise due to climate change?

Eh, the second part to me seems like a less useful forecasting. Generally it seems fore asting is focused on human/political events that obviously don't already have useful modelling.

For climate change, there's already a lot of existing models out there calculated by much more rigorous metrics than "the Laplace rule" or just naively adding together averages.

Climate models just give us scenarios. Conceptually they give us the function f(emissions) = warming + error. But we don't know the probability distribution of emissions, so we don't know the probability distribution of f. Coming up with a reliable probability distribution requires considering things like politics, world events, economic growth, technology breakthoughs, etc. IPCC etc make attempts to model the distribution of future emissions, but it's less rigorous and scientific than the actual underlying climate model.

> the probability distribution of emissions

We know that every species consumes all available resources until an external force, whether it's predators or environmental collapse checks its growth. Humans are no different - any time a poor person uses a paper straw, a rich one will fly their private jet. Muh backyard arguments on nuclear means we will continue to burn coal and gas as fast as possible, even while renewables help drive up more demand.

So I think it's safe to assume that emissions will continue growing.

I'm hoping for a rigorous analysis. For example, X := "arrival time of fusion with a cost lower than natural gas in $ per kWh" is a random variable with a distribution. Our assumptions about the distribution of X will impact our estimated distribution of emissions, and hence impact our estimated distribution of f(emissions). There's lots of these questions that contribute, both technological and behavioral in nature.

Yes there's a lot of existing models, and a lot of the existing models have been wrong in the past. Climate science has always been very politicized. I for one would welcome predictions from people who have a track record of predicting things correctly. Climate science has a mixed track record [0].

[0]: https://cei.org/blog/wrong-again-50-years-of-failed-eco-poca...

FWIW, I'm very much NOT endorsing cei.org, nor saying climate change is not a big problem. Just saying that the track record of climate scientists is not great. Climate change is important enough to warrant the attention of people who can actually predict things.

Honestly, that sounds completely arbitrary, especially the one about 1/75. It also ignores anything that would alter any thinking person's probability calculations in the real world. For example, if tomorrow the Chinese premier says that Taiwan belongs to China and starts amassing navies, the % wouldn't change. I'd personally much _much_ rather listen to a view from some scholar or expert in the geopolitical situation than these "base rate" calculations.

It's supposed to be a completely uninformed prior. The only information, as you can see, is introduced in the formula with the above explanation.

Forecasting in general is a silly affair. As a data scientist I get all kinds of requests for all kinds of predictions, and my only real job is to part those requesting parties from their money.

The Chinese government has been saying that Taiwan belongs to China and amassed navies every year for almost 80 years now. (Longer if you count previous governments.) Base rates are important because what you think is a highly significant trigger event that will surely have major consequences may actually be a frequent occurrence that is typically inconsequential.

Or in other words, if you want to say "this time is different" and deviate from the base rate, you actually need to look at the previous times to find out whether it really is different this time, and if so, by how much.

Ok, you can adjust what geopolitical event we think it's significant. Again, since I'm not an expert in the matter, I'm likely to get it wrong, as it seems I did here :) But my point stands, I'd trust expert opinion over these simplistic, falsely confident quantitative approaches.

The percentages of democratic citizens assimilated into china would reach quiet a interesting percentage with taiwan assmiliated into the mainland. You can censor and propagandize all you want, but people who travelled or know better, just wont blieve a word of it happily ever after. One wonders what the internal "knowing" citizen rate is by now in china.

The party relies on divide and conquer of uniformed fools. But the fools are in the minority by now - and the divide and conquer is one sneaky trustfall decentralized communication platform away from never working again.

I think the idea is that instead of trying to argue, just be empirical. New information would have to be added along with context on how that information has impacted odds in the past

If the information is entirely new (ie, what impact does aliens taking over the Moon have on China/Taiwan) then there'll always be difficulty estimating that impact

This is just the base rate, not the final prediction. Your actual prediction can incorporate any information you want, but the base rate gives you a starting point and a sanity check.

Also, as the article notes, scholars like Tetlock have studied the the track record of scholars and experts and found them to be less accurate than this type of approach.

No, Tetlock et all have found very large crowds to outperform smaller groups of experts. This however is an article about small groups of non experts competing against each other. It’s not even blending the predictions across teams.

It’s a literary sleight of hand but useful to note as it undermines the entire premise of the article. That’s because these predictions are bunk because these techniques don’t work with the stock market which uses far more rigorous statistical methods for pricing (which happened 60s-90s with the rise of quants).

That's a good point. Again, even as a base rate I wonder if it's any good as it seems extremely hand wavy and arbitrary, but I can see how it being part of a multi-source approach can help.

There are comments similar to this all throughout the thread. It reminds me of the movie Moneyball. Even when the numbers are clear to see, people will still much rather follow their intuition. The Oakland Athletics' insistence on sticking to the numbers anyway is what led to their enormous success, and now the same strategy is copied by sports teams around the world.

It can be hard to convince people with math alone.

I'm onboard with trusting data driven and numbers approach. This one just seems like a bad one if it doesn't account for how obvious actions would alter its results. But as some other comment said, maybe this base rate is not the whole story and it's just one ingredient in the total risk calculation. That makes more sense.

I see a particular problem with this kind of prediction which maybe someone else can explain. If you are predicting one off events like China invading Taiwan, or London being hit by a nuclear weapon, as a percentage, how can you ever know if the prediction was accurate? Whether the event happens or not does not validate or invalidate the percent chance of it happening. Is there some kind of aggregate that happens to all the predictions to validate them? If you give invasion an 8% chance and invasion happens it doesn't mean you're wrong, just that the world fell into that 8%.

You look across unrelated events that you scored similarly. In aggregate, across all things you predict as having an 8% chance to happen, 8% should actually happen.

That sounds reasonable. The article doesn't dig into it. Does this group actually make enough predictions to have this kind of validation?

Yes, serious forecasters predict on hundreds of questions & the relevant websites prominently feature "binary calibration" plots.

This is mine on Metaculus (mentioned in the article):

https://imgur.com/a/d3s67xk

There's still a bit of noise at 205 predictions but you can see a pattern emerging!

But if you can pick what you’re predicting on and pick the percentage you assign to an event, I fail to see how that gives you a calibration.

Because you make lots of predictions, and smooth out the numbers. If the events you give 8% probability actually happen 35% of the time, you aren't as calibrated as you are if they happen 10% or 5% of the time. The calibration numbers measure how much you are off by.

This makes no sense. If I have 1000 predictions and vary my estimates from 0-10%, that’s only 100 samples assuming you round to the nearest whole percent. And there’s no correlation between any of those samples. For example, I could say the probability of a lightning storm tomorrow is 1% and the probability of a war with China in the next year is 1% - calibrating amount those 1% events is clearly non sensical. You could try to calibrate among similar events but then you have no way of estimating that a 10% war between US and China vs a 1% of war between Russia and China.

Basically this is an exercise of garbage in and garbage out.

I think youre kind of right, its not really meaningful to know “im right about things that happen 5% of the time!”. But if a lot of things you bet on structurally happen to be low probability and of the same type (eg you bet on one type of catastrophic event like pandemic, crop disease, or war etc) then its useful

Or you’ve just learned how to optimize your score on the game you’re playing but the game isn’t actually about predicting the future.

Proper scoring rules like Brier mean that any attempt to optimize your score by definition mean that you are improving your calibration.

The article does dig into it. They have Brier scores.

See https://samotsvety.org/track-record/

That sounds a bit like the gambler's fallacy to me.

Doesn't seem right, if a state capital city base chance for a nuclear strike is 1% and you have 195 capitals or so 2 of them would be hit by now.

Maybe your base rate is way off then?

Nuclear strikes on N different capital cities aren't statistically independent events, and that kind of computation only makes sense for statistically independent events.

You're correct it mostly only works for a spread of predictions that don't have such a large influence on each other.

If one capital gets nuked the odds of another one suffering the same fate skyrocket.

Yeah, it's just about possible to imagine Kyiv or Tehran gets nuked in a "limited engagement" that never escalates further. But if, say, Lisbon gets nuked then the whole world is at war.

That might give you some confidence. But if you're looking at your performance on unrelated predictions, it won't necessarily detect problems with a given prediction.

I predict there's an 8% chance of one drive in my four-hard-drive array failing in the next year.

I predict there's an 8% chance the next UK election ends without any party holding a clear majority, leading to a labour-lib dem coalition government.

For the first prediction to be accurate, I just need to read the backblaze hard drive stats and multiply.

For the second prediction to be accurate, though? That depends on a lot of factors that are a lot harder to know. For example, would the lib dems be likely to enter a coalition, given how the last one went for them in 2015?

Sure, you cannot expect to reliably detect problems with a given prediction, since it could always be right or wrong due to chance. But also, what would you do if you had the ability to detect problems with a prediction?

If you just want to use a different prediction method that fixes the problem and makes better predictions, looking at aggregate performance over many unrelated predictions does help. Compare how well different methods do, go with the best.

But you could come up with a probability prediction on whether a random prediction from the predictor is accurate.

Probabilities turn out to be surprisingly meaningless sometimes. Especially for rare unrepeatable events.

A good way to make them a bit more concrete to put some money on it, but even then it's hard to be sure. You can however use this to measure how good someone is in predicting. Take sports for example, by betting according to someone's prediction you can see how much money you earn (if their probabilities are 'true' you are almost guaranteed to earn money, eventually). But really in that case you're hoping that someone's ability to predict one sports match is indicative of their ability to predict the next, which is a good bet, but obviously doesn't hold for these kinds of miscellaneous rare events.

In this case you can imagine a betting system, and you can imagine some people might turn out to be better at it than others (though the relevant measure is not the one used in this article), so I suppose you can talk about someone's ability to predict these events. Though in the end the bookie always wins, so make of that what you will.

Of course this being probability theory all of these methods only work with high probability, so they may not work at all.

In a good prediction market the bets function as binary options, with a payout at 100 for being correct within the time frame.

so its more about what price between 0 and 100 would you bet on that belief

at 20 you have a 500% gain if it resolves at 100

and this willingness to bet is telegraphed to the crowd, it doesn't really mean conviction but maps to the crowds aggregate understanding of the probability occurring or not

You don't know for that event.

If you predict 100s of events, then use a method like Brier scores, you can get a good idea of how good you are. Even then it's probabilistic, but with enough samples it becomes incredibly unlikely you are just the worlds luckiest guesser.

  > it becomes incredibly unlikely you are just the worlds luckiest guesser

And yet that is still more likely than the idea that you can predict the future.

In any case, we are not talking about _you_ becoming the world's best guesser, we are talking about _somebody_ becoming the world's best guesser. That is going to be someone, so no need to be surprised when they emerge.

you're right, but people predict the future all the time, if there's a pre-established model for that prediction. I just threw a ball in the air and predicted how long it'd stay there </akshually>

so I think this comes down to how the predictions were arrived. One way to do this is to ask individuals who both have similarly high predictive scores and bet on the same types of events to explain some of their past predictions, and if their methods are similar then you've learned a new predictive tool.

How is the difficulty of a certain prediction scored and weighted?

In these competitions it generally isn't. So, you can't compare, for example, my brier scores for predicting events a, b, c and yours for events x, y, z.

There aren't actually that many people sitting around predicting things and keeping track of it. The whole monkeys tossing coins argument sounds clever, except that there aren't many monkeys in the real world.

> Is there some kind of aggregate that happens to all the predictions to validate them?

Most likely an application of the law of large numbers

https://en.wikipedia.org/wiki/Law_of_large_numbers

Taleb's works and often posts cover exactly this.

The standard solution in ML for this is cross entropy loss. You can think about it like this: from the probability estimate you derive two codewords, one saying it will happen and one say it won't happen. Then when the event happens (or doesn't) you write down the corresponding code in your log file. Then after a large number of predictions you can compare the size of people's log files. Someone having a shorter logfile means they used a more efficient encoding which means their probability estimates were better.

But yeah it's not possible to do this after just one event, you need some track record to be able to say that someone is statistically significantly better.

Previous responses have given you some great food for thought, but my two cents is that what's being discussed in the article is no different from horoscopes or the old lady in the tent with the crystal ball.

I'm inclined to agree with you. Events like China invading Taiwan and a nuclear strike on London cannot be aggregated with other predictions because they are unique. There's no correlation between predictions to allow for it. Showing that you have some accuracy predicting any number of other things does nothing to prove you have accuracy predicting a % chance of a unique event like those ones. Further, if those events happen or not there's no way to validate the prediction. There aren't 100 Chinas ready to invade 100 Taiwans. Even expanding to other invasions of a very broadly similar nature doesn't give enough data points. Looking at historical data is bum too, there's too much geopolitical and technological change since. And that makes the base rates meaningless. All my opinion of course.

> Showing that you have some accuracy predicting any number of other things does nothing to prove you have accuracy predicting a % chance of a unique event like those ones.

When someone has a track record of consistently predicting these "unique" events better than average, then clearly there must be some pattern there that they're picking up on and the events aren't as unique as one would think.

At the end of the day, someone who financially invests based on these predictions will eventually end up richer than someone who doesn't, whether you believe that should be possible or not.

They are mixing predictions that do allow for accuracy on base rates with ones that don't. There is clearly a bias that this particular group has leveraged rather than them being better at prediction.

Proving that this isn't pure chance is impossible.

This is the great advantage of financial markets, where a prediction is not expressed in a percentage, but betting a certain amount of money on an outcome.

I think this is essentially the divide that separates frequentists from bayesians. I'm sure someone can chime in and correct my summary if it's wrong, but broadly speaking, my understanding is that frequentists consider probabilities to only make sense in the context of a repeatable event where you can measure how often something occurred, whereas bayesians consider probabilities to represent the "likelihood" of something happening even in a one-off event, which often is conveyed in terms of whether you'd be willing to take a given bet with certain odds; if a bayesian thinks something has an 80% chance of happening, they'd consider it a fair bet to offer someone $4 if it does happen but get $1 from them if it doesn't, since 80% is four times more likely than 20%.

The downside of bayesian probability that you point out is that isn't not straightforward to evaluate whether odds were correct or not. An example of this would be the 2016 US presidential elections; even the highest estimations of the odds of Trump beating Clinton were 30%, which after the fact was often cited as the predictions being "wrong". It's not easy to falsify this though, because a 30% chance doesn't mean something is _guaranteed_ not to happen; maybe it really was 70% likely not to happen, but we happened to end up in the 30%!

Frequentism doesn't suffer from this issue, but it also makes it impossible to talk about certain types of events (like presidential elections). In practice, bayesian probability gets used a lot when people want to talk about those sorts of events because there's not really any obvious alternative.

No that’s not the divide. Bayesian requires you to have an estimate of the prior and a reasonable way to update that estimate by validating against ground truth. Here there’s no ground truth and no principled way that the prior is being picked. It’s numerology masking itself as statistical rigor.

I have no problem with 538 saying that politician A has a 30% chance of winning because it’s a blended weighted average of several independent measurements of the ground truth (polling). There’s no independent measurements of the ground truth happening here. Indeed, things regarding war plans would be classified documents random people wouldn’t have to come up with a better estimate.

The base rate arguing seems like specious reasoning. For example, if you had a volcano that erupts roughly ever 100 years, base rate reasoning using the past 99 years of data would suggest that the probability is 0 and using 1000 years of data would suggest it's ~10% when in reality your base rate in the year following an eruption is 0 with every passing year your probability of an eruption would increase & increase past 10% for every year past 100 that goes without explosion. Same goes for something like war where pressures build up and war becomes more likely rather than less. So getting judged that you're better at predicting by giving low probabilities for rare events isn't that insightful because you'd be outperformed by someone who predicts a black swan event because the magnitude of the event matters.

> The prediction got some press attention and earned rejoinders from nuclear experts like Peter Scoblic, who argued it significantly understated the risk of a nuclear exchange. It was a big moment for the group — but also an example of a prediction that’s very, very difficult to get right. The further you’re straying from the ordinary course of history (and a nuclear bomb going off in London would be straying very far), the harder this is.

Yup, the group got it right but predicting a rare event doesn't happen isn't that difficult, it's just notable because everyone was overly freaked out, particularly in the media due to self-repeated sensationalism. Peter Scoblic is correct that the risk is significantly understated because it's not correctly adjusting for the impact of the black swan event happening (e.g. if a nuclear explosion were to occur, you'd expect nuclear retaliations).

A hermit lived by the volcano, and every day he would put up a sign for all the tourists hiking up to see it:"100% accurate prediction: Won't Erupt Today."

One day, 30 years later, the volcano's caldera began to spit magma and bubble with gas, so that morning the old hermit walked outside and changed his sign: "99.99% accurate"...

That is why it is important not only to consider the accuracy, but also the information gain of a prediction.

How are you calculating and using this information?

> base rate reasoning using the past 99 years of data would suggest that the probability is 0

Looking at edge cases is good for sanity checking, so it's a good habit, and I commend you.

In your example, though, we can also consider the base rate of an event which hasn't happened in 99 years as 1/101, per Laplace's rule of succession. https://en.m.wikipedia.org/wiki/Rule_of_succession

This is a great application of bayesian approach.

> base rate reasoning using the past 99 years of data

Like many things, if you do it badly it doesn’t work. If you had that little data, you’d look at the rate across many similar volcanoes.

Focusing on base rate makes you more effective than others because people tend to only focus on the delta from the base rate. Tensions are “elevated”. Ok, elevated from what? People don’t actually ask that. They pull a number out of thin air and double it.

If you intentionally consider “what is the base rate?” and “how is this different from the base case?” you empirically end up with better results.

You’re also not married to the base rate. If you think a factor makes the odds 100x higher, go for it. You just have to say that explicitly.

I think you mean ~1%, not ~10%. I’m not sure that the rest of what you describe is really what is meant by “base rate”. I think you’ve mixed a few concepts together:

- if you estimate the probability based on bad data, you get a bad answer.

- the base rate is a very simple model for the chance something happens – count similar events and divide by the number of potential similar events. One might describe it as an early prior before considering other information

- whether or not something (the eruption) happens in a year

- the probability one predicts for an event when considering more information. For example with the ‘pressure building’ model of the eruption you might decrease the probability immediately after an eruption and increase if it’s been a while (and increase a lot if smoke is coming out)

- sure it’s easy to be right predicting that unlikely events won’t happen. I think the claim of the OP is more that one may hope that the good prediction record transfers to the prediction of unlikely events

I think it's important to call out that 'base rate' means very different things to people who have taken an introductory Stochastics class. When you can bring in the memorylessness property of the exponential distribution, Poisson distributions and gamma distributions you can get some non-intuitive results.

I think the article focused on base rates because they're a relatively unusual and legible "trick" to coming up with a forecast, but really they're only one element of a forecast; typically a forecaster will think about many different ways to "attack" a question and synthesize them (somehow!). Choice of denominator for your base rate is very important also and can radically change the answer you get.

The sites which host these forecasting competitions correct for the bias against rare events through what's called "proper scoring" rules -- there's some specific maths to it, but the short version is that you're exponentially rewarded for being a correct contrarian and exponentially punished for being confidently wrong.

There are limits to that too, of course -- the folks in the article will "only" have made on the order of mid hundreds to low thousands of predictions, so roughly speaking, you can expect these people to be calibrated for 1% or 0.5% odds but probably not 0.1% odds.

Base rates work pretty well, at least for all cause mortality... hurricane counts per year, and financial markets (over very short time periods). I was using the Good Judgment project as motivation to practice R programming for a while, until one day I saw that literally EVERY person forecasting the ending value of the Hang Seng index tied to my probabilities. Therefore, EVERYONE was calculating base cases from historical market data and entering those results.

> if you had a volcano that erupts roughly ever 100 years, base rate reasoning using the past 99 years of data would suggest that the probability is 0

Sparsity is a problem whenever you use data to predict or model something. Your example here is essentially subsampling only zeros from a sparse time series. The existence of sparsity isn't a new insight that invalidates everything. It's a challenge to be overcome by careful practice.

> when in reality your base rate in the year following an eruption is 0 with every passing year your probability of an eruption would increase & increase past 10% for every year past 100 that goes without explosion.

Sure, with large amounts of prior knowledge, you can do better than a naive base rate starting point. I'm sure that practitioners know this. Even in this contrived example, the base rate would have been a good first guess.

> predicting a rare event doesn't happen isn't that difficult

Doing it accurately (in the sense of having a low Brier score) is apparently difficult.

Isn’t nuclear war specifically a bad thing to try and make predictions about? The cheat is to always predict no.

It seems unlikely that the U.K. is getting nuked alone, they have a bunch of nuclear armed allies. Even if you don’t believe in quantum immortality, if you are predicting ‘yes,’ you only get to collect your points in the ‘yes, and I survive, and so does the betting market’ case.

Cute zeroth order approximation, but you can also shift consumption forward if you think a nuclear war is near, or devote more resources to preventing it if you think the probability is higher.

OT, but are you talking about a specific situation here?

> something like war where pressures build up and war becomes more likely rather than less.

The likelihood of war between the UK and US, for example, has not steadily increased.

RAND has been in the consensus prediction business since .. well.. RAND was a thing.

This is pretty much normal. It's just that it carries on under the surface, most people are unaware where "sources state that there is a 20% chance of .." comments from the military-strategic orbit come from.

These % figures aren't very meaningful. I personally think they are mis-read more than they are understood. I would not cross the road on a 1 in 10 chance of death. I undertake activity which at least formally has a 1 in 100 chance of negative outcome frequently, and most of us do 1 in 1000 or upward without thinking about it.

The Anti-Vax Phobics were rioting on a 1 in 100,000 risk issue.

I tend to think China-Taiwan by 2030 is in the 5% bucket as well for a number of reasons. Principally, the join over news stating the Chinese Military is massively corrupt, and there are generals weeding out over-claimed capacity, the lack of visible improvement in their naval fleet beyond one Aircraft carrier, and the massive bright red wave of blood which will stem from an opposed landing from sea, in modern warfare. Blood is a remarkable thing, in terms of it's influence on the people at large. China is not Russia, they are nothing like as fatalist about their sons and daughters. An occupation of Taiwan will incur massive loss in the generation which is the one-child policy outcome: Many chinese families will lose their investment in the future. Not to mention the missiles coming back the other way will make this invasion a weapon with two sides, one of which faces coastal investment on the mainland.

That and the highly internally facing quality of the rhetoric: Talking up Taiwan is seasonal politics as people jockey for control of the structures. It's not preparatory to invasion.

>> China is not Russia, they are nothing like as fatalist about their sons and daughters

What is your evidence for this claim? How many people needlessly died due to Chinese communism? Maybe 50,000,000?

All sacrificed to an ideology. What has happened before can happen again.

Also, at least the histories I've read say the Chinese Communists used 'human wave' tactics in the Korean War.

> the lack of visible improvement in their naval fleet beyond one Aircraft carrier

Their fleet is expanding rapidly. Usually the discussion from experts is about how quickly China's naval power is growing. China happens to have two carriers now, and one more under construction.

Regarding aircraft carriers:

China doesn't need an aircraft carrier for Taiwan, which is ~100 miles off the mainland; air force bases on the mainland work much better.

Also, the relevance of aircraft carriers to high-end warfare is now in doubt: Accurate anti-ship missiles can hit carriers at much greater range than carrier planes can attack. China in particular has spent decades building a military that keeps US carriers too far from Taiwan to join the fight.

Generally, carriers may only be good for symbols of power (perhaps China's purpose for building a couple) and against less capable enemies - the Houthi's, for example, not China. They also enable you to have an airbase without local permission - if the US wants to bomb Somalia (as they currently are doing), and nobody nearby wants to be involved and let the US use their airbases, the US can put a carrier in international waters. It's a floating, mobile bit of domestic territory.

> That and the highly internally facing quality of the rhetoric: Talking up Taiwan is seasonal politics as people jockey for control of the structures. It's not preparatory to invasion.

Such talk can create a movement that takes on a life of its own; wars have been fought because someone inflamed the population and the leaders had no choice (or, being human, were inflamed themselves). One way to look at it is that such talk increases public support for war.

Closer to 1/5000 for the vaccine thing, depending on the sub-population.

This is what drives me mad about this kind of article:

> The team had an answer, and it’s an answer that goes some way toward explaining why this group has managed to get so good at predicting the future.

Then a whole section and a half of stuff I don't care about to get to the answer:

> there was broad agreement that the base rate of war — between China and Taiwan or just between countries in general — is not very high

I feel like that sometimes too, and I get especially annoyed when I think it is being done to to make me scroll through more ad impressions. I don't think that is the case with this one though. Writers are creative, and they want people to enjoy their writing on a level that is more than just digesting the pertinent facts. My guess is that this author thought they'd whet our appetite enough that we'd want to read the rest of the article. The rest of the article IS interesting but IMO kind of a non-sequitur to the main premise and to your point, gets in the way.

At least half of the article was a synopsis of Tetlock's books. So as a summary, it was okay. If it gets people to go and read the real research though, it's worthwhile.

That reads like a puff piece. There should be a list of correct predictions at the very beginning

For people who are religious, how do you factor in an external entity (omnipotent, benevolent in nature, cares about guiding humanity's karma, etc.) weighting the odds of certain events?

This is something I've been thinking deeply about for a while now. Curious to hear other people's thoughts.

I mean there are a huge number of viewpoints about this kind of thing even within a single religion like, for example Christianity. The long running divide between Calvinists and Armenians, for example. Calvinists say we don't have free will that God is pulling all the strings, Armenians say that God doesn't go against human free will. I have to say I'm more on the Armenian side - God isn't orchestrating events so much, we humans are given a large amount of latitude.

... but I guess I'm not following how this question is related to the article? They're not claiming to use some kind of supernatural means to predict the future.

If you really wanted to accurately predict the future, wouldn't you need to take into account the omnipotent external entity's opinion on things?

Opinion, no. Interventions yes.

And even the latter only if you mainly wanted to predict events where said intervention will occur and be a major factor (e.g. predicting an earthquake doesn't really need to take into account that someone will be spared from dying there by divine intervention).

Just Armenians? Don't Catholic and Eastern Orthodox in general believe the same? (Besides, iirc, Armenians are part of the wider Eastern Orthodox faith).

Yeah, I misspelled it. Arminianism named for Arminius. Yes, Catholics and Orthodox also fall into that category. It has nothing to do with any ethnic or geographic group.

> but I guess I'm not following how this question is related to the article? They're not claiming to use some kind of supernatural means to predict the future.

Probably because they don't think it exists.

Doesn’t the Abrahamic god specifically want to be impossible to detect? If you can get a god to influence the dice, you can easily come up with an experiment where you detect that it exists.

>Doesn’t the Abrahamic god specifically want to be impossible to detect?

No. Where does this idea come from? And certainly not in all denominations, including not in some major ones.

In fact (assuming you go but their stories) He often makes his present very clear and detectable.

“My thoughts are nothing like your thoughts,” says the LORD. “And my ways are far beyond anything you could imagine. For just as the heavens are higher than the earth, so my ways are higher than your ways and my thoughts higher than your thoughts.

That's not about being "impossible to detect".

That is about being superior and incomprehensible.

If there were some objective way to detect a god, somebody would describe it, and that would quickly become the only religion. Since most gods are very powerful, we can only suppose they are hiding on purpose.

Something appears to have changed after all the stories were written down.

The "base rate" argument is a rather odd one to make. The base rate of war was hasn't changed much from 2022, that didn't stop Ukraine from happening. Neither did it stop people from calling it. (It certainly didn't stop Samotsvety, who made that call correctly as well)

And given that each year sees ~50[1] new armed conflicts (for a total of 195 countries total) I'd posit that the base rate significantly exceeds their estimate of 22%.

Yes, of course, a war between China & Taiwan is not just like any war, and there are significant characteristics that influence the likelihood that aren't present in other wars, but at that point, talking of a base rate is just misleading.

And when you then "average" the "base rate" of hand picked special cases like "how often did control of Taiwan change", you're waving your hands. Wildly.

Which means the point isn't the idea of base rates per se, but choosing the right base rates - or answering the question "what is relevant here", and then analyzing from there.

The other success factor for forecasting orgs is choosing what you forecast. As such, I have questions about the actual forecasts made being locked behind a paywall: https://samotsvety.org/track-record/ (It doesn't mean they're not good at forecasting, but it's hard to judge how good)

[1] https://ourworldindata.org/conflict-measures-how-do-research...

It really is silly.

> One was the rate at which provinces claimed by China (like Hong Kong, Macau, and Tibet) have eventually been absorbed, peacefully or by force; another was how often control of Taiwan has changed over the last few hundred years (twice; once when Japan took over from the Qing Empire in 1895 and once when the Chinese Nationalists did in 1945); the third base rate used Laplace’s rule. Laplace’s rule states that the probability of something that hasn’t happened before happening is 1 divided by N+2, where N is the number of times it hasn’t happened in the past. So the odds of the People’s Republic of China invading Taiwan this year is 1 divided by 75 (the number of years since 1949 when this has not happened) plus 2, or 1/77, or 1.3 percent.

The first rate is maybe since 1949 (or maybe 1841 or 1557?), the second is over an arbitrary number of centuries, and the last is also since 1949, but could be calculated wildly differently if rephrased as "years mainland Chinese government has control over Taiwan" and extended back to the Qing dynasty. Somehow these are compatible to do an average of.

Then "He nudged up to 12 percent." Not that estimated percent chances can ever really be "wrong".

It seems sensible enough. It's not being presented as the One Weird Trick that lets you see the future, just as something simple and obvious that (implication: bizarrely) other people who try to predict the future often don't do. So even by just doing very basic things like that, you can end up ahead. Not right, not even most of the time, just more right than other people.

We might think that this seems rather obvious, and that's probably true. Samotsvety are being compared to people like IC analysts, politicians and pundits. Not people exactly famous for their firm commitment to rationalism and accountability. If you compared their track record vs top hedge funds it might look a little different, but of course hedge funds don't try to estimate the probabilities of things like a nuke hitting London.

I assume the people who are actually the best at predicting world events are making huge sums of money in the stock market. Or posting on /pol on 4chan. Not entering prediction competitions.

This assumes that prediction of world events tracks stock performance well. Predicting the performance of stocks requires also predicting a lot of the noise in factors like the error in prediction made by other investors, what marketing moves are made by the people who have an interest in the company looking good in a given quarter, the degree to which companies will be manipulated by large-scale investors for reasons orthogonal to that companies ostensible business models... etc. Some quantitative finance algos obviously do well, but it's a different skill from predicting the coarser phenomena we call "world events".

Also, if this all basically sounds like the economy is run like a mob casino... yes

plenty of people in the stock market make money when they're wrong and lose money when they're right. it's only a game of predicting the future if you're talking about prices.

> I assume the people who are actually the best at predicting world events are making huge sums of money in the stock market. Or posting on /pol on 4chan.

Why assume that those would be people's highest priorities? If I had some superpower, I would not spend my time using it for either of those things.

How is this different from the scam where someone sends out stock tips for different companies to different people? The ones that were successful they send out another round of random stock tips. Some of those will be right. Then they can target the people they sent the right ones to and say, look we predicted these so we are successful.

Basically, if you have many people predicting something, just from randomness some will be more successful and some won't be.

You could watch Samotsvety to see if they continue getting lucky in the future.

Xi Jinping is on the record, saying they are taking Taiwan. https://www.nbcnews.com/news/china/xi-warned-biden-summit-be...

The only questions are when and how and what will the rest of the world will do about it.

While I think this is really interesting, I'm not sure how useful it is. The world doesn't change because something may have a 5% chance of happening; it changes because it happened anyway. As a general rule, the most predictable outcomes have the lowest impact.

Vested interest skews your opinions more than if you don't care either way. Who knew?

If I wanted to try my hand at this hobby, what’s the best place to get started?

Read Superforecasters. Look at the two linked-from-article forecasting sites. Look up Adam Braff's blog - he runs an annual forecasting contest. Enter prediction markets. Read up about Brier scores. Try, fail, try and learn.

Kalshi, Metaculus.

Most superforecasters are retired academics or quants. I wouldn't recommend the investment in time that it would require.

But what would be your odds of success? ;)

Good Judgment Open, Metaculus

Forecasting tournaments: https://www.metaculus.com/ https://www.infer-pub.com/

Forecasting your own life: https://fatebook.io/

Training your calibration: https://programs.clearerthinking.org/calibrate_your_judgment... https://www.quantifiedintuitions.org/calibration

Books to read: Superforecasting, Scout Mindset

Sorry, is there a TLDR? I found the article difficult to follow in their main argument. Where can we compete with this team? Who and how will evaluate our results?

https://www.infer-pub.com

It is meaningless to put any number on an event like this. Unless you can find the very large number of universes where that same situation arises, you can never know if you were right.

Maybe a prediction market-based approach can weigh the beliefs of people about the event, but that doesn't mean it's a real probability.

Probability theory has given people the mistaken belief that everything is like a card or dice game. It isn't.

https://en.wikipedia.org/wiki/Dutch_book

https://en.wikipedia.org/wiki/Von_Neumann%E2%80%93Morgenster...

https://en.wikipedia.org/wiki/Bayesian_inference

Maybe everything's not a card game, but you really don't want to find yourself in a situation that isn't.

tl;dr

Bayes'

Nassim Taleb just published something interesting, about 'What do you do when you must estimate a probability and there is ZERO information?' https://fooledbyrandomness.com/blog/index.php/2021/09/07/est...

What is the point of forecasting events? The groups who need to act on the events will ignore the forecast or use it to self delude themselves into not acting, others will prepare for events by ignoring the forecasting.

Ukraine for example, prepared itself despite forecasts of low probability of an attack. Israel's military intelligence on the other hand knew the details of Hamas attack to the point and chose to ignore it despite high probability of the event.

I'd answer this from two perspectives.

As a forecaster: It's fun! It's an interesting way to learn about the world -- rather than gathering inert facts, you're forced to integrate them into a mental model, and then your model is tested for its validity empirically. It's also difficult and competitive, if you like that sort of thing.

As a consumer of forecasts: There's good research that prediction markets and other forecast aggregators are the best technology we have as a society for quantifying uncertainty. Not everyone will listen to them, just like not everyone eats their veggies, but c'est la vie.

I don't think there's too much risk of self-defeating forecasts (where a low forecast lulls decision makers into a false sense of security) - at least not yet. They're still pretty niche.

> The groups who need to act on the events will ignore the forecast

I know pretty surely that for example people in respective areas of finance are pretty interested in forecasts and will often act on them.

Forecasts are specifically important in hedging macro risk exposures as well as in black swan event long tail strategies.

It’s hard to believe after the Crimea annexation anyone forecasted a low likelihood of invasion. In fact my understanding (limited) of the forecasting community consensus is that Ukraine and other neighboring Russian former Soviet nations with strategic access to ports have been considered high risk of invasion for some time.

I know the US government has depended on forecasting in intelligence and defense for a long time built on the RAND legacy. I find it hard to believe most other sophisticated nations don’t also do similar forecasting and planning around forecasts given how influential these efforts in the US are.

Reply on news.ycombinator.com