Comment by RugnirViking | Hacker News Reader

Comment by RugnirViking | Hacker News Reader

This is very interesting. A discussion of the process of estimating and predicting future events, specifically going into their focus on "base rates". Apparently they are significantly better than other groups that enter most predicting competitions. (who knew there was such a thing?)

I particularly liked the idea of base rates and averaging them. [1] I can see here that the idea of them is not to be on their own accurate (some of the methodologies individually are dumb), but do get an idea for the scales of numbers to talk about (the difference between a 5% event and a 10% event is very hard to notice as a human)

[1] "One was the rate at which provinces claimed by China (like Hong Kong, Macau, and Tibet) have eventually been absorbed, peacefully or by force; another was how often control of Taiwan has changed over the last few hundred years (twice; once when Japan took over from the Qing Empire in 1895 and once when the Chinese Nationalists did in 1945); the third base rate used Laplace’s rule. Laplace’s rule states that the probability of something that hasn’t happened before happening is 1 divided by N+2, where N is the number of times it hasn’t happened in the past. So the odds of the People’s Republic of China invading Taiwan this year is 1 divided by 75 (the number of years since 1949 when this has not happened) plus 2, or 1/77, or 1.3 percent.

Sempere averaged his three base rates to get his initial prediction: 8 percent"

Thanks!

Out of interest - why don't you guys do way more of this for financial markets?

Presumably you don’t publish your strats there.

I'm not sure it's quite the same thing though. It's not a strat, it's a judgement call. And, they can put a bet on at time t and then publish at time t + 1 min. This is done by hedge funds, they do publish/pitch their ideas sometimes.

Yeah, good question. A few years ago the answer may have been that we thought we could influence more money than we could make.

Fwiw, when I've tried it, it has tended to suck in a whole lot of my time.

For me, other projects seem more meaningful. E.g., right now I'm setting up this: https://alert-team.org, another member is behind https://theaidigest.org/progress-and-dangers, a third works at https://forecastingresearch.org/, etc. Maybe we're an idealistic bunch, but also maybe we're making a bad judgment call, and still yet it's possible we couldn't cut it.

> Maybe we're an idealistic bunch

I mean, it makes me quite happy that y'all apparently use your skills for wholesome, humanitarian things instead of just focusing on making $$$.

Aren't you just itching to see if you can do it against the hottest competition on the planet? Ie, and I mean this with respect - financial markets are the major leagues of prediction - it feels a little bit like you are playing in the minor leagues. And it's not just a chest beating exercise - you guys appear genuine and to want to push forward the science/art of prediction. But if you can't beat the current best, you aren't state of the art, let alone able to push things forward. You might be pushing things backwards.

fwiw, it appears you'd have a decent chance. You guys seem to have an extreme level of objectivity and rationality. There is a lot of domain knowledge required to bet on say, healthcare stocks or whatever, but maybe too much domain knowledge harms rather than hurts.

this

Naively, if any of the predictions could be made +EV accounting for time preference, the boffins' predictions should have been mirrored into the market to someone's profit.

I remain unconvinced that they're not merely very lucky, the same way that some hedge fund operators or day traders are lucky.

Has your group made predictions on nuclear fusion timelines, or mean temperature rise due to climate change?

Eh, the second part to me seems like a less useful forecasting. Generally it seems fore asting is focused on human/political events that obviously don't already have useful modelling.

For climate change, there's already a lot of existing models out there calculated by much more rigorous metrics than "the Laplace rule" or just naively adding together averages.

Climate models just give us scenarios. Conceptually they give us the function f(emissions) = warming + error. But we don't know the probability distribution of emissions, so we don't know the probability distribution of f. Coming up with a reliable probability distribution requires considering things like politics, world events, economic growth, technology breakthoughs, etc. IPCC etc make attempts to model the distribution of future emissions, but it's less rigorous and scientific than the actual underlying climate model.

> the probability distribution of emissions

We know that every species consumes all available resources until an external force, whether it's predators or environmental collapse checks its growth. Humans are no different - any time a poor person uses a paper straw, a rich one will fly their private jet. Muh backyard arguments on nuclear means we will continue to burn coal and gas as fast as possible, even while renewables help drive up more demand.

So I think it's safe to assume that emissions will continue growing.

I'm hoping for a rigorous analysis. For example, X := "arrival time of fusion with a cost lower than natural gas in $ per kWh" is a random variable with a distribution. Our assumptions about the distribution of X will impact our estimated distribution of emissions, and hence impact our estimated distribution of f(emissions). There's lots of these questions that contribute, both technological and behavioral in nature.

Yes there's a lot of existing models, and a lot of the existing models have been wrong in the past. Climate science has always been very politicized. I for one would welcome predictions from people who have a track record of predicting things correctly. Climate science has a mixed track record [0].

[0]: https://cei.org/blog/wrong-again-50-years-of-failed-eco-poca...

FWIW, I'm very much NOT endorsing cei.org, nor saying climate change is not a big problem. Just saying that the track record of climate scientists is not great. Climate change is important enough to warrant the attention of people who can actually predict things.

Honestly, that sounds completely arbitrary, especially the one about 1/75. It also ignores anything that would alter any thinking person's probability calculations in the real world. For example, if tomorrow the Chinese premier says that Taiwan belongs to China and starts amassing navies, the % wouldn't change. I'd personally much _much_ rather listen to a view from some scholar or expert in the geopolitical situation than these "base rate" calculations.

It's supposed to be a completely uninformed prior. The only information, as you can see, is introduced in the formula with the above explanation.

Forecasting in general is a silly affair. As a data scientist I get all kinds of requests for all kinds of predictions, and my only real job is to part those requesting parties from their money.

The Chinese government has been saying that Taiwan belongs to China and amassed navies every year for almost 80 years now. (Longer if you count previous governments.) Base rates are important because what you think is a highly significant trigger event that will surely have major consequences may actually be a frequent occurrence that is typically inconsequential.

Or in other words, if you want to say "this time is different" and deviate from the base rate, you actually need to look at the previous times to find out whether it really is different this time, and if so, by how much.

Ok, you can adjust what geopolitical event we think it's significant. Again, since I'm not an expert in the matter, I'm likely to get it wrong, as it seems I did here :) But my point stands, I'd trust expert opinion over these simplistic, falsely confident quantitative approaches.

The percentages of democratic citizens assimilated into china would reach quiet a interesting percentage with taiwan assmiliated into the mainland. You can censor and propagandize all you want, but people who travelled or know better, just wont blieve a word of it happily ever after. One wonders what the internal "knowing" citizen rate is by now in china.

The party relies on divide and conquer of uniformed fools. But the fools are in the minority by now - and the divide and conquer is one sneaky trustfall decentralized communication platform away from never working again.

I think the idea is that instead of trying to argue, just be empirical. New information would have to be added along with context on how that information has impacted odds in the past

If the information is entirely new (ie, what impact does aliens taking over the Moon have on China/Taiwan) then there'll always be difficulty estimating that impact

This is just the base rate, not the final prediction. Your actual prediction can incorporate any information you want, but the base rate gives you a starting point and a sanity check.

Also, as the article notes, scholars like Tetlock have studied the the track record of scholars and experts and found them to be less accurate than this type of approach.

No, Tetlock et all have found very large crowds to outperform smaller groups of experts. This however is an article about small groups of non experts competing against each other. It’s not even blending the predictions across teams.

It’s a literary sleight of hand but useful to note as it undermines the entire premise of the article. That’s because these predictions are bunk because these techniques don’t work with the stock market which uses far more rigorous statistical methods for pricing (which happened 60s-90s with the rise of quants).

That's a good point. Again, even as a base rate I wonder if it's any good as it seems extremely hand wavy and arbitrary, but I can see how it being part of a multi-source approach can help.

There are comments similar to this all throughout the thread. It reminds me of the movie Moneyball. Even when the numbers are clear to see, people will still much rather follow their intuition. The Oakland Athletics' insistence on sticking to the numbers anyway is what led to their enormous success, and now the same strategy is copied by sports teams around the world.

It can be hard to convince people with math alone.

I'm onboard with trusting data driven and numbers approach. This one just seems like a bad one if it doesn't account for how obvious actions would alter its results. But as some other comment said, maybe this base rate is not the whole story and it's just one ingredient in the total risk calculation. That makes more sense.

Reply on news.ycombinator.com