Heuristics that almost always work | Hacker News Reader

Heuristics that almost always work

astralcodexten.substack.com

1,000 pointsmudro_zboris520 comments

This story reminded me of a story written by a Czech biologist who studied animals in Papua-New Guinea and went to a hunt with a group of local tribesmen.

The dusk was approaching, they were still in the forest and he proposed that they could sleep under a tree. The hunters were adamant in their refusal: no, this is dangerous, a tree might fall on you in your sleep and kill you. He relented, but silently considered them irrational, given that his assessment of a chance of a tree falling on you overnight was less then 1:5000.

Only later did he realize that for a lifelong hunter, 1:5000 are pretty bad odds that translate to a very significant probability of getting killed over a 30-40 year long hunting career.

It's a great example. This is the very reason I have scaled back the amount of time I rock climb as I've gotten older -- not because any individual outing is dangerous, but there's an element of Russian roulette wherein the mere act of doing it more often dramatically changes the risk.

Also known as the Kelly criterion. If one possible outcome of an action is associated with a great enough loss, it doesn't make sense to perform the action no matter how unlikely the loss.

No, Kelly is about what fraction of your bankroll you should bet if you want to maximize your rate of return for a bet with variable odds.

It's essential if you want to:

* make money by counting cards at Blackjack (the odds are a function of how many 10 cards are left in the deck)

* make money at the racetrack with a system like this https://www.amazon.com/Dr-Beat-Racetrack-William-Ziemba/dp/0...

* turn a predictive model for financial prices into a profitable trading system

In the case where the bet loses money you can interpret Kelly as either "the only way to win is not to play" or "bet it all on Red exactly once and walk away " depending on how you take the limit.

That is a much narrower view of the Kelly criterion than the general concept.

The general idea is about choosing an action that maximises the expected logarithm of the result.

In practise this means, among other things, not choosing an action that gets you close to "ruin", however you choose to measure the result. Another way to phrase it is that the Kelly criterion leads to actions that avoid large losses.

Actually

https://en.wikipedia.org/wiki/Kelly_criterion

"The Kelly bet size is found by maximizing the expected value of the logarithm of wealth, which is equivalent to maximizing the expected geometric growth rate"

In real life people often choose to make bets smaller than the Kelley bet. Part of that is that even if you have a good model there are still "unknown unknowns" that will make your model wrong some of the time. Also most people aren't comfortable with the sharp ups and downs and probability of ruin you have with Kelley.

I've long found that Wikipedia article woefully lacking in generality.

1) The Kelly criterion is a general decision rule not limited to bet sizing. Bet sizing is just a special case where you're choosing between actions that correspond to different bet sizes. The Kelly criterion works very well also for other actions, like whether to pursue project A or B, whether to get insurance or not, and indeed whether to sleep under a tree or on a rock.

2) The Kelly criterion is not limited to what people would ordinarily think of as "wealth". It applies just as well to anything you can measure with some sort of utility where compounding makes sense.

The best overview I've found so far is The Kelly Capital Growth Investment Criterion[1], which unfortunately is a thick collection of peer-reviewed science, so it's very detailed and heavy on the maths, too.

[1]: https://www.amazon.com/KELLY-CAPITAL-GROWTH-INVESTMENT-CRITE...

Isn't that called Pascal's Wager?

Pascal's Wager is a fallacy resulting from plugging infinity into your risk analysis and interpreting ∞/0 in a way that suits you.

Pascal's wager is an example of motivated thinking - there were very real and certain consequences to him if his wager didn't demonstrate you should obey the Catholic Church.

Which of course directly leads to Pascal's Mugging: I can simply say "I'm a god, give me $10000 or you will burn in hell for all eternity". Now if you follow Pascal's Wager or GP's logic you have to give me the money: I'm probably lying, but the potential downside is too great to risk upsetting me.

There's actually a rational explanation for that: humans don't care very much about burning in hell for all eternity, when it comes down to it.

There's actually a similar though experiment that might seem even more bizarre: I could tell you "give me $100 or I will kill you tomorrow" and you probably wouldn't give me the $100. That's because when it comes down to it, humans don't see the loss of their life as that big a deal as one might think. It's a big deal, of course, but in combination with the low likelihood, still not big enough to forgo the $100.

Here's the thing: if you have just killed five people, "give me $100 or I will kill you tomorrow" becomes a much more effective threat.

One-time games and repeated games have different strategies.

It becomes more effective only because it changes the probability estimation of the outcome.

Life is a repeated game of decisions that compound on each other, so that difference is irrelevant.

If anyone wants to play around with an interactive explanation of the The Kelly criterion: https://explore.paulbutler.org/bet/

> the mere act of doing it more often dramatically changes the risk.

Kind of. However, you already know that the first N outings didn't have a disaster. So those should be discarded from your analysis.

Doing it N times more has a lot of risk, doing it the N+1th time has barely any.

The "slippery slope" principle applies here though: N+1 enables N+2, which enables N+3 and so on.

but the risk is independent. so once you do the N+1 time safely, you are back to N and your next time is _also_ just an N+1.

Continuing to do something regularly doesn't ever mean you're just going to do it once more.

Psychologically, behaving in a certain way makes it more likely that you'll behave in the same way in the future. That's an integral idea underpinning justice systems.

True but it would be incorrect to assume that you can safely keep basejumping every day in a year, just because you haven’t died in the last 50 days. Eventually the stats say you will be 87% likely to have an accident when you consider your choice at the beginning of the year. It might be day 20 or day 300, but you won’t know what case you end up in. The chance of your next jump being your last is always the same, but that doesn’t decrease the risk of repeated trials.

Not exactly. If you've done it 50 days without an accident, your current chances of the accident happening in the remainder of the year are NOW less than 87%.

If you've made it Jan 1 to July 1 months without an accident, the chances of you making it to Dec 31 are now better than they were on Jan 1 -- because now they are just the chances of you making it six months, not a year.

The chances of flipping 6 heads in a row are 1/64. But if I've already flipped 3 in a row... the chances of flipping three _more_ heads in a row is 1/8, the same as always for flipping 3 heads in a row. The ones that already happened don't effect your future chances.

I meant to say starting a new year after the 50 past days, I see that wasn’t clear though.

Risk is independent of prior events, habits are not - I think that was what the anthropologist story is about

The risk is independent but the marginal enjoyment isn't. You don't get double the satisfaction from climbing twice as much.

Yes, but when you make a plan to find an acceptable cumulative future risk, planning to do it once a week for the rest of your life is planning to expose yourself to significantly more risk than doing it twice a year for the rest of your life.

You might still die in one of the next 20 instances. But you've added a lot more not-dead time in between them!

Saying "I can do one more with minimal added risk" every single time after not dying is true and yet pointless, because it's not a given that "minimal added risk" = "not dying." It's survivorship bias to not think frequency doesn't affect the cumulative odds of your future planning solely because you've already done a lot of trials.

Slippery slope is a fallacy, not a principle. Just because you took N steps, that doesn't necessarily mean you will take N+1 steps.

It's a convincing fallacy because sometimes you do take N+1 steps. But just like in the article, heuristics aren't always right.

Slippery slope arguments aren't inherently fallacious. If you can justify one more climb on the grounds that probability of injury or death is very low then you will be able to justify every subsequent climb on the same basis.

Slippery slope arguments are inherently fallacies. They don't prove that something will happen.

Just because you can justify the next climb on the same basis, that doesn't mean you will. You could decide that you've already tested the odds one too many times.

"You could decide that you've already tested the odds one too many times" was the original point. Someone responded that the N previous times don't matter and N + 1 has barely any risk. Another poster countered that that argument as stated applies not just for N + 1 but for (N + 1) + 1 etc and therefore the slippery slope principle applies.

Of course if you add in "you could decide that you've already tested the odds one too many times" then it's a fallacy to invoke slippery slope because an off-ramp is explicitly specified. In this case slippery slope was mentioned only because N was dismissed as irrelevant.

Don't get on that greased sliding board that ends at the top of a cliff. Once you start sliding, it will be hard to stop because of the grease, and then once you slide off then end you will fall and die.

Do you really think this slippery slope argument is a fallacy? FWIW, wikipedia acknowledges slippery slope can be a legit argument when the slope, and it's chain of consequences, are actually real. https://en.m.wikipedia.org/wiki/Slippery_slope . Indeed, this is the very basis of mathematical induction.

From your linked article:

> The fallacious sense of "slippery slope" is often used synonymously with continuum fallacy, in that it ignores the possibility of middle ground and assumes a discrete transition from category A to category B. In this sense, it constitutes an informal fallacy.

"If you take N steps, you will take N+1 steps" is a fallacy whenever it's possible that you won't take N+1 steps.

Not what was said. What was said: if you DON'T take the Nth step, you then WON'T take the N+1 step.

"Not A -> Not B" is different logic than "A -> B". A is necessary but not sufficient for B.

>If you can justify one more …

Reminds me of Terry Pratchett quote "No excuses. No excuses at all. Once you had a good excuse, you opened the door to bad excuses.”

Full quote is fifth here: <https://www.goodreads.com/work/quotes/819104-thud>

When accounting for human psychology it does have validity: doing an enjoyable activity "one more time" has a risk of a habit forming, which has a non-zero probability. It is indeed possible.

The argument can certainly be used in a fallacious manner (e.g. by greatly exaggerating the probability of the further steps, saying they are inevitable if the first step is taken, etc.). It's logically valid to say that the first step enables subsequent steps to be taken.

Edit: I'd say that the slippery slope is perfectly valid rule of thumb in a lot of 'adversarial' situations. Once one side makes an error or fails somehow, the balance between the two sides can be disrupted leading to one 'side' gaining momentum. Just as between people, a similar 'adversarial' process can occur within the minds of individuals: between two ideas or patterns of thought/behaviour, one idea can gain momentum after a decision has been reached. Precedence is a strong force.

Like all fallacies, it's only a fallacy when it's fallacious.

Otherwise, it's just a regular d argument.

Maybe fallacies could be renamed "logical hazards" or something like that. Arguments that are at high risk of being false and require extra care, but not automatically false.

A pet peeve of mine is that the slippery slope fallacy can be defined as "modus ponens but wrong".

A fallacy should be a incorrect shape of an argument, a incorrect reasoning, not just a false statement.

If you'll die if a roll of three dice comes up sixes, you're not really in a lot of danger. If you do it every day, you have about 15 months to live.

>you have about 15 months to live

Or a few minutes ... or 20 years.

That's the thing w/ statistically independent trials.

That's like the difference between

You could win 100mm in the lottery (true statement!)

Lottery tickets are a good investment (almost always, false statement).

Planning on "well it could happen, technically" isn't a good approach.

But when looking at a possible positive outcome, such as the lottery case, it can "make sense" to buy one ticket.

Your chance of winning goes from No Chance to A Chance, which is an infinite improvement.

That's not how this works as a rational investment choice.

It's true that you can never win a lottery you don't enter, but the expected value of that ticket is vastly lower than what you paid for it. That means, as an investment, your $10 will be expected to do better in literally anything with a positive return.

If you are buying > $10 worth of dreaming (for you), fine - but that's consumption.

If you've already done it for 12 months without it happening though, the next 3 months are no more dangerous for you than for someone starting from scratch.

Very true. The only winning move is not to play!

That's true, but usually when we are deciding which actions to take, we're not comparing "I take actionA" versus "I take actionB," rather than comparing "I take actionA" versus "some random other person takes actionA."

OK, the next 3 months are no more dangerous for you than if you hadn't spent the last 12 months doing it. What you did in the past has no bearing on the chances going forward. I'm not sure if it's more clear to say it like that or not. Clearly, humans have a lot of trouble speaking and thinking clearly about statistics.

The next three months are no riskier than your first three months were. They don't become more risky because they will add up to 15 months total -- once you've already finished the first 12 without incident.

For the dice roll example that is true. But other examples it isn’t. For example the MTBF of a device that has run for x hours approaching the MTBF is probably more likely to fall in the next x hours. Or if there is some cyclic behavior. Like waiting outside for a hot day.

Funny book recommendation: "The Dice Man" by Luke Rhinehart.

Anyone who's rolled double natural 1s with advantage would never take this bet - and your example is twice as likely to occur!

If you think a 1/216 chance of sudden death isn't a lot of danger, I don't want to go rock climbing with you!

Don’t look at the actuarial tables. The odds are worse than that over a year’s time after ~35

What am I missing? 6 x 6 X 6 = 216 or about 7 months.

Dice (typically) do not have a memory, so whatever happened yesterday will not influence what happens today. If you roll it daily, your chance of surviving at least N days is (215/216)^N, for the specific case of "rolling three 6 on three 6-sided dice" that puts you at ~50% at 149 days and at ~10% at 496 days.

At sufficient scale, even incredibly unlikely things become quite probable.

    runs <- 10000
    x <- vector(mode = "numeric", length = runs)
    for (i in 1:runs){
      while (sum(sample(1:6, size = 3, replace = TRUE)) != 18){
        x[i] <- x[i] + 1
      }
    }

    summary(x)
    quantile(x, c(0.5, 0.8, 0.9)) 

    > summary(x)
       Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
        0.0    62.0   149.0   216.2   300.0  1902.0
    > quantile(x, c(0.5, 0.8, 0.9))
    50% 80% 90%
    149 350 495

A simple simulation. Run 10K times. Count the number of times it takes for three dice to add up 18.

The numbers very much agree with you. The median is 149. The 90th is 495 in the simulation, which is close enough to 496. There is very much a long tail in the data. So, the median and the average will not be the same. Is it a coincidence that mean is a 216?

Off the top of my head, I don't know. It MAY be related to the fact that 6*3 is 216, but I don't have deep enough statistics knowledge to say for sure. You coudl try it again with 3 8-sided dice and rolling 24, that should give you ~50% at 344 iterations, and ~90% at 1177 iterations. If my supposition that the mean is related to the possible rolls, then the mean should end up being 512.

Iteration counts gathered with Python and a (manual) binary search (actually faster than writing code).

    runs <- 100000
    x <- vector(mode = "numeric", length = runs)
    for (i in 1:runs){
      while (sum(sample(1:8, size = 3, replace = TRUE)) != 24){
        x[i] <- x[i] + 1
      }
    }

    summary(x)
     Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
      0.0   146.0   353.0   511.8   708.0  5112.0 

    quantile(x, c(0.5, 0.8, 0.9))
     50%  80%  90% 
     353  824 1187

Strangely enough the mean agrees. The other ntiles are off a bit, but that's randomness for you.

RandomSwede's comment is accurate, but maybe the below can help add some 'flesh' to their response.

Basically, the problem is that you can't just multiply it all together.

(1/6) ^ 3 is correct, and the probability of rolling 3 sixes is indeed 1/216 today, but if you repeat independent events, you don't just add up the probability.

Imagine instead of dice it's coins, and it's only two. Your odds of getting HH today are 1/4, but the odds of getting HH by day four are not now 4/4. We know that it's possible, although unlikely, you could flip coins for the rest of your life and NEVER get two heads. So we know that you can't ever have odds of 4/4 (or 1), only odds that approach 1. So that means that we can't say 216 days from now will be 216/216.

Instead, you need to work out the probability of the event NOT happening, and then repeatedly NOT happening independently (so we can multiply together to get the probability.

For our four coins, the probability of NOT getting HH is 3/4. On Day 2, the probability of NOT getting HH on both occasions will be (3/4)×(3/4), (9/16, 56.25%). By day 3, it will be (3/4) × (3/4) × (3/4), or 27/64. On day 4, it'll be 81/256, or 31.6%. Now we can subtract from 1, to work out that by day 4, the odds of us having hit HH are almost 70%.

As RandomSwede explains, there's a 50% chance that you will have rolled three sixes by day 149. By day 496, you're down to 10%.

    runs <- 10000
    x <- vector(mode = "numeric", length = runs)
    for (i in 1:runs){
      while (sum(sample(1:6, size = 3, replace = TRUE)) != 18){
        x[i] <- x[i] + 1
      }
    }

    summary(x)
    quantile(x, c(0.5, 0.8, 0.9)) 

    > summary(x)
       Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
        0.0    62.0   149.0   216.2   300.0  1902.0
    > quantile(x, c(0.5, 0.8, 0.9))
    50% 80% 90%
    149 350 495

A simple simulation. Run 10K times. Count the number of times it takes for three dice to add up 18.

No, I don't think this is a coincidence, but I'm not completely confident in saying that.

Thinking about it doesn't make me feel like I'm solving a maths problem. I start stacking ideas and concepts in a way which makes me feel like I'm overlaying them in a way which is incorrect.

It makes me feel like I'm solving a riddle, which hints to me that maybe it's actually a question of semantics and definitions rather than a maths problem.

The expected value will be 6³ = 216 days or about 7 months. Where do you get the factor of two from?

Also, “not really in a lot of danger”? Those odds are worse than that of a 100 year old in the USA (they have a life expectancy of over two years)

Certainly, as an additional risk, it’s high.

You forget: once you roll three 6s in a row, you're dead, and you don't roll any more. Your expected calculation assumes that people keep rolling after they get 666.

Though I'm not sure where they got their figure from, because there isn't an “expected time to live”; there's a 90% probability to live time, a 5% probability to live time…

There’s a difference between expected value of number of days you’ll survive and the number of days a given fraction of the subjects will survive, but I don’t see either supporting the claim “If you do it every day, you have about 15 months to live”.

  (215/216)^450 ≈ 0.124

, so about one in eight will survive for 15 months or more. The “5% probability to live” time is around day 645 (about 1¾ years):

  (215/216)^645 ≈ 0.0501

the “half will survive at least for” point is around 5 months:

  (215/216)^149 ≈ 0.501

Yup. Don't do stuff (repeatedly) that have an absorbing barrier - https://medium.com/ml-everything/nassim-taleb-absorbent-barr...

The parent comment talks about scaling back the amount of rock climbing they do in order to reduce risk.. And now you are saying that they should go one more time, because a single climb is low risk?

Yes. I am saying their analysis of risk is incorrect, and therefore if that's the only reason they aren't climbing then they should climb more often.

I disagree, their analysis is perfectly correct.

The more frequently you take a risk, the greater the chance that risk materialises.

Parent wants to lower their overall risk, but doesn't want to stop climbing entirely. So they climb less often.

I think you're reading it wrong.

After a long life of rock climbing, there's no significant risk of doing it one last time or 10 last times (ignoring the effect of old age itself and whatever).

But when you're in earlier stages of your life, you're asking a different question: You're asking, is this something I want to do hundreds or thousands of times in my life, knowing that each of those times has a small chance of ending my life? This becomes a completely different question.

If I'm 35, maybe I will climb 30 times per year on average for 30 years until I'm 65. That's 900 climbs in total. If my goal is to not die or experience serious injury from rock climbing even once in my life, I have to consider the chance that any one of those 900 climbs will result in serious injury or death. I don't know the numbers for the risks involved, but it seems reasonable to be cautious.

Maybe I don't want to give up on rock climbing altogether, but maybe I can scale it back. If I limit myself to 1 climb per year, that's 30 climbs in total. Much lower risk than with 900 climbs.

This is not a logical fallacy.

That would be a reason to have not climbed more than a specific rate ever. It wouldn't be a reason to scale down the rate of climbing as you age.

The assumption is that it's desirable to have a descending climbing frequency instead of uniform.

This makes a lot of sense, as when you're younger frequent climbing would help you to develop proficiency quickly and your body allows you to joy it fully. Plus the social benefits are probably higher when younger.

Once you're older, it's potentially less enjoyable (as your body ages) and you don't need to worry as much about rapidly gaining proficiency.

You're making it sound like it's a decision they made when they got into rock climbing initially, that they would climb frequently while young and then scale back as they get older.

Now, making that decision at the outset does make sense, because it will drastically reduce the number of climbs you make in your life compared to climbing frequently throughout your life, and rock climbing while young is less risky than rock climbing while old.

But importantly, I don't think that's what GP did. It sounds to me like GP spent their youth climbing a lot without considering their mortality, but then decided to scale back because they realized climbing that often for the rest of their life would be dangerous. Maybe they spent the time from 20 to 35 climbing 30 times per year, in keeping with my earlier example. That means they've already climbed 450 times. Risky, but they made it through alive. At 35, they start to consider their own mortality, and they have the choice between climbing 900 more times by keeping to their current rate, and climbing 30 more times by reducing their rate (or something in between). Deciding to scale back makes sense.

There is no logical fallacy.

I think what you're missing is that they are not avoiding "going rock climbing one more time"; they are avoiding "being a person who habitually rock climbs", because while each excursion is low-risk the aggregate effect will be high risk. It's like smoking -- one cigarette won't appreciably impact your health, but "being a smoker" will.

None of this intended to cast aspersions on rock climbing in particular, just pointing out that a reasonable person, understanding independence of events and not falling prey to any fallacy, could reasonably make this decision based on their personal risk tolerance

Yes, or more accurately there is a frequency of climbing outings at which the marginal increase in satisfaction from an extra climbing is no longer sufficient to justify the increased risk.

I think it the argument is to make a lifetime risk assessment, opposed to an individual event risk assessment.

If your tolerance is X% death/life, you can calculate the climbing frequency that falls below the threshold.

On the plus side, if you assume the events are independent, you can recalculate and increase the frequency after each climb.

I don't know how you can make this claim objectively without knowing that individual's preferences.

If an individual decides their risk tolerance is that they will not accept a one in a million chance of injury from rock climbing, how is their analysis incorrect?

Under this assumption, by the principal of mathematical induction, you can easily do it K more times for any K without taking on barely any risk at each step of the way.

This is called the Turkey fallacy: the turkey was feed by humans for 1000 days, and after each feed event he updated his belief that humans care for him until it's now almost a statistical certainty.

Is this the reverse of the Gambler's Fallacy? Instead of "The numbers haven't hit in a while, therefore they're going to hit soon." it's "The numbers haven't hit yet, therefore they're never gonna hit."

Also known as complacency. Working in a woodshop, one of the things you are most vulnerable to is failing to respect the danger you're in. This is why many experienced woodworkers have been injured by e.g. a table saw - you stop being as careful after such long exposure.

A related thing is normalization of deviance. You start removing safety because you see nothing bad happened before, until you are at a point where almost no safety rules are respected anymore. You can see this a lot in construction videos.

Yup, complacency can kill you.

In this case [0], a skydiver forgot to put on his parachute...

https://reverentialramblings.com/2018/08/15/the-skydiver-who...

Wow, that's terrifying and a good cautionary tale.

Also, when I read

> I’m hoping you can you forgive me as a minister of religion for likening this story to a spiritual cautionary tale. Yes, we do need to live each day as if it might be our last.

I thought, "Hmm, sounds adventist", and sure enough :-)

Oh man, that's terrible. I can certainly understand how someone without a checklist that is verified by two people can do that, especially if you have a backpack on to mask the fact that the parachute is missing.

Many times if I wear a tight jacket in the car, I forget to put my seat belt on, because I unconsciously mistake the pressure of the jacket for the seatbelt's, even though putting on a seat belt is usually the first thing I do.

Poor guy.

I generally take off my jacket before driving for that very reason.

Luckily newer cars won't stop beeping if you forget your seatbelt, so the problem is mitigated. Not so for parachutes, apparently.

To be honest, I never wear a parachute when driving!

Better not drive close to cliffs then!

And why pilots traditionally work from checklists, even if they've done the process thousands of times.

That only applies if you are updating priors. In this case the odds are fixed, the GP is correct.

The odds of a rocking accident are known and fixed?

Probably not. But they aren't affected by the previous N climbs, at least as described by GP post. They are considering a fixed odds event, and the probability of (bad thing happens) over a sample path through time. That's not the turkey fallacy.

In other words , the difference between the turkey and the climber is the climber knows the odds (at least nominally) , and it’s important .

Surely sometime about the turkey getting fatter each time complicates this example.

hows yesterday's tree impacting today's?

All this reminds me of “if you are immortal and cannot be harmed, what are the odds of getting ‘stuck’?” I’d venture 100%.

In a skill-based game, N+1 has less incremental risk than adding 1 more trial with N-1 games did.

This assumes a lot about the underlying process, particularly independence. Whilst assuming independence might hold reasonably well for low numbers of samples, the assumption might be increasingly (and dangerously) misleading. The intuition expressed by GP captures that.

The point is that they're changing their habits. Of course we ignore the n times they've gone before, now instead of their habits meaning they'd go m more times in the future, they're going to be going p times in the future for some p that is much less than m.

So it's not about how often they've done it over their lifetime so far, but about how many times they will be doing it over the rest of their life.

> It's a great example. This is the very reason I have scaled back the amount of time I rock climb as I've gotten older -- not because any individual outing is dangerous, but there's an element of Russian roulette wherein the mere act of doing it more often dramatically changes the risk.

Indoor climbing, and especially bouldering, can be a lot of fun at the right gym, and with dramatically reduced risk of death (though injury is still a very real possibility, I say, recalling all the time I spent nursing my sprained ankle).

This is called ergodic theory, the more you repeat an action that could result in catastrophe, the likelihood of that catastrophe occuring will be close to 100% if the number of events is high enough.

But that doesn't imply any higher probability. Say chance of dying when doing alpinism is one in a thousand.

The 1000th time you go climbing the chances of dying are still 1/1000.

If you get 100 heads in a row, the 101th time you launch a coin the chance of getting heads is still 50%.

Right, but it's easy to conflate two very things here:

"What are my chances of dying in a climbing accident", and

"What are my chances of dying today if I go climbing".

If you are on a plane, you* have a lower risk of some kinds of cancer than the airline staff do. This has nothing to do with the flight you are both on, and everything to do with accumulated flights

"you*" = for most people, i.e. barring a counteracting risk factor.

This is exactly why people who do high risk think it will never happen to them.

"There are bold X, and old X, but no old, bold X."

Replace X with any practitioners subject to sufficient risk as a result of their practice.

I first heard it in the context of mushroom foraging.

Mountaineers is where I heard it.

The risks are highest when learners are at a beginner to intermediate stage. They know the basics, and have gained some confidence, but don't know enough to get themselves out of trouble.

This is called Stage 1 in the Gordon Model of learning: unconscious incompetence.

Also known as an advanced beginner, the stage before competency. Someone who has enough knowledge to, as they say, "be dangerous".

They call it the "intermediate syndrome" in freeflying.

While this is true, in the context of alpine climbing where I first heard this statement, the bold alpinists who die young are very much not beginner-intermediates. I've interpreted this differently than just the "Bathtub Curve"[1] applied to dangerous pursuits.

Rather, there is a certain amount of objective risk in alpine environments, and the more time you put yourself in that environment, especially in locations you aren't familiar with, the greater the chance that something will eventually go wrong.

I'm always surprised by the number of famous alpinists who weren't killed on their progressive, headline-capturing attempts but rather on training attempts and lesser objectives.

[1]: https://en.wikipedia.org/wiki/Bathtub_curve

My wife teaches people to ride horses for a living so we talk about the safety of that.

You hear a lot about people who get seriously injured riding who are often professionals or people who ride competitively at a high level. They are doing dangerous things and doing a lot of them.

We don't think it is that dangerous for people who ride at the level we do, out of maybe 15 years we've had one broken bone.

The other day I noticed that we had acquired a used horse blanket from another barn in the area which is a running joke at our barn because of their bad safety culture. They are a "better" barn than ours in that they are attached to the show circuit at a higher level than the bottom, but we are always hearing about crazy accidents that happen there. When I was learning to ride there they had a confusing situation almost like

https://aviation-safety.net/database/record.php?id=19810217-...

with too many lessons going on at once where I wound up going over a jump by accident after a "near miss" in which I almost did. (I never thought I could go over a jump and survive, as it was I had about two seconds to figure out that I had to trust the horse and hang on and I did alright...)

Another good allegory is that, in the US Air Force, the flight crews considered most dangerous are those with the highest collective rank. Sure, the young crews are learning but the old ones still think they know it all and have often forgotten critical details.

(Example) When you go climbing somewhere, you have like a 40% of getting killed that you can mitigate completely by skill, and an additional 0.1% chance that something goes wrong by some fluke, that you can’t mitigate at all.

Pretty good if you go climbing 10 times a year. Pretty bad if you go 1000 times.

Need to consider that headline capturing objectives ar a few in a lifetime and training goes on all the time.

Isn't this somewhat expected?

They wouldn't be famous if they didn't succeed on headline-capturing attempts and there are only so many you can realistically do in life. They are dead however as doing dangerous things often enough will kill a substantial number of practitioners.

No, the risks are greatest when you reach complacency. Beginners, even bold ones, take some care. You mostly see this in things like forklift drivers because it takes years of doing the same thing every day before you really get expert enough to be complacent

There is also something called "Normalization of Deviance", defined better by a quote: "Today, the term normalization of deviance — the gradual process by which the unacceptable becomes acceptable in the absence of adverse consequences — can be applied as legitimately to the human factors risks in airline operations as to the Challenger accident." *

Most of you have probably heard of it in the context of fighter pilots doing riskier and riskier maneuvers, but it seems to apply to drivers who speed a lot. 80 starts seeming really slow to them after doing it for years.

* https://flightsafety.org/asw-article/normalization-of-devian....

Thanks for posting these, I'd only seen Normalisation of Deviance mentioned in these two youtube videos by Mike Mullane and never thought to look any further:

https://www.youtube.com/watch?v=Ljzj9Msli5o

https://www.youtube.com/watch?v=jWxk5t4hFAg

and the uploader references some further links:

https://www.fireengineering.com/leadership/firefighter-safet...

https://www.flightsafetyaustralia.com/2017/05/safety-in-mind...

and references this book (about the Challenger Disaster):

https://www.amazon.com/gp/product/B011DAS53Y/

which has an overview here:

http://web.mit.edu/esd.83/www/notebook/The%20Challenger%20La...

including these two excerpts I found interesting in this context: "Chapter nine she explains how conformity to the rules, and the work culture, led to the disaster, and not the violation of any rules, as thought by many of the investigators. She concludes her book with a chapter on lessons learned."

"She mainly emphasizes on the long-term impact of institutionalization of the political pressure and economic factors, that results in a “culture of production”."

Vaughn's book The Challenger Launch Decision doesn't tell this truth: the root cause of the accident can be traced back a decade to the acceptance of a design that was "unsafe at any speed".

Every other manned space vehicle had an escape system. The crew of the Challenger was not killed by the failure of the SRB or the explosion of the external tank, but rather when the part of the orbiter they were in hit the ocean. They could have build this into a reinforced pod with parachutes or some other ability to land but they chose not to because they wanted to have the payload section in the rear.

In the case of Columbia it was the fragile thermal protection system that did the astronauts in. There was a lot of fear in the first few flights that the thermal tiles would get damaged and failed and once they thought they'd dodged that bullet they didn't worry about it so much.

"Normalization of deviance" was a formal process in the case of the space shuttle of there being meetings where people went through a list of a few hundred unacceptable situations that they convinced themselves they could accept, often by taking some mitigations.

When the design was finalized it was estimated that a loss of vehicle and crew would happen about 2%-3% of the the time which was about what we experienced. (Originally they planned to launch 50 missions a year which would have meant the continuous trauma of losing astronauts and replacing vehicles.)

It's easy to come to the conclusion that it was a particular scandal that one particular concern got dismissed during a "normalization of deviance" meeting but given a poorly designed vehicle it was inevitable that after making good calls for thousands of concerns there would be a critical bad call.

"Normalization of deviance" is frequently used for a phenomenon entirely different than what Vaughn is talking about, something informal that happens at the level of individuals and small groups. That is, the forklift operators who come to the conclusion it is OK to smoke pot at work, the surgeon who thinks it is OK to not wash his hands, etc. A group can pressure people to do the right things here, but it's something different from the slow motion horror of bureaucracy that tries to do the right thing but cannot.

I call that the rattlesnake principle. Treat all dangerous tasks like you’re dealing with a rattlesnake.

I'm reminded of Louis Slotin experimenting with the "Demon" core. The core was surrounded by 2 half spheres of beryllium. The core would go critical if the 2 spheres were not separated from each other.

The standard protocol was to use shims between the halves, as allowing them to close completely could result in the instantaneous formation of a critical mass and a lethal power excursion. Under Slotin's own unapproved protocol, the shims were not used and the only thing preventing the closure was the blade of a standard flat-tipped screwdriver manipulated in Slotin's other hand. Slotin, who was given to bravado, became the local expert, performing the test on almost a dozen occasions, often in his trademark blue jeans and cowboy boots, in front of a roomful of observers. Enrico Fermi reportedly told Slotin and others they would be "dead within a year" if they continued performing the test in that manner. Scientists referred to this flirting with the possibility of a nuclear chain reaction as "tickling the dragon's tail", based on a remark by physicist Richard Feynman, who compared the experiments to "tickling the tail of a sleeping dragon".

On the day of the accident, Slotin's screwdriver slipped outward a fraction of an inch while he was lowering the top reflector, allowing the reflector to fall into place around the core. Instantly, there was a flash of blue light and a wave of heat across Slotin's skin; the core had become supercritical, releasing an intense burst of neutron radiation estimated to have lasted about a half second. Slotin quickly twisted his wrist, flipping the top shell to the floor. The heating of the core and shells stopped the criticality within seconds of its initiation, while Slotin's reaction prevented a recurrence and ended the accident. The position of Slotin's body over the apparatus also shielded the others from much of the neutron radiation, but he received a lethal dose of 1,000 rad (10 Gy) neutron and 114 rad (1.14 Gy) gamma radiation in under a second and died nine days later from acute radiation poisoning.

https://en.wikipedia.org/wiki/Demon_core#Second_incident

That reminds me of when a family friend from church needed help clearing his land and thought the easiest approach would be to teach an overconfident 14 year old (me) to drive his tractor. He told me I would never be worse at operating it than the second time we went out. He was right.

Yet the older they are the more they test in production.

A WTA top 70 tennis player from my country (aged 35+, thus possibly facing the end of her pro career) recently rephrased a well known proverb: "What doesn't kill you, makes you stronger -- or, crippled."

There's chance of death, but there's also duration of suffering while dying.

I'm guessing that falling from a cliff is "better" than dying from a poisonous mushroom. The latter scares the hell out of me. The former is a glorious ride until the ride is over (regrettably).

I regret to inform you that possible negative outcomes of falling from a cliff include life-long pain, paralysis and brain injury.

>The former is a glorious ride until the ride is over (regrettably).

If you sense you're falling to death, it wont be too glorious (personally), but freakish. It can also always fail to bring death!

This line is famous in aviation, where X = pilots

Yeah after I read that many years ago I saw it come up in several other contexts (including aviation) and realized it probably originated elsewhere.

I would imagine that if you scale back enough tho, you won't be as sharp. Sure the odds increase the more you do it, but not just because you do it, but often because of other variables, such as the weather, not listening to your body, over confidence, etc.

That's the reason why I don't cycle in London. When I moved there, I thought I would be using my bike daily like I was used to. But over the span of a few years, I'm pretty sure the risk of serious injury becomes significant.

For rock climbing, you're probably right. I remember training in a climbing hall, when I saw someone falling off the highest wall. The tenant of the hall didn't look surprised at all. Apparently, it happens frequently.

That being said, if you serious about security, I'm sure the risk can be minimal.

You can contrast the odds of getting injured with the health benefits. The cardiovascular benefits would seem to outweigh the risks of getting injured from a mathematical point of view.

See e.g. https://blogs.bmj.com/bjsm/2018/12/12/pedal-power-the-health...

Lots of ways to get even better cardio benefits with much less risk.

Same reason I sold my motorcycle when I was 30. I had a good run of not being maimed, but the odds were not good for the future.

You would probably be at less risk if you continued rock climbing but eschewed driving (or riding in) a car.

I took the same approach with motorcycling. I decided to do it for 3 years while at university because it had a transformative impact on my lifestyle. But I also decided the odds were way too bad to keep doing it my whole life. So I stopped and haven't done it since then.

I only owned a bicycle for several years at the same age, and have since mostly used a car to get around. I've been in various, relatively minor accidents with both, and have always wanted to try using a motorcycle, but it seems to take the risks of both biking and driving and puts them together!

"Climb if you will, but remember that courage and strength are nought without prudence, and that a momentary negligence may destroy the happiness of a lifetime. Do nothing in haste; look well to each step; and from the beginning think what may be the end" ~Edward Whymper

This is why I don't drive or talk to people anymore.

Isn't this from Jared Diamond? I read this in The World until Yesterday I thought, from his time in New Guinea.

Is 1:5000 an actually good assessment? I'd suspect it's very much not, especially if you avoid obviously dead trees and maybe move if a storm comes.

You could calculate it by taking the average lifetime of a tree and dividing by the length of time slept and the number of trees in squashing distance.

Only if tree death is uniformly distributed over a tree's lifetime for some bizarre reason. And it isn't.

The logic in the story is BS. First, you would never, ever get into a car with that logic. Second, trees just aren't that ephemeral. People who live in a forest would be very aware of when they do or don't fall (or more problematically, drop large branches.) It's not as straightforward as just avoiding storms or dead-looking trees. Sustained wet weather, especially after a period of dry weather, is a common cause. As is the opposite for some trees (eg oak trees drop limbs in sustained hot dry weather.) As for disease or other causes, an experienced hunter in a familiar area could tell at a glance.

The message I got from the story is that they probably did have a very good reason. They either thought it would be too hard to communicate, or they were themselves cargo-culting the falling tree excuse when the reality was more likely to be... I dunno, snakes or nasty bugs or annoying sticky sap or whatever.

I think your explanation is pretty plausible, I wonder why people are downvoting it.

Like these guys probably have homes, with bedding of some sort, maybe they'd rather sleep next to their wives than some caterpillars. If I was giving somebody from a far-off place a tour of my workplace and, on the bus ride home, they suggested it was getting dark and we should camp on the sidewalk I'd probably not go for it. If they were really insistent I'd probably amplify the danger of sleeping on the sidewalk to shut them up.

> First, you would never, ever get into a car with that logic.

The logic isn't applicable to any set of risks. As deadly as cars are, the risk of car crash death is much, much lower than 1/5000 per trip. It's probably closer to applicable to being a drunk driver, and "you would never operate a car drunk" is pretty accurate for many people.

1:5000 would suggest an average lifetime of 13.7 years for a grown tree (i.e. not counting the years where it's too young to sleep under), and that's before the ability to avoid trees that look more likely to fallover.

I don't know anything about trees around there, maybe they're really short-lived? For forests around here, it's a gross overestimate.

Somehow I suspect trees in Papa New Guinea are different than the trees where you are

or perhaps the main difference between Papua New Guinea and the GP's location is that people feel entitled to make up weird questionable stories about PNG and not about their place.

Don't know about Papa New Guinea, but where I live we have had lots of healthy trees fall over from a bit of wind when the ground is soaked.

Remember that the relevant set isn't the set of all trees, but the set of trees old enough to be large enough to sleep under. That means the average lifetime would be: (number of years it takes for a tree to become big enough to sleep under) + 13.7.

I would still guess that the number is wrong, I suspect that trees have a longer life on average from becoming what we would consider "big" until they fall over. But it's still an important detail.

I mean “big enough to sleep under” is around 50+ years for most hardwoods IIRC. Also, you have to consider groups of trees that can fall on you in the surrounding area. So it’s really the probability that a tree falls within any given 50m radius, towards the center. I think the stated probability is pretty accurate. Also as humans, we tend to pick trees that provide the best shelter, thus older. From experience, if one of those big older branches decide to fall, you don’t want to be underneath. As a kid, a huge oak tree lost a huge branch right in front of me. It scared the crap out of my 8yr old self, so much that I still remember it quite vividly, it would have killed my brother and I if we hadn’t just moved out of the area. The limb looked perfectly healthy btw, green and all. There was no indicating factors, according to the adults at the time.

Even if you multiple by ten this still means you'll end up with one dead hunter every few decades (depending on how often they hunt and how many hunters there are exactly). That seems like quite a high risk for a small self-sustained community.

Also, I bet it's not just the tree you're sleeping under that poses a risk, but also other trees in the vicinity that might fall on you. In a forest there are probably a bunch of trees "in range".

All of that said, I've often camped and slept in forests, as have many of my friends, and I've never heard of anyone being killed or injured by a falling tree, or ever heard or seen any "don't sleep under a tree, it might kill you"-advice, so I don't know...

I think most trees fall when there is stormy weather. A wind free day, I think much less than one in 5000 trees fall.

If the story is based on Jared Diamond’s story, the tree was dead, and the estimate was 1:1000. https://www.openculture.com/2015/08/jared-diamond-underscore...

Not an expert on this, but I suspect you wouldn't want to move camp during the night if you can avoid. Especially in the rain.

PNG has a lot of eucalyptus trees, and some species drop their branches with zero warning, which originated as a mechanism for surviving droughts. I don't know enough to tell whether that's what's going on in the estimate.

Great anecdote.

The tricks in this article might work in the short term. However, over the length of a career, it's difficult to outrun a negative reputation forever. Especially in the age of the internet, people will eventually catch on to what you're doing.

Also: in the version of the story I remember reading somewhere, all night they kept hearing trees fall. And that had a significant effect in affecting their impression of the probability.

Taleb has a similar example: the difference between playing Russian roulette once for a chance of winning $10 million, versus playing it repeatedly.

Great example of a non-ergodic event. The outcome (odds of dying) when considering of one individual longitudinally is entirely different from the outcome when considering a population of individuals at a single point in time.

https://taylorpearson.me/ergodicity/

I am just wondering what skydivers must think every time they do it ... given so many trials, the odds of nothing happening are going down exponentially right?

Reminds me of the fact that on overage, one out of every hundred places you know will be experiencing a once-in-a-hundred-years event.

When you hear once-in-a-hundred-year event, it makes it sound quite rare. One might look around and say (for example, in relation to climate) "why are so many of these happening?"

But it is unsurprising statistically. If you know just a thousand distinct geographic places, about 10 of them would experience such an event each year.

Note that the size of the place is important.

"highest temperature ever recorded in town X" does not mean much.

"highest temperature recorded in country Y" on the other hand is more significant, especially if the country is large.

There's something intuitively misleading here, though I'm not sure what it is. I am looking at 100 square inches of floor right now and nothing unusual is happening in any of them.

Are there 100-year events associated with 100 square-inch plots of floor? No... But there are with coastlines, cities, dams, forests, deserts, islands, volcanoes, etc.

Of course there are other compounding factors. Not all events are one-in-one-hundred; some are one-in-ten, others are one-in-five-hundred, and with increasing scarcity by order of magnitude.

Another compounding factor is that the borders for "geographical areas" are fuzzy. Does a one-in-a-hundred year event that happens in Vermont also qualify as having happened in New Hampshire? Probably depends on the type and the specific measurements and the expectation according to history.

And what about aggregate events? E.g. "It's been five hundred years since we've seen this many tornadoes, which tend to happen once every ten years."

And in the opposite direction of the 100 square-inch plot of ground, there is blurring in the other direction: What do they mean in aggregate if they are fewer than expected at a global scale?

So my original comment was just a simple way of taking the average of all of these competing factors and stating a general truth that's somewhere in the middle. If you have X things that are expected to have 1/Y probabilities, then the probability of you experiencing any of them is not 1/Y, it's more like X/Y. (As in very likely.. that's more than 1 so obviously not mathematically correct.) But we often don't think about rare events this way - as no specific event being probable, but some collection of improbable events being almost certain. We perceive them all as 1/Y. It's just a very local way of thinking.

It has to do with how correlated conditions are in your 100 places. Conditions on one square inch of your floor are very highly correlated with conditions on another square inch of your floor, so they wouldn't be able to experience a 1-in-100 event independently -- if one part of your floor did something unusual, the other parts probably would too. Conditions on the floor the next room over are also pretty highly correlated, but not quite as high (maybe a fissure could open up and swallow the kitchen, but not your office). So in order for the parent comment to be true, their thousand distinct geographic places would need to be statistically independent from each other.

Of course in practice, it's quite hard to know whether conditions in one location are independent from another, or whether there's some degree of correlation or an underlying causal factor. This is why we have climate scientists.

I'd expect some degree of spatial correlation.

Jared Diamond talks about dead trees: https://www.openculture.com/2015/08/jared-diamond-underscore...

Yup, and the article ends with this:

>>All these things are almost always true. But Heuristics That Almost Always Work tempt us to be more certain than we should of each

The most important thing to realize about risk is this:

The difference between [using knowledge, skill, technology, and planning to manage risk] vs. [getting away with something]

If you aren't managing risk, it is managing you. And you can get away with something for a long time, but it is always a matter of until you don't, and then it is too 'effin late for you.

When you are managing the risk, you can make an entire career or lifetime of doing things that will otherwise kill you in seconds, scuba diving, flying, mountaineering, building tall structures, working with molten metal or dangerous chemicals, etc., etc., etc.

You can also get away with very dangerous things for at least enough time to fool you into thinking you are smart. The article discusses this at length.

This is why when you need to understand and manage the risks, and also be very alert to close calls - they mean that even though you thought you're managing, you've actually gone into the land of [getting away with it], just saved by Pure Dumb Luck. Don't say "it's okay it worked", look at why, because you might not have as much PDL next time.

I'd like to see the average life expectancy graph rephrased as "probably of dying today given current age".

Back of the envelope calculation: Life expectancy of 72 years times 365 gives about 26k days, so your average chance of death on a given day is on the order of 1 in 26,000.

Perhaps sleeping under a tree is the "not putting on your seatbelt" of the jungle.

Australian here. Never ever ever park, and especially never ever ever ever camp, under gumtrees.

They can and do drop large limbs at any time without warning.

There are gum trees in PNG.

Even if you're going to do it only once it is still very dangerous compared to the risks we normally take. 1:5000 is 200 micromorts, about 25x as dangerous as hang gliding! [1]

[1] https://en.wikipedia.org/wiki/Micromort

But the odds aren’t additive. If you sleep under a tree 4999 times and nothing happens, that doesn’t mean the 5000 time you sleep under one you’ll get crushed.

Each and every time you have a 1 in 5000 chance.

I don't believe the odd that you get crushed once if you play it multiple times are not as good as the odd of not getting crushed when playing only one time.

So you have 0.02% chance of getting crushed each time.

(1 - 0.0002)^10950 = 0.1119 1 - 0.1119 = 0.8881

So the probability of getting crushed at least one night over 30 years is ~89%

I think, my stats are from school over 10 years ago.

This is because, for example, if you flip a coin, each time you still only have 50% chance of getting tails. But the chances that you flip it 10 times in a row and never get heads are a lot less than 50%.

That's said. Another tricky bit is, what was measured when we said 1/5000 chance? This is where data can get confusing. Was that the odd of a tree falling at any given night? Or was that the odd of someone being crushed by a tree in their sleep at night? Or was it the odd of someone being crushed by a tree ever? Or the odd of a particular tree falling at night?

That's often where any prediction already begins to break down. For example, sorry to use the vaccines as an example, but when we say 90% efficacy, it means, out of x number of people who got a vaccine during the trial, 90% of those didn't get covid during some period, while in the placebo group it would be some other % who got it.

Reasoning about this already is tricky. You don't know the priors. What was the odd your participants were exposed to COVID? What if you'd measured over a longer period of time? What if that was just a lucky bunch?

Don't have to be additive to "add up".

>If you sleep under a tree 4999 times and nothing happens, that doesn’t mean the 5000 time you sleep under one you’ll get crushed.

No, but it means you have far more overall chances of getting crushed if you do it 5000 times, than if you do it once or twice.

I often wonder these days how it was to live in so much risk and unknowns in prehistory. Nothing was sure, understanding of the world was so limited ..

Also, you're not trying to just limit falling debris on yourself, but anyone in your party since you'd hate to lose a friend that way.

Jared Diamond tells a similar story about tree falls in the jungle.

https://www.nytimes.com/2013/01/29/science/jared-diamonds-gu...

> for a lifelong hunter, 1:5000 are pretty bad odds

There’s a term for this which I’m unable to recall and it’s not easy to Google. Would greatly appreciate if someone could help me out here!!

Effectively it’s integrating risk over time, but that’s probably not what you’re thinking of.

Is it Micromort? I just learned about this from a comment above.

https://news.ycombinator.com/item?id=30267553

Compound interest? Exponential decay? Both apply here.

Probability of surviving a night under a tree is (1-1/5000).

15 years of hunting is roughly 5000 nights. The chances of never getting hit by a tree over that period are (1-1/5000) for each night, which compounds to

    (1-1/5000)^5000 ≈ 1/e ≈ 1/2.7 < 0.4

That's to say, the odds are 3:2 (at least!) that you'd get killed by a tree in 15 years.

Make it at least 5:1 for 30 years.

That's to say, at least 5 out of 6 hunters who sleep under a tree every day wouldn't survive doing it for 30 years.

And that, children, is why credit cards are a scary thing.

Ergodicity? E.g. A single player playing Russian roulette for a 100 times v/s 100 players playing it once are very different risks for the single player :)

That's it, thank you!

For those curious, (4999/5000)^3650 is approx .5. Meaning your chance of dying under a tree is about 50/50 after 10 years.

It seems like 1:5000 is not an accurate probability, but just a number which he chose in order to convince himself that sleeping under a tree was not a risky activity. But he chose a bad number and later realized that an argument based on that number was not convincing.

If the chance was actually 1:5000, the longevity of trees would be similar to that of hunters sleeping under them - or lower, since hunters can sometimes avoid the hazard, but trees are exposed to it every night (and all day as well). Actual data on tree mortality seems to indicate that the chances are much lower than this. Almost certainly they do not make sleeping under trees a significant risky activity.

Takes the whole tree collapsing to kill the tree. A large enough branch (or a coconut, if we're on Papua) should suffice for a hunter.

The hunters said 'a tree might fall on you and kill you'? This is supposed to be a nugget of ancient wisdom. Now it turns out it's actually a coconut falling out of the tree? Are you providing this extra information? Or was it somehow implicit in the original telling?

Is it possible that both the original teller, the reteller and you too, are starting from the (unfounded) assumption that the hunters know what they're talking about, and are adapting the facts to fit this assumption?

I read somewhere that the Mongolians believed flowing water was sacred and they had a taboo about defecating near it. For the wrong reasons they accidentally prevented cholera. If avoiding sleeping under trees prevents tree falls, lightning strikes, coconut comas, snakes from dropping on you, and whatever else then it doesn't really matter why the hunters don't sleep under them does it? Not having coconuts to worry about is just a happy bonus to the survival statistics.

Even if it is, no one suggested that local hunters sleep every night in the forest. If they happen to sleep under the tree once in a year, it won't increase their risk of death significantly.

There could be another reason why sleeping in the forest was dangerous – e.g., poisonous snakes or dangerous animals etc. Also if the temperature drops at night and they didn't have enough clothing, they could get hypothermia and so on.

How did he even come up with these numbers, anyway?

https://pubmed.ncbi.nlm.nih.gov/6502774/

There’s a real dearth of info around coconut fatalities. This 1984 paper examined a 4 year timespan of all trauma-related admissions to one hospital in Papua New Guinea. 9 of them (2.5% overall) stemmed from coconuts. In 3 of the cases, all children, the patients slipped into coma.

From what I’ve read elsewhere (no good links, sorry), coconut injuries worldwide seem to have fallen significantly due to better harvesting practice - the age of a coconut and their chance of falling appear to be linked. While I can’t find a good paper saying this, I do buy it.

So, perhaps not 1:5000, but (at least at one point in time) definitely a risk.

Taleb also discussed this in his work.

Any payoff from a trade which has even a very tiny probablity of making you go bust is zero.

Because once you go bust you are not going to be doing trading anymore.

He calls these Uncle Points.

What is his name? Is he, by any chance, from the biological faculty in Ceske Budejovice?

There are more dangerous things for a Papua New Guinea hunter though.

I read them as sloppy caricatures that provide little value to the conversation.

Let's take the security guard: "The only problem is: he now provides literally no value. He’s excluded by fiat the possibility of ever being useful in any way. He could be losslessly replaced by a rock with the words “THERE ARE NO ROBBERS” on it."

That is blatantly not true, the guard provide value since the wanna-be robbers don't know whether the guard will be of any value, and a rock could not provide any deterrent. Losslessly?

The doctor (if there is someone skeptical of doctors it is me, if I were less lazy I would have greatly enjoyed fighting to make some useless US dermatologists lose their license): "Her heuristic is right 99.9% of the time, but she provides literally no value. There is no point to her existence. She could be profitably replaced with a rock saying “IT’S NOTHING, TAKE TWO ASPIRIN AND WAIT FOR IT TO GO AWAY”.

This is an unhelpful caricature, too, because if we add "and call back if it does not", it looks like a very reasonable approach.

Un-nuanced and quite sloppy presentation of heuristics.

Maybe we should replace you with a "THIS NEW IDEA ISN'T NUANCED ENOUGH TO REPRESENT THE WORLD" rock.

I agree with you. The value of the heuristic is in the 99.9% and the value of the person is in the 0.01%. That's why doctors are so valuable. It's for those times when someone comes back and says, "It didn't work." If we waste the doctor's time on the 99.9%, then they lose their primary function of being the expert during the unexpected. For an individual, they may seem interchangeable with a rock, but for a society, they are not. This is because while the probability of cancer per unit of human doesn't change (enough to matter for this illustration), the more people there are, the probability of cancer per unit of time does increase. 0.01% of a lot is still something.

Crucially, however, the security guard and doctor are roles that have value in their responsive nature. Then you have the part of the article where you are talking about roles that have value in their predictive nature. And for those, yeah, the author has a point.

> If we waste the doctor's time on the 99.9%, then they lose their primary function of being the expert during the unexpected.

I don't agree that this is a waste of time. The reason why we go to the doctor when we have unusual pains and aches is because we are not qualified to decide whether it's a minor annoyance or a symptom of a life-altering problem. We're seeking out an expert to help make that determination.

If two people could walk in with the same described symptoms, but one just ate something that disagreed with them, and the other has stomach cancer, then doctors must do at least a basic diagnostic exam on the spot, and not turn the patient away for days or weeks to see if the problem goes away on its own. This isn't a "waste" of time at all.

I just saw a PA a couple weeks ago about a knee injury. The end result was that she did tell me to take ibuprofen for a few weeks and then report in how it feels. But, before that, she spent a good 20 minutes asking about the history of the injury, and then felt around to see if there was anything noticeably wrong. And after that she gave me an idea of what the next steps would be if it doesn't get better on its own. The entire experience gave me confidence that she was attentive to what I had to say, knew what she was talking about, and had a plan in case things don't get better.

If she had merely listened to me for 5 minutes, and then told me to take ibuprofen for a few weeks and call back if it's not better, and that was it, I would not have felt good about that encounter, and would have gotten a second opinion from another doctor.

Of course, this also raises the question of whether or not doctors who put in that bare-minimum, insufficient amount of effort (as described in the article) actually are common. I really hope they aren't! But maybe they are, I dunno. The author is a medical professional (albeit on the mental health side), so I would expect he'd have a better idea of how common that is than I do.

I agree with your take on the doctor heuristic. But there is also the worry that the patient won't call back for longer than they should, and the underlying disease could have a chance to get a lot worse in that time. If a doctor really isn't doing any basic diagnostic work, and is just telling people "take some aspirin, and call back if it doesn't go away after X days", that is still IMO negligent. If I have cancer, I don't want to wait 2 weeks or a month longer than necessary to start treatment. That could be the difference between life and death.

On the security guard, you're right, but I think it's still interesting to think about. If the security guard really does just assume any noise heard is not robbers, then they are effectively not doing anything. Their deterrence value is non-zero, agreed, but maybe it's cheaper and more productive to install motion-activated security cameras at ingress points, and big honkin' signs at the perimeter that say "Security monitoring in effect 24/7". Yes, you have to pay for the monitoring, but that's probably cheaper than paying for a butt-in-seat. And safer, too! Most security guards at most places probably aren't trained for much beyond the basics. I think it's more likely than not that someone will get hurt if robbers do show up and a security guard gets involved. I would much rather some stuff get stolen than people get hurt. (Certainly there are some things that are critical enough that require a well-trained security force. I just think these things are less common than most people would think.)

> Un-nuanced and quite sloppy presentation of heuristics.

I think the lack of nuance is in part the point. We should be examining places in our lives where we go by our gut feeling, or by the common case, when we probably should actually be doing research, diagnosis, or testing. Even if the results aren't as dire as some of the non-nuanced takes, they're still bad enough to warrant some thought.

The Hydroxychloroquine discussion is pretty bad as well.

We were confidently against Hydroxychloroquine because it was well tested in the RECOVERY trial in June 2020 [1].

The time to be optimistic was during the trial-phase of the test. Once that time passed, everything beyond that point was unnecessary and unhelpful hype.

------

What's amazing to me, is that the RECOVERY trials showed that Dexamethasone cut the death rates in half [2]. Where did the Dexamethasone hype go? Why was the political discussion on the snake-oil Hydroxychloroquine?

Even months before we had a vaccine, doctors had discovered how to save roughly 10% of lives in the most severe category. (29.3% death rate vs. 41.4% death rate for those on ventilators). Doctors quickly started using the $20 steroid to save so many lived throughout the pandemic.

Instead of hyping the stuff that _WORKED_, the political system was hyping snake-oil like Hydroxychloroquine and Ivermectin. Why were so many people optimistic on such bullshit?

------

Here is where being skeptical helps. Whenever Hydroxychloroquine or Ivermectin came up in a stupid discussion, I'd pivot the discussion towards Dexamethasone and Monoclonal antibodies. People need hope. People want hope. Before we had vaccines, people wanted to know that doctors had an idea of what to do. Dexamethasone and Monoclonal Antibodies did the job and did it properly (saving lives long before vaccines were developed). No one cared if it was "hydroxychloroquine" or "dexamethasone", these are all just giant chemical that few people memorize. People wanted hope, and it was important we kept that hope factually correct.

You can't promise the world with bullshit snake oil. When someone gets COVID19, they'll demand Ivermectin (yes, my sister is a doctor and she's seeing patients who are literally dying and demanding Ivermectin from her). Ivermectin has given these people false hope and is turning them away from the actual care that would save them.

[1]: https://www.recoverytrial.net/news/statement-from-the-chief-...

[2]https://www.nejm.org/doi/full/10.1056/nejmoa2021436

Perhaps it's the very fact that dexamethasone was useful (and therefore used) that explains why you never hear about it.

Since you mentioned the political context, it's not necessarily just "something that works" that you're looking for in a political sense, but "something that works that THEY don't want you to have." If everyone agrees on it, there's no action, no wedge.

The fact that most people are going to recover anyway is also fertile ground for snake oil, as everyone runs their own uncontrolled experiment and declares success when they survive, and most do.

I think you missed the point. The idea here is that there were people that said before any results were in that hydroxychloroquine was not gonna work, operating with a heuristic that almost always works. And in this case they were correct, which the article doesn't seem to disagree with. The point is that at some point the heuristic will fail, not that it did in the case of hydroxychloroquine.

I remember hearing hydroxychloroquine conspiracy theories well into the election of November 2020. In particular, when Trump revealed his positive diagnosis, plenty of people were talking about Hydroxychloroquine at that point.

That crap was disproved 5 months earlier, and plenty of people still didn't get the news.

...ok? Again, not the point being made by the article so not sure what point you're trying to make.

The article ignores useful treatments like Dexamethasone and monoclonal antibodies.

The correct skeptic was pro dexamethasone anti Hydroxychloroquine.

Seeing things from only an anti-htdroxychloroquine perspective is 100% misleading. There were working drugs that saved many lives during the stupid Hydroxychloroquine hype.

------

What timeframe are we talking about here? What else was known and tested? There is a political group who kept pushing HCQ for months, and then mislead the public again with Ivermectin a few months later.

Ivermectin was entirely a 2021 phenomenon as well. Not only did we know that Dexamethasone + Monoclonal antibodies worked, we also had 3 competing vaccines and the "Pfizer anti-viral pill". Why the hell were people talking about the snake-oil Ivermectin?

The article gives fluvoxamine as an example of a useful treatment that the heuristic-following skeptic would have been wrong to criticize:

"(shame about the time she condemned fluvoxamine equally viciously, though)"

But we weren't "heuristic following skeptics" with regard to the pandemic.

The RECOVERY trial basically tried everything that had a chance of working. Hydroxychloroquine was part of the tests and did very poorly.

------

After the RECOVERY trial in June 2020, it was no longer about "what worked", the question was "what works better?". By the time Ivermectin became a discussion point in 2021, it wasn't good enough to just "work", you had to prove that it was at least as good as Dexamethasone + Monoclonal antibodies.

Its not about being a "heuristic following skeptic". Its about knowledge of treatments that do work and the tests they underwent to prove their efficacy.

The correct answer for Ivermectin was "Hey, we have this test with Dexamethasone + Monoclonal antibodies that showed efficacy over 3000 people. Where is the evidence for Ivermectin?"

Oh, you don't have evidence yet? How about we wait until you have evidence before you claim that IVM is more useful than the current Dexamethasone + Monoclonal antibodies cocktail?

But it's a strawman because sceptical people were in confused "wait for the science" mode not "it is impossible" mode.

But it's worth pointing out that (i) the medical profession, who are the appropriate people to be evaluating HCQ, didn't rely on their heuristic but did the test and (ii) if you as a layman abandon the heuristic that you shouldn't be receptive to the advice of conspiracy theorists and alternative therapists protesting the medical profession because they might be right about something, following that advice isn't largely cost-free unlike the other examples Scott Alexander set up, and is in fact more likely to harm you than the heuristic

"Extraordinary claims without evidence or rationale are bogus" as a heuristic seems pretty good to me?

When presented with a problem and infinite solutions, you need to start narrowing down which paths to investigate somehow. Come back with evidence and the conversation goes further.

At least in the case of the doctor, the numbers are also way off. A typical doctor sees thousands, if not tens of thousands of patients in their career. This would certainly translate to _way_ more than two or three misdiagnosed cancers.

Definitely. I suppose it's like an on-call engineer with a dodgy metric that "cries wolf" very regularly. There are hundreds of deadly conditions that have "flu-like symptoms", cancers that manifest in a myriad different ways (from anemia to blurred vision), and other serious conditions that start out as plain old lethargy or listlessness and get diagnosed as depression. I have encountered a complacent doctor who misdiagnosed my sister's cancer as "pain" (??) prescribing pain killers for weeks while the cancer spread and additional symptoms appeared. [Edit: I misremembered as months not weeks, but it was about 6 weeks]. (She died). My father is 95 years old, and had extremely low blood pressure for months, making him constantly sleepy. My mother accidentally made the discovery after she bought a blood pressure monitor for my sister (who has high blood pressure). They took him off the medication, and he's back outside chopping small pieces of firewood and taking short walks. As far as the doctor was concerned he was tired because he was old, and his body was "winding down like an old clock". Too lazy to take his fucking blood pressure. Doctors get extremely complacent, burnt-out, depressed etc. They frequently miss the 1:1000 time that the symptoms are something serious, so I literally never trust a doctor to make a diagnosis without carrying out actual tests. If I'm ever ill enough to need one, I'll have done my own research before visiting one. I don't care if it pisses them off.

Are you reading the deeper lesson though? The individual examples aren't meant to be authoritative. He was trying to illustrate the very thing you are bringing up. Namely that lazy heuristics create information cascades. The information cascades can have positive effects, as you point out, but they can have profoundly negative consequences, which is the point of the whole article.

We shouldn't use or intellectually tolerate lazy heuristics because they can create immense amounts of counter-productive sense-making, and consequent negative social outcomes (a poorly managed pandemic, for example). The reason this article is hitting a nerve is because he is basically describing the current state of sense-making in the US (and maybe even the West more broadly?), which is quite poor — worse in some areas than others, but still quite degraded all around.

On your doctor take, you do know that the other author of this post is a licensed and practicing Physician, right?

Right. I think it has to be boiled down to a simple parable to be sufficiently clear, but it is really attacking a belief that is quite common.

I like this essay on Overconfident Pessimism[0] because I think it gets at the same thing, especially in the context of confident dismissals of important future technological changes:

>There may also be a psychological double standard for "positive" and "negative" predictions. Skepticism about confident positive predictions — say, that AI will be invented soon — feels like the virtuous doubt of standard scientific training. But oddly enough, making confident negative predictions — say, that AI will not be invented soon — also feels like virtuous doubt, merely because the first prediction was phrased positively and the second was phrase negatively.

[0]. https://www.lesswrong.com/posts/gvdYK8sEFqHqHLRqN/overconfid...

"On your doctor take, you do know that the other author of this post is a licensed and practicing Physician, right?" --

Your observation is indeed much more interesting than the whole article, since it shows a reasonable (and not a caricature with zero value) heuristic. You are saying that I (who may or may not be a physician, but for argument's sake let's say I am not) should not have an opinion that is different, when discussing the behavior of doctors, from the opinion expressed by a licensed and practicing physician. Valuable heuristic?

As for the article itself, my problem with is was not on the problems that reasonable heuristic can generate, but with the useless caricatures.

A doctor who does not visit any patient and simply gives aways a couple of aspirins, is criminally negligent. A doctor who does not call for an MRI for any common symptoms (think headache) that may have been caused, among many other possible causes (dehydration, stress, tension etc.), also by something much more serious (brain cancer) is using a reasonable heuristic, which sometimes may go wrong because for very aggressive cancers, a couple of weeks of delay in starting treatment or having surgery can make the difference between life and death.

A personal case. I went to a doctor with a dermatitis and the doctor recommended, guess what?, a topical steroid cream, which is recommended by dermatologist like a barber recommends a haircut. The heuristic is, dermatitis of unclear origins --> let's try a steroid cream. After I did a bit of research on my own (5 minutes, maybe less), I found out that for my conditions the steroid cream should be absolutely avoided since it makes the condition worse. The question is and I let you choose the answer: (1) was the doctor using a reasonable heuristic?; (2) was the doctor incompetent and/or an idiot; (3) was the doctor negligent (there is some overlap with (2))? Should I wait for the opinion of a "licensed and practicing Physician" or I can have my opinion?

I'm confused. You say it's a bad caricature, then provide an anecdotal example that seems to closely match the caricature.

I think OPs point is that the article ignores the reason why the heuristic is used in the first place, which is to optimize the process at scale.

The real cost of gathering additional data and knowledge about the circumstances in the 0.01% chance you know that you have at this point, at scale, means you'll end up with worse overall outcome by not being able to scale to all events that need the attention.

Now, if a doctor is just being lazy, doesn't check anything, says you'll be fine take Tylenol and then spends the next half hour reading a book until the next appointment. Or simply wants to go through twice as many patients to make more money. Ya sure, that's just being lazy and useless, and negligent, replace them by a rock at that point.

But if there are 100 people with initial symptoms, and only 1 doctor. And the deep dive to properly asses the likelihood of a 0.01% chance event in the case of symptom: "My hip hurts when I walk" takes multiple hours, a lab test, many follow ups, etc. While this happens and maybe out of the 100 patients waiting, some have symptoms like: "I'm actively bleeding out my mouth.". "I have spores on my skin." "I'm in so much pain I can't fall asleep." All with known much higher likelihood of something pretty bad and urgent.

That's why triage is so important. And this use of the heuristic at scale might make sense when considering the cost/time and available resources trade off.

At the individual level, it means eventually someone will get shafted by this, they'll be sent home with Tylenol, and 3 days later will have a stroke and it would have turned out they are the really rare case where hip pain could indicate a risk of stroke due to say a blood cloth.

But at a larger scale, many more people will have received the treatment they needed more urgently.

I agree: Null-confirming signals should not be considered evidence to discard the null hypothesis. Decision trees are OK, esp those that have "wait and see" near the root. "See if it goes away" is indeed an information-seeking behavior, and a low-cost one at that.

In other words, your doctor could have been losslessly replaced with a rock that says “apply a steroid cream”, and that heuristic would have harmed you.

If it had benefited thousands of others, does it matter in the larger view?

The naive alternative is the doctor always spending time to get to more nuanced advice, that will be more often than (every time the rock was as good) a waste of resources (time/money/opportunity cost), and which in a lot of cases would also be more harmful (iatrogenic harm) than the rock advice.

"But, why 'always'? It's enough that the doctor goes for more nuance when he sees a reason to!", you'll say.

Well, that's what doctors actually do. They are not glorified rocks, they are bimodal (rock mode vs looking deeper mode) - and they're most rock because (even if the miss a few 'loop deeper' cases) because it's way more efficient than the alternative (always or mostly non-rock, that is, digging deeper without first seeing strong indications that they should dig deeper).

Perhaps it could be more helpful to think of the caricatures as "spherical cow" models the author is using when trying to isolate and illustrate a particular social (?) dynamic.

Of course real-world models will be much more nuanced, but the really interesting bit is that you don't need all that nuance to produce the particular pathology outlined in the article. Specifically, selecting on "who is most right most of the time?" can end up causing your city to be covered in lava, missing a store break-in, or what-not.

There are even more levels to explore with this idea. For example, should you always ignore the heuristics and go for the earest, honest experts? Maybe. In the volcano example, the cost of a false negative is so high that you probably are okay with the incurred costs of false positives by the experts.

However, in the case of the Futurist, the false-positives incurred for a non-rock opinion might end up netting you less karma points or whatever. It's somewhat fun doing a re-read trying to evaluate the cost-benefit tradeoff yourself in each case!

> Are you reading the deeper lesson though?

The "deeper" (?) lesson and subtext I'm hearing from the OP is that we should get excited like little children about each and every new fad, because, you know, this may just be the 0,01% when it actually matters, and we don't want to miss it.

Well, I don't mind using "lazy heuristics" and waiting around a little to see if something happens. No need to hurry or jump around. Plenty of time.

...no the surface lesson is that lazy heuristics produce habits that provide the illusion of certainty without any rigour backing up those beliefs; the outcome may correspond with reality but the process getting you there isn't rational or dependable, because it always produces the same output regardless of input.

The "deeper lesson" if there is such a thing here is that experts are people too, and just as fallible, only in ways that are generally invisible to everyone but another expert.

You have a very weird idea of what security guards do. They don't sit in front of the entrance all night, scaring off would-be robbers. They are inside the building, in their booth, probably looking over some monitors. They do not provide a deterrent, they just call the cops if something does happen. Robbers have no clue if they are there or not. The heuristic works fine.

There are plenty of places with visible security guards.

Well, robbers might in fact know if a particular place has security guards, and they would definitely notice once society replaced them all with rocks.

If they replaced them with rocks its probably not worth robbing the place. I mean, it says so on the rock?

I have a very clear idea of what security guards do and I don't know where you got that I believe they "sit in front of the entrance all night" (which they do very often in South America, by the way, among other locations).

Any "reasonable robber", which means maybe not your early teen looking for some adrenaline or the out-of-their-mind tweakers, knows or checks whether there are any security guards in a building in which they intend to make a robbery.

You may attract lot more robbers if anyone can easily know there is no security guard but just a stone.

Where I come from, companies don't waste real estate with monitor rooms for security guards. Instead the monitoring is outsourced, and the security guard has a fixed position and a set number of patrols they need to do.

It wouldn't change much if the rock said "IT'S NOTHING, TAKE TWO ASPIRIN AND WAIT FOR IT TO GO AWAY AND CALL THIS NUMBER IF IT DOES NOT"

So where can I found the trial for rock-based medical care?

I know of a doctor who gave a seminar at our workplace.

His case was. The cure for most non serious illness, is mostly to wait. Not even medication.

This is why most people think homeopathy works.

>I read them as sloppy caricatures that provide little value to the conversation.

That about sums up the whole movement AstralCodex is part of.

Exactly this.

Null-confirming signals should not be considered evidence to discard the null hypothesis. Decision trees are OK, esp those that have "wait and see" near the root.

These derogatory calls to action and flame-posts against arbitrary "experts" are just tiring. Bring the evidence.

The doctor rings true. I had 3 separate doctors on 3 separate occasions diagnose my 21 month old son with an ear infection, instead of the plum-sized malignant brain tumour that it was.

From their point of view, pediatric brain tumours are very rare and ear infections are common.

Their 99% heuristic almost killed him.

That was 2010. He survived and is now a vibrant 13 year old, but only because of one curious intern/fellow at the children's hospital ER that decided to order a CT to rule out the remote possibility. Her diligence got him admitted and into surgery within a day.

As in the examples in the article, it's those early in their career who are less likely to be following the heuristic.

That is amazing, and congratulations! Do you mind sharing some more info on the cancer type? I thought most malignant brain tumors didn't have outcomes like this.

It is a tough call this one. If my calculations are correct, based on data from[1], male infant head CT scans cause one excess death per ~3k scans. There must have been something else than a remote possibility to warrant ordering it.

[1]: https://www.ajronline.org/doi/full/10.2214/AJR.12.10294

I interpret the linked article different. Can you elaborate?

For example (from the summary):

===============================

"CONCLUSION. Radiation exposure from the use of CT in the evaluation and management of severe traumatic brain injury causes negligible increases in lifetime attributable risk of cancer and cancer-related mortality. Treating physicians should not allow the concern for future risk of radiation-induced cancer to influence decisions regarding radiographic evaluation in the acute treatment of traumatic brain injury."

(edited: formatting)

Excess Cancer Mortality per 100,000 individuals exposed to 100mSv is ~1000 in the 1 year-old male cohort. A single scan is ~3mSv, which makes the mortality ~30 per 100k, or 1 in ~3333, assuming the linear no-threshold model[1].

When someone comes to the ER with a traumatic brain injury, a CT scan is the standard practice because they are at a high risk of dying and presumably the scan is important for reducing the risk.

[1]: https://en.wikipedia.org/wiki/Linear_no-threshold_model

I'm so glad you mentioned the linear no-threshold model.

Do you believe in it? I don't, particularly at the single digit mSv level.

At some point there will be enough data from different countries to confidently support or reject it. Until then I see it as the preferred hypothesis by Occam's razor and also because I'd rather err on the side of caution.

You need to read those things literally. It doesn't mean CT scans should be used whenever a doctor thinks. It means that a doctor evaluating a TBI should be ok with using it. Basically, the odds switch when there is an additional symptom that there is something wrong with the head.

Maybe being an ER doctor itself could bias their decisions more favorable toward CT scans. ER people may see dead people & people with tumors more often than pediatric doctors in their offices. The usual pediatric doctors would see much more patients with ear infections.

Also, to ER people, knowing the root cause is probably important for patients' survival, whereas for pediatric doctors, more often than not, lives are not at stake and customer service/easing parents' fear is more important.

Thanks for sharing, and hope all is well now.

This also rung true for me, and is rather scary/disappointing.

I have a rule now, after experiencing cancer & failed remissions with a family member, which is to always do the deep check. Esp in any case of pain or discomfort that isn't normal.

Even a recent shoulder tear from lifting, the pain/discomfort wasn't something I had experienced before, so I went to a local ER (was worried about a dislocated shoulder). The young physician tried to usher me out with his confidence it was a normal injury most likely (he was right). I asked why not just do an x-ray and he gave me a response based on his heuristics. But the x-ray was an option if I wanted. So we did the x-ray and 20min later he was right.

> The doctor rings true. I had 3 separate doctors on 3 separate occasions diagnose my 21 month old son with an ear infection, instead of the plum-sized malignant brain tumour that it was.

This one is difficult. The first doctor with the first diagnosis was likely doing the right thing. Although tragic, it's exceedingly rare for someone that young to present with a brain tumour.

If a doctor was ordering CT scans every time someone arrived with ear infection symptoms, they statistically be causing more harm than good. CT scans raise an individuals risk of cancer ever so slightly. Not a big deal for individuals, but scaled up to an entire population it's actually a real creator of cancer risk.

Now obviously if you have some serious symptoms then the incremental risk of a CT scan is less than the risk of missing a serious diagnosis. However, jumping straight to the CT scan without good reason would be a mistake. So now doctors are in a position where they must integrate diagnostic history into the equation. Showing up to multiple doctors with an "ear infection" that isn't responding to any treatments warrants deeper investigation. If the issue isn't resolved, proceeding to imaging makes sense.

This is why it's important to begin return visits with a summary of what's been done so far. If you get a doctor who wants to re-start the diagnostic process without accounting for previous work done so far, unfortunately the only real solution is to move on to someone else.

Exactly, the post is a Scott Alexander post so he'd never object to the concept of priors. There's nothing wrong with having a strong prior that the volcano's not going to blow up this year, that it's just the wind not a break-in, or that it's most likely an ear infection with a tumour so unlikely as to not be worth the radiation risk and cost of a CT.

Really, the article is about people who don't bother to monitor the likelihood that updates their prior because the prior is so damn explanatory anyway. But, a doctor treating an ear infection isn't doing this because the act of treatment is also a test. It's not the 'did the volcano erupt' or 'did we get burgled' test with huge downsides because even if there's a serious underlying issue a week or two is unlikely to change too much. If the purported infection doesn't clear up then it's time for the doctor to update their prior and reconsider what investigations or treatment options are now appropriate. Even better than taking your history to your next doctor, keep seeing the same doctor.

I'm very glad to hear that your son had a happy outcome!

"Ear infection" is often not even a heuristic. Pediatricians sometimes say that when it's not obvious what's wrong with a small child that has a little a bit of fever and is crying. Parents need to hear a reason and "ear infection" sounds more convincing than "probably nothing", and sometimes parents can be, well, difficult. I've even heard a pediatrician admit as much once. On another occasion, a nurse practitioner diagnosed our child with an ear infection without even bothering to check the ears.

Emergency physicians, on the other hand, are always thinking about the worst case scenarios and trying to rule them out.

Running a CT scan on every kid suffering from ear pain is a bad idea, not only economically, but from the negative effects of radiation exposure to so many kids.

I know in your head the scenario was "were it not for that one person, my kid would've died", but the more likely scenario was, "the symptoms got worse and then a doctor decided it was enough to warrant another look and the tumor was detected, surgery was performed, and my son was ok". It was wonderful that it was caught when it was, but odds are very high your son would've still been ok even if it had been caught over the next several weeks.

Also, I'm very relieved to hear your son is ok, as a father of a 13 year old son, you lived through a nightmare!

The last sentence of the edit the author provides is the key insight of this piece.

> the existence of experts using heuristics causes predictable over-updates towards those heuristics.

That's the essence of this piece. If you expect that consultation with experts will leave you with a more accurate picture of things than before consultation, you should first be sure that their heuristics are not equivalent to reading a rock with a single message painted on it, otherwise no matter what your conclusions will be biased towards that rock. "X is an expert and X says Y is good so I should have more confidence that Y is good than before" is not a useful conclusion if that conclusion came from X looking at a rock that says "Y is good."

The Queen example in particular, but all of the others as well, is a warning that looking only at the accuracy of predictions is not enough to avoid this problem. In order to make sure that those predictions are useful for yourself, you have to ensure that those predictions actually incorporate new information.

Which is the rub, right? How can a non-expert reasonably come to a conclusion of whether or not an expert's prediction is baseless or is actually solid/insightful?

consult another expert and hope his heurestics are different from the latter, ad infinitum (or you yourself end up creating a meta-heurestic of their heurestics)!

One way is to see if they are doing any new context specific work to make their assessment.

Did the doctor run any tests, perform any investigation, or just tell you the most likely cause for your symptoms. That is to say, did they preform any expert analysis on you specifically or simply tell you a statistic for people like you?

I came up with the Goldilocks (meta?-)heuristic[1] for that: Only trust someone to say X is too high if they can also tell you when X would be too low.

A corollary of which would be e.g. "Don't trust a skeptic that says 'X won't Change The World' unless they can tell you which developments would Change The World."

[1] Or Scylla-Charybdis Heuristic if you prefer: http://blog.tyrannyofthemouse.com/2015/12/the-scylla-charybd...

That is actually really useful.

Glad you like it! I prompted big subthread on it a while back, which you might be interested in:

https://news.ycombinator.com/item?id=10505339

I like this idea. Thanks for sharing!

Simple, baseless predictions have no grounds.

Clearly by listening to some rando on YouTube that is pointing out how the whole system is rigged /s

In seriousness, I think it is actually possible for people to understand enough information that they need to make a decision, even if they don't understand it to the level of an expert. I apologize for bringing Covid into this, but here was my analysis for understanding the mRNA vaccines:

1. The mRNA vaccines contain a small snippet of mRNA wrapped in a lipid bubble. This mRNA codes for spike proteins that are present on SARS-COV-2.

2. Your body takes up these lipid cells, translates the mRNA into spike proteins, and then your body recognizes those spike proteins as foreign and builds an immune response to them.

There is really nothing in the above (i.e. mRNA translation, the immune response, etc.) that I didn't learn in high school biology. There are certainly a ton of details that an expert is much more aware of. And, in evaluating my risk, there is certainly a ton of stuff there that I don't know, e.g. what's the probability of my body having a severe negative (a) immune response or (b) other reaction to the spike proteins in my body.

But all that said, even given all of the things I couldn't know because I'm not an expert, the rough details made it clear to me that, in any case, getting vaccinated should certainly be less detrimental than actually getting Covid, which was highly likely. That's why I get frustrated by some of the "trust the science" messages. You don't need to "trust" the science. The basics of the science are understandable by anyone with a high school degree.

Another thing I think is important to understand is that it may make a ton of sense to give very different societal recommendations versus individual recommendations. For example, I think both of the following are easily provably true:

1. Publishing recommendations of "eat less and exercise" is ineffective in combating obesity at the societal level.

2. For an individual, eating less and exercising is the number one way to lose weight.

That is, we know that most people are unable to stick with the recommendations of eating less and exercising more, and we have decades of data to prove it. For an individual, though, if you are able to set up a system to stick with your plan, this is the best way to lose weight.

By getting a second or third opinion, no?

Not really. Get three opinions from professionals giving the default answer and you have three default answers. Because it’s non-competitive it doesn’t even require coordination. Get three home appraisals in 2007 and they’re all coming in at the offer. They don’t have to know each other, they’re just aligned with the same individual incentives.

> Get three opinions from professionals giving the default answer and you have three default answers.

What's the likelihood they've all standardized on the exact same default heuristic?

But even if they did, at least in some of the examples giving the same default answer would be literally impossible when 2nd opinions are taken into account.

The security guard example is instructive. Scatter a truckload of security guards throughout the entire building. They cannot all occupy the same space at the same time. Consequently, the sound of ostensible wind to one security guard is the sound of a robber breathing to another security guard.

Scatter a truckload of rocks throughout the entire building. Now you have a bunch of goddamned rocks.

I'm no digital signal processing professional but by substituting rocks I'd say we suffered a loss in fidelity.

> What's the likelihood they've all standardized on the exact same default heuristic?

In some cases really high. Professionals are often under the same constraints, have no reason to be diverge, and even are incentivized to converge in opinions. These are not independent probabilistic events.

To my earlier example, _many_ appraisers adopted the heuristic of “appraisal = offer + irrelevant_random_noise”.

You security guard example doesn’t really apply to professional opinions. They’re usually done independently. By hiring multiple security guards, you’re forcing them (or at least encouraging them) to spread out. Sure, you’d get a similar effect if you hired ten doctors to spend 20 minutes with you all at the same time. They couldn’t all listen to your heart and tell you to take an aspirin. But if you visit them one at a time they can. So it’s more like ten security guards all watching one camera feed from different rooms.

Examples of this problem aren’t made up. Citigroup accidentally sent $900 million dollars to creditors. An issue I believe is still in litigation about a year later and has been a huge loss. It was approved by three people.

Kahneman addresses this in Thinking, Fast and Slow. (Which I highly recommend, by the way.)

He argues that "expert intuition" is only helpful in an environment where intuition can be trained. That is, where there is obvious, immediate, and frequent, feedback on actions. All the examples given in the post take place in environments where there is no opportunity for the "experts" to receive feedback on their advice.

For sure. It basically shuts down the idea that you can assess your confidence in an expert based on experience! Instead you have to understand the underlying principles, which is no easy task. To me the solution is to delegate the oversight; hire someone else to understand and assess the reliability of the experts. Obviously that has a whole mess of problems too…

> To me the solution is to delegate the oversight; hire someone else to understand and assess the reliability of the experts. Obviously that has a whole mess of problems too…

Mainly, it has the exact same problem you chose it in order to avoid, you now have to understand and assess the reliability of an putative expert in the domain of understanding and assessing the reliability of experts in your original target domain.

>The only problem is: he now provides literally no value. He’s excluded by fiat the possibility of ever being useful in any way. He could be losslessly replaced by a rock with the words “THERE ARE NO ROBBERS” on it

except for the value of having a security guard visible so that 99% of the robbers who might conceivably want to rob a Pillow Mart decide to go rob Quilting Heaven down the road instead.

Instead of a rock you just need an inflatable security guard.

What's frightening about all this is that this article has gotten 15 upvotes despite 100% of the comments so far being about what a pointless article this is.

yeah I don't know I think people might spot the difference. But maybe most of these robbers have a method that says if you see anything like a security guard don't figure out if it is one just go rob somewhere that definitely doesn't have it because there's always somewhere else to rob - but maybe one of the groups of robbers is better than everyone else and they have a real robber mastermind in charge who determines the security guard is inflatable. Then it's on.

on edit: added in missing two words that clarified meaning.

People mostly seem to object to the security guard example as missing a part of the analysis.

I suspect a lot of them are stopping when they see that when is bogus and aren't bothering to poke holes in the rest of them. Readers have their own heuristics, but that doesn't mean the rest of the examples aren't full of holes.

It's really not about several examples of the same thing, but interrogating our intuitions around each one. I at least found I felt differently about different heuristics being replaced by rocks.

Seems like the article writer has fallen for the very issue they're trying to portray, by writing so many examples that they've stopped thinking and just resorted to writing "PERSON CAN BE REPLACED WITH ROCK". If this isn't ironic then I don't know what is.

"Rationalism" is so ironic you might say it is an iron or at least made of iron.

All of them are missing a big part of the analysis, that it is often not a choice of “a rock” but of a person with a heuristic that works way less than 99.9% of the time. A security guard that constantly harasses shoppers, a “futurist” that buys into every new fad, the conspiracy theorist that believes everything on Facebook.

Would anyone really think a weatherman that had say a 70% correct heuristic was good? Or go to a doctor like that?

I was thinking of that scene in Modern Times where Charlie Chaplin’s ‘Tramp’ gets hired as a security guard.

Or security cameras.. or a drone..

Are you saying 15 votes isn't a lot? Also 100% of 2 comments is still 100% of the comments.

It's a lot of votes for something for which the best analysis is something like "move along folks, nothing to see here..."

People who agree upvote and have nothing more to say, people who disagree come to post their objections.

Also, people upvote what they finf interesting rather than what they agree with.

I didn't unvote it and I think each of the example is missing subtle aspects.

But I think it's interesting enough to discuss. The main thing is that are a whole lot of human activities where one can imagine completely rote activity could replace thinking. But in all of these, a deeper look shows to subtle factors actually require a human being to be present.

It's a bit like self-driving cars. 90% of driving is really easy to get working. 99% is moderately hard. 100% looks like it won't arrive for quite a while.

The author is very popular among the tech/rationalist crowds (they are not the same crowd, to be clear), the topic is of interest for the same crowd, but the examples get a C-. It is challenging to write interesting and accurate pieces twice a week, and this is neither. That would be another heuristic.

Simple metaphors and examples always have holes in them. They exist to illuminate the point author is making. They are not the argument.

Yeah this was a particularly weak one though. The doctor example was much stronger.

It's not 99% of the robbers, you're exaggerating. Maybe at best you fool 5% of them, after all if they were that dumb to be fooled so easily they would have been caught by now. But now you have diminishing returns. Just like you, they know a security guard isn't effective, ... and down the game theory rabbit hole we go.

A coworker and I were once stuck in an office building for an hour or two. We were working as consultants at a client's building and ended up working rather late. Not particularly late by software programmer standards, but clearly exceptionally late by the culture of the client company.

At some point in the evening all the exit doors, including the front door, became armed, and this was conspicuously noted as when we packed up for the night and tried to exit to the parking lot, we realized we couldn't open the door without an alert being sent to the police (not just the security company). There should have been a guard at his station (desk, CCTVs, etc) in the entryway, but we found none.

We waited for awhile. Then we walked up, down, and through every corridor and restroom of that 4-5 story building, multiple times, looking for the guard. When that failed, we called the security company to ask them if it was okay to open the door. They swore there was a guard on duty and asked us to wait a little longer in case he was doing rounds. Despite knowing that couldn't possibly be the case, we obligingly passed more time waiting in the entryway. Then we walked up, down, and around the building again, but this time splitting up and shouting. Nothing. Nobody.

We go back down and inform the security company that we weren't going to wait any longer and that we'd be triggering the silent alarm as we left. And guess who exits the elevator just as we were about to open the door.... Apparently he had been sound asleep in a cozy nook somewhere in the upper floors--presumably in a conference room or more likely a private office, the former being something we inspected in passing (glass walls), the latter we didn't feel comfortable opening and entering, and both being the last place you'd expect to find a security guard. IIRC, he wouldn't admit it outright, but just played coy. We weren't mad. A little tired and frustrated because as consultants we still had to get in early the next morning, but that was mostly offset by the sheer absurdity of the situation, and by the fact that he seemed quite elderly.

Anyhow, you may assume too much if you assume the security guard actually maintains some kind of useful presence. I guess these days it's more common to have electronic way stations to log a guard doing rounds. I dunno if this building had such measures (this was circa 2001-2002), but as the sole guard he probably was expected to spend most of his time, if not all of his time, manning the security desk, providing ample opportunity to be doing something else, instead.

That security guard was performing the important function of allowing the management to legally tick the "we have a security guard" box on the insurance form.

This is what happens when we allow proxies for truth fill in for truth. Or another way to think of it when metrics become the goal.

Also, security guards aren't putting their lifes on risk for someones tv. Most guards would just call the police and wait.

That's true, but the (poorly expressed) analogy is specifically referring to the security guard's decision whether to leave his office and investigate a noise. Of course, the mere existence of the security guard in an office (if known to potential burglars) does likely provide some deterrent, but I think the analogy is referring specifically to cases where a burglar would make an attempt regardless of the burglar's knowledge of the existence of a security guard.

I'm surprised how many people are just nitpicking the examples like they are supposed to be rigorous analogies.

The point that I walked away with is that oftentimes experts use these same heuristics even when people assume they are not. People think that experts don't have to use them because they have better tools and skills at their disposal. However, for reasons involving human factors, they oftentimes do use them. Finally, these opinions then get thrown into the body of evidence as if they are ground truth values.

I wanted to take the overall point but I did get bogged down in the examples. The overall tone of the essay is philosophical and allegorical. For an essay with that tone to really make its point it needs to be crystalline in its clarity and feel a real depth of reasoning, otherwise it will lose the reader. Instead it makes rather extreme points in the examples that don't have to do with what I imagine the author wants you to take from the essay. Specifically, the whole business about there being no point to the existence of the people involved. That invites nit-picking ("the security guard is still valuable because they're there! The doctor can still diagnose obvious ailments!"). Then there's the aside about the different Covid treatments, which basically feels like a dig to people who in real life probably took a serious look at the differing treatments before discarding them. These things distract from the much more interesting point that aggregating opinions from experts will not necessarily increase the certainty of the opinion. I think behind this essay is a way better revision where the author has taken a lot of the feedback from these responses and done another draft. In the meantime it doesn't achieve its desired effect as a piece of writing.

Another commenter left what I thought was a rather essential analogy which was whether or not people should run from the hint of a tiger. Running too little invites tiger attack. Running too much invites excessive anxiety. Both have health ramifications: too little fight or flight response and a creature is easy prey, too much fight or flight response and the creature is expending too much energy in the fight or flight state. If the bushes rustle in the right way next to a herd of antelope, the herd will run, and the size of the herd doesn't act as a linear multiplier on the chance that there is a tiger. To follow the analogy rather painfully, the OP article is critiquing the members of the herd who would say, "Don't run, idiot, there's no tiger", when there's an equal critique to be made against the members of the herd who run all the time and are in a constant state of anxiety.

A simpler analogy may've been a smoke-detector that doesn't actually activate when there's smoke. It'd be cheap, super-efficient (no battery needed!), light-weight, no false-alarms, and still usually work correctly 100% of the time for most people.

Another simple analogy may've been a regularized neural-network that's super-efficient because it always returns a nominal-result, not needing to do any calculations. It could work ~>99.9% of the time because the nominal-result is ~>99.9% prevalent.

The author was trying to point out scenarios where lazy-neglect may seem viable. By contrast, some readers may want to give those in the examples the benefit-of-the-doubt, as they might actually be taking no-action as a conscientious decision. While we often want to give folks the benefit-of-the-doubt, that's presumably not how the author intended those examples to be read.

The author was presumably trying to paint pictures in which folks might thrive through neglectful practices, rather than trying to characterize all folks who superficially resemble those in the thought-experiments as being neglectful.

> For an essay with that tone to really make its point it needs to be crystalline in its clarity and feel a real depth of reasoning, otherwise it will lose the reader.

This has a lot to do with the community he's part of and still implicitly writes for. He's blown up in recent years, but the blog still has a strong core readerbase of people with significantly longer attention spans than the average person, among other things. In all the years I've read his work, the meandering, often-humorous examples are half the fun, which makes the discovery of the thesis more enjoyable as well.

No evidence is provided that any experts are doing this - not sure where you got "oftentimes" from.

This reminds me about an anecdote about how stock market works I heard when I was young. The story goes like,

It was autumn, and the Red Indians on the remote reservation asked their New Chief if the winter was going to be cold or mild. Since he was a Red Indian chief in a modern society, he couldn't tell what the weather was going to be.

Nevertheless, to be on the safe side, he replied to his Tribe that the winter was indeed going to be cold and that the members of the village should collect wood to be prepared.

But also being a practical leader, after several days he got an idea.

He went to the phone booth, called the National Weather Service and asked "Is the coming winter going to be cold?" "It looks like this winter is going to be quite cold indeed," the meteorologist at the weather service responded.

So the Chief went back to his people and told them to collect even more Wood.

A week later, he called the National Weather Service again. "Is it Going to be a very cold winter?" "Yes," the man at National Weather Service again replied, "It's definitely going to be a very cold winter. "

The Chief again went back to his people and ordered them tocollect every scrap of wood they could find. Two weeks later, he called the National Weather Service again. "Are you absolutely sure that the winter is going to be very cold?"

"Absolutely" , the man replied. "It's going to be one of the coldest winters ever. " "How can you be so sure?" the Chief asked. The weatherman replied, "The Red Indians are collecting wood like Crazy."

> The Futurist

> He comments on the latest breathless press releases from tech companies. This will change everything! say the press releases. “No it won’t”, he comments. This is the greatest invention ever to exist! say the press releases. “It’s a scam,” he says.

He's got the name backwards on this one. What he's describing is more of an anti-futurist. IMHO, futurists and the ones that make implausibly grand predictions about the future that almost always end up not being true.

I think the irony here is more pleasing.

Bruce Sterling did an OK job. Heavy Weather reads like prophecy, these days.

I think you are listening to the wrong futurists then.

Legitimate futurists are objecting to some pop-culture nonsense with NFT's, cryptocurrency, etc., often describing such things as scams. I suspect that the author may've had that in mind there.

It may be sorta like the problem with science: there's real science and pop-culture science-flavored junk, and pop-culture audiences may perceive real scientists as dismissive because they're always so critical of the latest pop-culture fads.

So while futurists may love new-tech and scientists may love science, pop-culture may see things differently because they see futurists/scientists dismissing (what they perceive to be) new-tech/science.

> It may be sorta like the problem with science: there's real science and pop-culture science-flavored junk

It's kind of an aside, but I think a lot of people (most?) who strongly identify with "science" really identify with science fiction.

Your idea that there's a format to the names is a great heuristic that works most of the time :-)

This seems like the weakest example. I immediately thought of that Paul Krugman quote from the 1990s where he pooh-poohed the internet, something that's stuck with him forever and made him a laughingstock where futurism is concerned.

Wow, the doctor story really hits close to home for me.

My mother was fat. She was feeling especially tired for several months. She went to her doctor. The doctor was historically kind of embarrassed that my mother was fat, told her to lose weight, and didn't palpate her swollen belly.

My mother went to the dentist. The dentist had known my mother for years, and palpated her belly. She sent her immediately to the emergency room.

Happily, my mother survived metastatic lymphoma after the removal of the 9" tumor in her belly, a heavy dose of chemo, and an autologous stem cell transplant. Modern cancer treatment is really impressive!

My mom has been in remission for over a decade, but I'm still really mad at her general practitioner.

One interesting twist on the doctor example: we know she is almost always right that it's nothing, that it will all go away and that just two aspirins are ok. The article correctly points out that she will miss one or two cases where it was a terrible disease instead, and that her prescription of aspirin and to go away and rest will be misguided.

However... doctors must do more than just cure you. They must also "do no harm"; in fact that is (or should be) their default. What if she intervened more directly in more cases, maybe poked and prodded and recommended more invasive treatments? She would get more cases wrong in the opposite direction (recommending a potentially invasive or even harmful treatment when some rest and an aspirin would have sufficed), maybe resulting in accidental death through action rather than inaction.

She must be alert, but hers is a good default/heuristic. It's not the same as a rock with "TAKE ASPIRIN" written on it.

And this is just an example. I think the heuristics that work 99.9% of the time do so because they do indeed work. Erring in the opposite direction can, in some cases, be also harmful.

Yeah this is literally found to be the case with back injuries I think?

Ironically, the author is a medical doctor.

I liked Bob Wachter's story about how taking aspirin accidentally revealed early stomach cancer: https://twitter.com/Bob_Wachter/status/1447972627956391941

Good point - he assumes that false positives are always less costly than false negatives, but that may not be true in this example.

The problem is that people are not statistics. It may sound reasonable on the surface to say that this heuristic minimizes harm on average because she doesn't perform unnecessary interventions on the 99.9%. However, there are still actual human beings in the 0.1% who are harmed. What you're really saying is that if a group of people is small enough, it's fair for them to suffer preventable harm if preventing it would expose the larger group of people to risk.

I'm not going to argue about whether that is true or not, because I think that clearly depends on many factors and may be unanswerable. But as a member of a minority group who is often denied health care, it is often denied for this very reason. If the wrong person is prescribed this treatment, it is harmful. I'm just saying that when you're in the 0.1%, it can be difficult to accept the idea that you have to sacrifice yourself because someone in the 99.9% might be at risk otherwise.

You have no way to know if you're in the 0.1%. It's not written on your body anywhere. So if an early test can save 1/1000 from dying, but the false positives from an early test kill 3/1000, false positives are more dangerous than the disease you may or may not have.

I am arguing that the doctor must remain alert and not be lazy (though doctors are often overworked and tired, but that's a different problem), but that her default of "aspirin and rest" is a good one.

There are an unfathomable number of possible things that could be wrong with you at any one time. All of those might not be present in 99% of people and present in 1%.

But the 1% is not the same for every disease. If you perform unnecessary interventions on everyone for every disease, then you also perform unnecessary interventions on the 1% of every disease for all of the other diseases that they don't have.

Now you've given everyone weird cancers because you've done thousands of x-rays and CT scans for all manner of things.

I think this is a little different - it's "doing the math." The best contemporary example I think is early/frequent mammograms/checks for prostate cancer. If we check how often what we can detect will develop into a threat to health, and compare that to the consequences of treatment at that early stage, we may determine that under some conditions the results of our diagnostics kill more people than the disease would, and therefore we shouldn't do them.

That's different than not treating people at all - even if it's not treating people at all, because the reason you're not doing it is because your diagnostics and treatments are inadequate.

This article is a little misleading because it conflates a few things.

The low base rate prediction problem is a problem not just because of lazy application, it's because the numbers make it impossible to do anything else in some situations. With a low enough base rate, you have to have a preternaturally good indicator to make anything but a negative prediction.

Then you have to resort to utility theory and decide if false positives are worth the cost.

Incidentally, the hiring example is poor because it's just not the same situation at all. The fact he's equating it to the other scenarios maybe says as much about the real problem as the scenario does itself.

Not to mention that "take two aspirin and let me know if it doesn't get better" is probably literally the best way to handle a case that could be something serious, but there's no way to feasibly discern yet whether it is something serious. Obviously the specifics of this type of scenario can change over time, e.g. if MRIs or CT scans became very cheap and easy to administer, I suspect they would become part of these routine exams.

Think about policing. 99.9% of people are innocent so therefore we don't search and arrest people at random. And when we do people rightly flip out.

Agreed. I'm a hypochondriac who can manifest all sorts of symptoms simply by reading them on the Internet. To my happiness, I finally found a doctor who calls me on my bullshit. As a result I've had fewer unnecessary tests and medications over time. Maybe at some point we'll both laugh off something that actually is serious. Which one will impact my longevity more? I can't tell you.

It's kind of like driving. You can become increasingly lackadaisical because you haven't had an accident recently, which invites accidents. You can become an excessively nervous driver because you perceive all possibilities, which invites accidents. Pretty much everyone who stays on the road an appreciable amount of time develops some balance between those two extremes.

Homer: Not a bear in sight. The Bear Patrol must be working like a charm.

Lisa: That’s specious reasoning, Dad.

Homer: Thank you, dear.

Lisa: By your logic I could claim that this rock keeps tigers away.

Homer: Oh, how does it work?

Lisa: It doesn’t work.

Homer: Uh-huh.

Lisa: It’s just a stupid rock.

Homer: Uh-huh.

Lisa: But I don’t see any tigers around, do you?

Homer: Lisa, I want to buy your rock.

[0]: https://youtu.be/xSVqLHghLpw

One of the amazing subtleties in this article... if you're a fast reader you'll stop paying attention to the stories as they go through... kind of amazing that this article is layered like that.

This article could be replaced by a rock with the words "Heuristics are valuable but will bite you in the ass" chiseled on it. /s

If he finished with that line with a photoshopped rock it would've been great.

I developed a rock inside my head, and the rock said "THESE PEOPLE WILL ALL FALL AFOUL OF RIGID DICTATES WRITTEN ON A ROCK."

Second-to-last story should have been about a person that did not fall afoul of the rock. Then in the last segment ask the reader if they developed their own rock.

Indeed, I was beginning to wonder if I could replace the blog with a rock.

I don't understand how articles like this get upvoted. Who is getting value from this?

This reads like some generic LinkedIn CEO post that sounds deep on the surface but actually means nothing.

It seems to make its point pretty clearly to me: "just naysay" is a strategy that is extremely effective/accurate in many domains, but provides no actual value compared to people attempting more honest evaluations or predictions.

I just realized it's possible I'm being whooshed by your comment.

I don't think he's being meta, just obtuse.

Well he also claims that the rock has higher Brier scores than all the futurists, which would imply that in this example the futurists aren't actually adding any value with their attempts at honest evaluations and predictions.

I mean, this is a weird example, but i feel like it's like Scott Auckerman says in Comedy Bang Bang, when he introduces the show. He'll do the welcome, and explain the show, and inevitably some guess will retort that this is silly, since everyone knows what the show is about, but he constantly responds with: "every episode is somebody's first episode."

These ideas that should be obvious to anyone who's studied advanced statistics, or formal logic, or read some books about extreme events, all likely already know, but the fact is, not everyone... better yet most people have not every studied these things.

These ideas are inherently interesting, and every year, there are new people coming of age that are introduced to these interesting ideas via an article like this, and then it'll get upvotes. The world is like a fire hose of young people. Add in a popular author who will likely get attention anyway, and here we are at the top of the feed.

But it's not interesting, it's not introducing anyone to advanced statistics or formal logic, it's not showing any real world uses that can be applied by anyone coming of age.

It's just generalized parables by someone not in any of the fields or positions mentioned, some weak conclusions, and a "Heuristics That Almost Always Works" book title.

I mean most people I know that are "very smart people" including myself regularly exercise the cool customer vibe of "yea right, that'll never happen." I think it's a good parable. I mean, it's not really important, and I didn't learn anything, but it's a good reminder that "probably not" is a lot different than "definitely not."

It's only a good parable if you don't follow the author to the conclusion. The heuristics almost always work. Full stop. You should listen to them, because they almost always work. You should not in any case work off the assumption that you're in the 0.01% of the time that the heuristic is wrong.

You will get surprised 0.01% of the time, and that's fine. If you don't follow the heuristic you'll be surprised far more often by way of being wrong.

This gets further weighted by costs. If the cost of a false negative is high and the cost of a false positive is nothing, always assume the positive and do whatever is required: check the window, palpitate the whatever, etc. You are certain it is nothing, but the cost of being wrong is so high that you do it anyway.

The author's whimsical point about needing people who buy into fairy-tales isn't useful or valid. You make these decisions based mostly on the costs of being right and wrong, and within that you assume things based on the stats of occurring.

> by someone not in any of the fields or positions mentioned

Author is an MD (doctor)...

Everyone thinks they're the person who's right that 1% of the time about the thing everyone else is conservatively wrong about.

For some context, Scott Siskind (of SlateStarCodex fame) has been a favorite blogger of this site for years, though admittedly this isn't one of his stronger pieces. Some of Scott's observations are super profound and enlightening, some of them are a bit more obvious. I enjoy his prose, so I tend to at least skim nearly everything he posts.

I'm with you. SSC posts are high on rhetoric and low on actual conceptual knowledge/insights. The examples are ridiculously long and somewhat contrived. s.

> This reads like some generic LinkedIn CEO post that sounds deep on the surface but actually means nothing.

I felt exactly the opposite. In my career as an engineer I regularly encounter experts who claim to be so, but offer no qualifications or expertise. Having the ability to respond to this type of stuff is valuable.

In my personal life, I've felt that many therapists exhibit this exact response. They choose to give heuristics and platitudes because, often times, they work. But it means they are giving up the expertise which they claim possession of.

I'm reminded of quite the childish thing by this article: "With great power comes great responsibility." If you claim to be an expert, you need to actually be an expert. I consider this the social contract of expertise and prestige.

This is what Nassim Nicholas Taleb has been writing books about. He calls them black swan events, because if you took a sample of 1000 swans, chances are you'd conclude that all swans are white, but it just isn't so. People tend to round down the probability of very rare events to zero, even when the upside of them is small and the downside is catastrophically bad. Examples: the 2008 housing crisis, Fukushima, and our current supply chain problems.

It's such an annoying metaphor for those of us who live in a country where all swans are black.

It's an annoying metaphor anyway. If you've defined a swan as a particular type of white bird, it's impossible for a black swan to ever come. "Black swan" is just a tautological term for new thing we've never seen before, but pretending to be a term for known thing that suddenly behaved differently.

Sometimes things happen that, in order to make money or cut costs, we convinced people were impossible.

> "Black swan" is just a tautological term for new thing we've never seen before, but pretending to be a term for known thing that suddenly behaved differently.

Right. "Black swan" means "a new thing we've never seen before," but of course few people go around thinking "I will never encounter a new thing that I've never seen before."

Maybe you’re right but I don’t see why this addresses the reply to (at least when I loaded it) the first comment on the post which claims the same thing and is disagreed with by the author.

If you get to the end of the article, the author explains why black swan events are not what he is talking about.

These are NOT black swan events. These are probably all White Swan events (possibly grey swan events, but i'd have to review stuff that i don't want to right now). E.g. High certainty, just low predictability. From the book, when you know the statistics of a rare event, and then the even occurs, it's absolutely not a black swan event.

For an event to be a Black Swan event, you literally need to have no possibly for the event in your deductive framework (e.g. the problem of induction which is what the book is actually about). In every single one of these examples, the possibly of the event occurring is accepted by everyone.

This is why Taleb lost his mind when people started calling the Covid Pandemic a "black swan event," which it was absolutely not. We know pandemics happen, we know about what power law they happen at. The fact we were not prepared at all is a problem of not being prepared for something we know will happen with certainty.

https://medium.com/incerto/corporate-socialism-the-governmen...

When I was doing a postdoc in Germany I would go eat mushrooms in Amsterdam and have the same trip about "exceptional events" over and over again.

Than Taleb wrote that book and I wished I'd written something about "exceptional events".

Then Taleb just coasted, drifted and became irrelevant.

Lol, sure, starting Universa after basically inventing tail-risk trading strategies is definitely "becoming irrelevant."

We know pandemics happen but we have no idea which viruses will become pandemic viruses - until one emerges, we're generally confident that there is not an imminent pandemic.

After SARS and MERS, the risk of another coronavirus-outbreak was well established.

Neither of those became pandemic viruses though. And the specific year over year risk was essentially unknown until a new virus emerged at such point we had a couple of months to determine what we thought would happen.

That's the specific event risk: pretty obviously if we had maintained effective pandemic response measures, and maybe focussed on general infectious agent spread control measures as a society (i.e. a year over year goal to reduce influenza cases, update building codes to require less touchable surfaces to navigate) then we'd be better off then we are.

The possibility of a pandemic caused by a coronavirus was well recognized before it happened.

Who is we? The CDC is actively watching for pandemic signals all the time.

We know pandemics happen, we know their rough power law occurrences. We know the most dangerous vectors of transmission. We can prepare for them. We typically don’t.

Just look at all the aging housing infrastructure on the California coast. We know there will be major earthquakes and we know how often they happen, yet the general populace cares more about how pretty the historic buildings look, even though we know they will kill people.

These are not black swan events.

IIRC, a black swan event highlights the problem of induction when there is _no_ prior event that you can use to learn from. Eg: humans thought all swans were white until the first black swan was encountered. So not just exceedingly rare but outside what you know AND extremely rare. Eg: Neither the housing crisis, Fukushima, supply chain crisis, nor the pandemic are black swans (they were seen before and was predicted/theorised) but the internet is.

I think it's what he calls "fat tails" : events happening at low frequency at the tail of the probability distribution, but which have a significant impact.

Not black swans but another and related topic he writes about called ergodicity.

You'd do a lot better with one strong example rather than seven weak examples.

For instance, if you are interested in Bayes Theorem like a lot of rationalists say they are, you could talk about the medical test which is 99.99% accurate but for which 90% of the positives are false positives.

Fortunately I have a rock that says "the next example will make the same point as the example I just read" so I saved a bunch of time.

Drug testing is a good example for most people to understand why Bayesian thinking is relevant.

https://www.mun.ca/biology/scarr/4250_Bayes_Theorem.html

Imagine that a driver gets hit by accident. He's tested as part of company policy, and tests positive. He gets fired, even though the test only really tells us there's a 33.2% chance he was actually using the drug.

Real world drug tests are a lot worse than 1% false positive and false negative rate.

Every time someone gets fired for a positive test, or loses custody of their kid, or so on, it reinforces whatever statistics are being collected as if the test were a ground truth. They're hardly ever questioned, and there's usually no recourse without an expensive legal fight.

The false positive rate for drug dogs is higher than 40%, for contrast. When a dog "alerts" its worse than a flip of a coin. All that matters is if an officer feels like fucking up your day.

Testing used in situations that are legally significant in people's lives should be required to reach a statistically valid threshold of accuracy, like 99.999% of the times this process is performed, it matches reality. A high sensitivity and high specificity aren't enough, but they're framed as highly accurate and reliable by often well intentioned people who simply aren't thinking in a Bayesian way.

>All that matters is if an officer feels like fucking up your day.

This is what most people don't seem to get. Devices like the ADE 651 or the GT200 were bought by the thousands by law enforcement agencies worldwide, not because they were stupid, but instead, so they could have another "data point" against you that they can use at their discretion.

"Sorry, this dot blinked three times so I'm gonna have to detain you: It's standard procedure, I'm only doing my job."

What's relevant isn't whether a technology or forensic discipline is good, just whether courts will accept it.

Antonin Scalia (in)famously commented in one of the Supreme Court's dog-sniff 4th Amendment cases that obviously the police would want dogs that didn't produce false positive alerts, since they wouldn't want to waste their time searching where there were no drugs. The resulting caselaw sets up a situation where a dog can be wrong over half the time and still be used.

The concept that "probable cause on four legs" would be used simply in order to get to search where they otherwise couldn't was apparently unthinkable.

Just like McConnell and anti corruption legislation. Nobody in the senate was corrupt, so why in the world would they need rules against corruption?

The flawless logic of our leaders is astonishing.

Animals are great at reading people emotionally. Given that the handler has some subjectivity odds are pretty good the handler perceives the dog alerts if the cops are themselves suspicious.

> You'd do a lot better with one strong example rather than seven weak examples

Tend to disagree. It's easy to dismiss one example as "well, medicine is special because XYZ." Multiple examples are the core aspect of showing a general pattern.

He could probably have stopped at 3, 4, or 5 though, not 7.

I haven't gone through the whole thing, but the point seems belabored and superficial tbh.

I imagine the whole piece is essentially a comment on having a discriminator with great true negative rate and terrible true positive rate in a context where there is a large class imbalance (very rarely do positives occur). In real life this is quite easy to account for (just fill in your confusion matrix and see how you stand). I also strongly suspect that it doesn't really happen that much. People do have a conscience, professional pride etc. At the very least they will get bored and actually do smth different from time to time.

> The Security Guard [...] The only problem is: he now provides literally no value. He’s excluded by fiat the possibility of ever being useful in any way. He could be losslessly replaced by a rock with the words “THERE ARE NO ROBBERS” on it.

At the very least such a security guard would act as a human scarecrow. More realistically, the guard would actually look for robbers from time to time, if for no other reason then because if he misses them he might be out of a job.

> The doctor [...] “It’s nothing, take two aspirin and call me in a week if it doesn’t improve”

In my experience this sums up the Dutch (country where I currently reside) medical system quite well :)) Somehow they manage to have good health results.

EDIT: moved concluding paragraph up.

A big straw man.

In real life some people are more or less diligent about their jobs, and more or less contrarian, and have different expertise, strengths and weaknesses.

Each of the vignettes portrays the counter position as stupid (literally using a rock as a metaphor).

The reality is much different. In each of the cases there’s an argument to be made that the proposition was flawed- the security guard never finds anything but instead of just not looking anymore, maybe they propose installation of cameras. The volcanologists aren’t very helpful if they don’t have predictive value - if they are always waffling then they are no more useful than the rock cult. And if they are over-activated, then they run the risk of “boy who cried wolf” or of being dismissed because too frequent false positives cost the rest of the society too much.

Overall I think the essay is shallow and not a useful treatment of the subject.

I think you're correct in thinking the situation is complicated, and wrong in thinking the author disagrees! Each situation is subtly different, and some seem more wrong than others. Regardless, a security guard recommending cameras is different from a rock precisely because they suggested that!

What I really don't get is how this article is getting so many upvotes but the comments are unanimous about how vacuous it is... Unless Aella asked her "fans" to vote it up or something. (I for one manufacture my flying monkeys with a statistical model but it's been a really long time..)

Oh man, next time I need an illustration of how the "villians/heroes" narrative can completely destroy any complexity in a human being, I now have a very well written example!

These people don't exist. Not a single one. Every time you're tempted to think someone fits one of these roles, remember how complex you are, and then think about how complex you likely appear to others. The rift you can see between your internal self and external self is there for everyone, in every situation.

Fun read, though!

I finished the piece noticing times that "heuristics that almost work" were a genuine temptation for me to adopt and having more empathy for people who end up adopting them. If you really don't feel like you could ever be tempted to become lazy in the way that the piece illustrates then I admire your integrity but there are actually many of us who are tempted to reduce the rigor with which we investigate things when we know what is almost certainly going to be the answer.

The mere fact that you think of using these kinds of heuristics as "lazy" and "acting without integrity" is itself an example of why this article creates a moralistic dichotomy that's both completely untrue and entirely unnecessary.

Don't fall for it. Laziness is a lot rarer than you may believe, even in yourself. People don't just "do bad things" very often for the sake of doing bad things, there are almost always reasons beyond the obvious.

Not seeking a reason beyond the obvious IS what the concept of Laziness means, so its not rare at all. When you don't care if there is a reason for not doing something, you call it Laziness.

It's not a temptation. It's a necessity. The mere act of getting off bed requires the heuristics that "I can probably safely get of my bed". Everyone uses heuristics that almost always works, because there's no practical alternatives. A doctor who doesn't use heuristics will need to test all patients for cancer, because there's always a chance that there's a tumor behind any symptoms.

The cat owner

Timmy has a shot gun and a cat. Every time he hears a crack of the floor, he picks up his shot gun and checks the door. It's okay; it's just his cat walking around the house. But this is merely a heuristics, he thinks to himself. Being a rational person, Timmy still rushes to the door with his shot gun every time he hears a crack in the floor. The lack of sleep has invited boldness to his head, but at least so far, he has kept out every bugler who didn't show up to his door.

The city lover

Mary and Susan live in a city, by a busy road. There is a crossing, with proper traffic lights and timers. Susan crosses the road when the green light is on, because that's when all the cars whose route clashes with hers are not moving. But all of them? Mary questions. So far, Susan has been fine, but Mary is a well-informed rationalist who doesn't believe in heuristics. She looks left, and then right, and distrusting her own sense---because we all know that human vision is not 100% reliable---she looks left again, and then right again. She rubs her sore eyes, and decides to go back home. The road is too dangerous.

---

Life would stop being plausible, if we always look out for the 0.001%. In some cases, the cost of not trusting heuristics is low, but then, that heuristics would be quite useless.

Some experts work on the 0.001% case and they are warranted to be paranoid; but even then, their alertness can only be confined to a narrow application. Everything else in life is still based on heuristics.

"Fifty thousand years ago there were these three guys spread out across the plain and they each heard something rustling in the grass. The first one thought it was a tiger, and he ran like hell, and it was a tiger but the guy got away. The second one thought the rustling was a tiger and he ran like hell, but it was only the wind and his friends all laughed at him for being such a chickenshit. But the third guy thought it was only the wind, so he shrugged it off and the tiger had him for dinner. And the same thing happened a million times across ten thousand generations - and after a while everyone was seeing tigers in the grass even when there weren’t any tigers, because even chickenshits have more kids than corpses do. And from those humble beginnings we learn to see faces in the clouds and portents in the stars, to see agency in randomness, because natural selection favours the paranoid. Even here in the 21st century we can make people more honest just by scribbling a pair of eyes on the wall with a Sharpie. Even now we are wired to believe that unseen things are watching us."

―Peter Watts, Echopraxia (2015)

Nassim Taleb would love this quote.

"Yes, the VIX is overpriced. Although you can't make a living by shorting it. Both statements are true at the same time."

There was that time I "invested" in an inverse VIX ETF. It was doing great until the day the VIX spiked and it got liquidated.

Funny enough after the liquidation I had about as much money left over as I put in originally. So I didn't lose much skin and got a great story to tell.

Surely there's some point where the ability of the individual to not expend energy inefficiently becomes balanced with the ability of the organism to evade tigers. It's clearly not true that the individual who thinks they hear a tiger more often (without bound) will be more successful at surviving/reproducing.

Sure, but that balance point still involves a lot of needless running before the energy costs become an issue.

This is really just an empirical question in ethology, but if I had to guess I'd say that prey animal in fact don't do a lot of needless running. On the contrary, I'd suspect their sense do a pretty darn good job at quickly determining the risk of predation.

I regularly see prey animals doing a lot of running from me when I am not a threat to them. Even remote observation of fairly large prey animals like goats in their natural environment show them seemingly randomly get spooked.

Don't be so hard on yourself.

Natural selection doesn't always favor the paranoid. Running like hell requires a lot of energy and if we're overly paranoid, we might just run ourselves to death whenever the wind is wilder than usual

Who says humans aren't running ourselves to death at the sound of the wind right now?

Context for the pedants:

This is dialog from a sci-fi novel about aliens and space vampires. It is figurative language to illustrate a concept.

Great novel full of evolutionary psychology trivia:

https://en.wikipedia.org/wiki/Watching-eye_effect

https://www.newscientist.com/article/dn9424-big-brother-eyes...

The parable of tigers in the grass might demonstrate a selective pressure towards caution, but doesn’t demonstrate any benefit towards ascribing agency to natural phenomenon.

The agency of the entity which caused the rustling is somewhat irrelevant for the hunters. The important belief for their survival is that this rustling entity is dangerous. If they hold this belief but attribute the behaviour of the entity to randomness, they’ll do just as well.

99.9% right isn't great: that means 1 in 1000 is bad. If this were a medical condition, that'd be a pretty high rate.

I work with large-scale testing, and we use a measure called "DPPM", or Defective Part Per Million (manufactured). For my team, a DPPM in 10s is noise/acceptable loss, ~100 we keep an eye on, and 100s-1000 is cause for investigation. Going back to percentage, that translates to 0.001% fail-rate is noise, 0.01% we keep an eye on, and 0.01-0.1% is cause for investigation (and 1% is "Stop The Line").

My point is that the "percentage" scale of failure/risk is one tuned to human perception: "1 in 100 is nothing!", but at the scale of events that we deal with in many areas of our modern life, it's actually huge.

Or: don't use humans for dealing with large numbers. Alternatively, exercise them with the occasional positive result to keep them on their toes.

I've generally stopped using percentages to communicate these days because it's apparent people don't understand them. Someone will say 80% like it's a sure thing - until you point out the failure rate is 1 in 5.

This is an interesting article. In fact I really do think all any of us do is heuristics - where "A heuristic is a mental shortcut that allows an individual to make a decision, pass judgment, or solve a problem quickly and with minimal mental effort."

I think this is how all of us are when we are doing something.

I also think this is existence is a free will experience.

That might seem incompatible, but I think our free will is engaged in selecting what heuristics we can choose. Alternatively, you can say we are programmable creatures, but we get to choose what programming we run.

I would also say, that the heuristics/programming we mostly run is that which has been provided to us by default - a consequence of our education and situation.

Not many of us take the time to review our programming - or to engage with the more 'meta' elements of our experience. I daresay, that the principles we run are not really coherent. Eg, we are reasonable, we do the right thing, we also take shortcuts to get what we want, or save time, etc. These are examples of principles that cannot all be true!

If you were to ask me, the best thing we can do in this life is to consider our values. Are these truth and reason, personal gain, saving time, etc. What means most to us? And the review the heuristics/programming we run according to those values. At least we have a foundation for our programs that we selected ourselves, rather than running the programs that were provided to us by default!

This sounds like learned helplessness. I have seen it in difficult R&D jobs where the hitrate from experimenting is very small, and people subconsciously give up. That's why hiring extremely motivated and intrinsically curious people for these roles is important.

Article was fine, until the Rationalism shout-out at the end (the link to LessWrong was a major red flag). That community is Very Concerned with race and eugenics and genetic determinism and are Just Asking Questions and is largely convinced it knows better than all the experts. There’s a reason Scott Alexander deleted his blog rather than let the light of the NYT shine on it. It would’ve painted him in a very unflattering light, so he decided to “become the story” instead., and his surprise gambit was a massive success at the time. Here at HN the pitchforks were out for the New York Times, and I confess I was even carrying one of those pitchforks.

The rationalist community operates at its best in the semi-dark where its ideas and culture aren’t subject to heavy scrutiny. Would be nice if an intrepid and famous journalist would dig deep but there are dragons lurking there and the operating individuals are exceedingly clever and good at camouflage.

Anyway the rationalist community is a bit of siren song for nerds. I fell for it once and took the bait, so I’m here trying to be more than a mere rock and warning other individuals to be skeptical as they read SA’s seductive prose. Don’t drink the epistemological poison of the rationalist community.

Generally agree with the article, though the reverse is obviously, "Heuristic which almost never works", and all the stories are about the 1 out 100000 times the weirdo was right and everyone lauded them as a genius but really they just got lucky the once.

There's 100000 or more weirdos out there claiming all sorts of spurious stuff, it's only natural one of them would be right

The reverse is heuristic that works 50% of the time.

If you have heuristic that works 0.0001% of the time - it's almost as good as one that is correct 99.999% of the time. You will notice and learn to just invert it.

I feel like a lot of this is just... bullshit. Yes, if a security guard literally decided to ignore every sound as the wind they would probably be useless - as long as their role is literally only fast response. Except they don't, and it isn't. Their role is just as often to politely escort someone out, or remind someone of their manners. So there's that but also, maybe over time the security guard doesn't ignore everything as wind, maybe the security guard...learns quite well what wind is like. If I hear a vague rumbling on a monday I know it's my neighbour bringing the bins out. Similarly, the security guard probably can go from 5 false positives a day to 0.01 false positives a day without increasing their false negatives at all.

It's like, it's a nice thought, but if you really apply attention to it, it doesn't hold up for long.

>Fast, fun to read, and a 99.9% success rate. Pretty good, especially compared to everyone who “does their own research” and sometimes gets it wrong.

I would wager the "do your own research" crowd hits much lower than 99.9% success rate, so what's the argument here? The person who accepts mainstream view - which is apparently what this article calls a skeptic - is no better than a rock. But that actually, the people who contradict the skeptic are no better than a rock thrown in a glass house.

In my experience though, the skeptic is far more likely to be a "skeptic" of the mainstream view and to advertise themselves as that (see: literallly thedailyskeptic.org) and in reality, they're taking the bad side of the bet, by this logic you're 99.9% right by dismissing contrarians and accepting the mainstream view. if you're constantly endorsing the contrarians, you're basically taking the 0.01%. And this isn't theoretical, the skeptics I listed earlier literally posted an article today telling us how global warming is fine, because the earth was warmer... 50 million years ago when no human life was viable. The problem with skeptics isn't their skepticism, it's where they choose to apply it.

There is value in there being contrarians, in order to hold people's feet to the fire, but that doesn't really apply if they're just bringing up dumb arguments and are no more complex than the people they're questioning.

Two words: precision and recall.

Accuracy is a very bad measure of any classifier performance (be it a machine learning algorithm, or a human expert) if the sets are not well-balanced. See https://people.inf.elte.hu/kiss/13dwhdm/roc.pdf for RoC.

One could add:

The Barking Dog

Barks at everything all the time, people learn to ignore it. Then, when there's real danger, no one cares, providing literally no value and becoming only an annoyance.

It's kind of like a dual for the security guard one.

That's only true if the burglar knows that the owners ignore the dog's barking.

I think the best known illustration for this concept is actually called "The boy who cried wolf"

Yes, but one important difference is that on OP's examples there is some sort of system that you put in place for a specific purpose, and it fails to do so in practice.

It would be akin to that lying boy being the officially appointed wolf-spotter for his village.

There's some very interesting discussion here and in the comments. Many have pointed out the similarity of ideas to Taleb's Black Swan, and extremisation, which was also brought up in Superforcasting by Gardner and Tetlock.

Instead of such a discussion, I'd like highlight a book that provides the "oposite" perspective: Gerd Gigerenzer's Rationality for Mortals. Gigerenzer presents the an anti hyper-rationalist perspective for heuristics, arguing that they're not only human, but necessary and inevitable for time and compute bounded beings.

> Whenever someone pooh-poohs rationality as unnecessary, or makes fun of rationalists

Fun and clever article, but for it all to land on that was jarring and disappointing. Preaching to the choir I guess.

He’s not complaining about those criticizing rationalists but warns fellow rationalists to not fall into this heuristics trap themselves.

1. I think the author forgot the term "Anomaly Detection" and is trying to reinvent it. Also, Anomaly or "Sensing something is wrong" is one of the most basic human instinct.

2. >By this time they were 100% cultists, so they all consulted the rock and said “No, the volcano is not erupting”. The sulfur started to smell different, and the Queen asked “Are you sure?”

Even the queen deciphered that something is wrong without any volcano knowledge. The author itself is providing an example of human instinct without acknowledging it,

3. The author assumes all guards as the same when in fact they all are different individuals. Sure most of them might be lousy at their jobs but there will be some who understand how rare the "robbery event" is and so will still look when there's sound.

4. The examples suffer from cold start problem. What if robbery happens the first month of a new guard. Will he still be asleep? If not, then Utility (hiring a guard in all cases) > Utility (not hiring a guard).

5. As another commentor mentioned that the value of having a security guard visible so that 99% of the robbers who might conceivably want to rob decides to go someplace else instead.

6. Contrarionism is seen as a virtue by certain people hence "I don't like this generation's music" and "Popular thing bad" phenomenon. Also, they make sure to be as loud as possible whenever their contradiction is right. This helps humanity in mentally modelling rare events.

All in all, the author is underestimating the capabilities of humans and humanity.

This is like the turkey on the day before Thanksgiving. "They fed me today. They fed me yesterday. They fed me the day before that. Of course they're going to feed me tomorrow!" Except they aren't.

So, yeah. Our heuristics fail on black swan events. There needs to be a balance between "trust your heuristics" and "watch out for black swans".

This reminds me of some of the studies illustrated in The Undoing Project (story of Daniel Kahneman and Amos Tversky, leading behavior psychologists) where it showed that incredibly simplistic algorithms could be doctors in predicting whether a tumor was cancerous or benign.

They asked the doctors what factors to look for, the doctors could accurately describe what to look for. They then created a program that did exactly those things and pitted it against the doctors expecting it to perform poorly and needing iteration -- but it beat the very doctors that described the process. It beat most doctors that describe the process.

More data here: https://fs.blog/algorithms-complex-decision-making/

A boring sounding book details more examples: https://www.amazon.com/gp/product/0963878492/

This essay, but not ironically.

If you think you have something that no one else in the world has noticed, you're probably wrong. You're going to need a LOT of evidence to prove yourself right. Lots of people & companies spend years, decades even, proving themselves right. You're not going to do it overnight and you're not going to do it with a wikipedia article.

I feel like this article does a much better job of reviewing Don't Look Up than the article that passed through here last week. As an allegorical reading, it calls out the major plot points of the film and hits on the ulterior motivations of both those using heuristics to naysay experts, and the experts who inevitably fall out of grace because of them.

What I'm getting here is we should listen to the 0.1% when they're right.

I'll leave it up to the reader to determine when that is

Somehow this article seem to make the mistake that "experts" are all just spending their time making predictions, and not spending anytime doing research and experimentation to acquire more data and evidence.

Most experts in most fields spend their time doing research and experimentation, in order to acquire knowledge and build a corpus of understanding that makes that 99.9% into a 80%, a 50%, a 20%, a 1%, etc.

The only "experts" making projections tend to be fake experts, they'll actually be policy makers, investors, marketeers, etc. (yes sometimes they'll hire an expert statistician to waste his time help them with such foolishness)

And ounce those people enter the game, they'll pester the real experts ad nauseam for estimates and for predictions, and at first the expert will say well more research/experimentation is needed. But the fake experts will say, ok, but ballpark, just an estimate, what do you think is most likely happening here? So the expert will say, ok, give me some time to really run the numbers and make sure at least I'm giving you accurate statistics. But the fake experts will pester some more, I need it by end of day, just tell me now, why would it take you so long. Eventually the experts will just make it up so that the fake experts leave them alone and they can go back to doing real work like research/experimentation/development, etc.

And this in my opinion invalidates the claims in the article. Because those experts cannot be replaced by a rock. The reason they'll be doing the same work as the rock, is because non-experts are going to want them to do so, by asking them the question the rock could answer, and refusing any answer that is probabilistic, they want certainty, not possibility. And to those people, it matters very much that the expert said so, because in the expert they trust, in the expert they can scape goat their failures, they did not make the decision, the expert did. A rock does not provide them with plausible deniability.

You are webmd. You take in symptoms from people with mild discomfort. You give a wide range of causes and diseases based on the symptoms. You include both spectrum of extreme events. You are worthless as a medical advice provider.

---

You are a talking head in CNBC. You have seen only two global financial crisis in your life time. You talk about a global recession that is about to happen every week. You try your best to keep everyone in the edge of their sit. You never talk about the how market is booming. You are on doomsday patrol.

---

Why does modern contrarian black swan enthusiast always talk about catastrophic events that has incredibly low likelihood of happening? You know what are the other things that have low likelihood of happening, winnings from gambling and lottery. The sentiment is that being conscious about risk is smart unless you are talking about exploiting tail events for profits.

Scott actually did a great writeup on Webmd specifically: https://astralcodexten.substack.com/p/webmd-and-the-tragedy-...

My favourite one is seeing IT admins plan for “this almost always won’t fail.”

Substitute “this” for SAN array, core switch, or entire data centre.

I’ve had someone argue with me at length that simultaneous multi disk failures in a RAID5 never happen.

Two weeks later it did and the main SAN disk array went up in smoke.

It is useful to remember that "almost never" is a synonym for "sometimes".

“Scientists have calculated that the chances of something so patently absurd actually existing are millions to one. But magicians have calculated that million-to-one chances crop up nine times out of ten.”

Terry Pratchett

I think a lot of folks are missing the point (and the author could have been more clear).

The point is not that bad heuristics are bad, but to think about when heuristics should be used and what value they add.

In the examples, heuristics shouldn't be used to reduce probabilistic occurrence to binary likelihood before deciding to act. Decisions should be informed based on the actual data when available. Application of a heuristic results in a loss of information, which reduces accuracy and applicable scope. Sometimes this can be entirely defeat the purpose.

Perhaps the recommendation is that if you are tempted to use a heuristic, stop and ask if it is necessary, and what you stand to gain from using it instead of other data or new analysis.

Some of these examples are wrong.

Both Skeptic and Futurist provide value for risk-averse people, as they help to explain away the changes. That’s why lots of people read these skeptics and futurists, because their message is predictable and brings very expected emotions every time.

Example with Interviewer assumes finding diamonds among low-promise CVs has value for a company — does not work that way for companies with huge hiring funnels. Interviewer that focuses on effective but imperfect screening will get far in these, all the while bringing lots of value to the company.

Article also simplifies these roles down to a single, almost binary, function. Which is pretty ironic, given the idea this article tries to communicate.

My oversimplification/takeaway: When false negatives are very rare and their costs very high you can't rely on an empirical assessment of a detector's accuracy, it will be near 100%. Because the costs of false negatives are so much higher than the costs of false positives you could focus on false negative rate, but there's not enough data points and the costs of getting a single false negative are too high to estimate this empirically. You need to audit the procedures to determine if the detector can reasonably detect the thing you are interested in.

The problem here is "the bigger picture". The security guard can improve his heuristic by studying criminology (or rather just talking to other security guards who have been robbed).

what sounds were made, are sounds of the wind different to breaking glass? Have burglaries been on the increase lately? When did he last check the car park was there a suspicious car there?

What I am trying to say is that each of the examples given is really just a strawman - in each case looking for more data - you can spot truth in the data

Eh... yes, this post is attacking a straw man, but that's okay, since the post is explicitly about attacking a straw man (spelled "rock" in the post)?

Hang on - did I read a sarcastic article and not notice?

> The only problem is: he now provides literally no value. He’s excluded by fiat the possibility of ever being useful in any way. He could be losslessly replaced by a rock with the words “THERE ARE NO ROBBERS” on it.

Not true. His value is in being there. He's not there to stop robbers, but rather to stop squatters or mischief makers who are looking for something to do (in which case there will be a lot more than just some odd noise). Nobody in a first world country is going to rob a pillow mart.

> Her heuristic is right 99.9% of the time, but she provides literally no value. There is no point to her existence. She could be profitably replaced with a rock saying “IT’S NOTHING, TAKE TWO ASPIRIN AND WAIT FOR IT TO GO AWAY”.

Also not true. If there's a real problem, it won't go away and the patient will return, either with no improvement, or a worsening of symptoms.

> It cannot be denied that the employees he hires are very good. But when he dies, the coroner discovers that his head has a rock saying “HIRE PEOPLE FROM GOOD COLLEGES WITH LOTS OF EXPERIENCE” where his brain should be.

But he hires good people for his company. So regardless of the soundness of his methods, he is providing value to that company.

Followed by a bunch more contrived examples and downright black swan events.

Bottom line: Yeah, they're using heuristics that will sometimes be wrong, but life itself is about compromises. There's no such thing as perfect information, so we use heuristics out of necessity. Worrying about the unlikely scenario is a recipe for paralysis.

> > Her heuristic is right 99.9% of the time, but she provides literally no value. There is no point to her existence. She could be profitably replaced with a rock saying “IT’S NOTHING, TAKE TWO ASPIRIN AND WAIT FOR IT TO GO AWAY”.

> Also not true. If there's a real problem, it won't go away and the patient will return, either with no improvement, or a worsening of symptoms.

You could still replace the first visit with a rock or automated response.

> > It cannot be denied that the employees he hires are very good. But when he dies, the coroner discovers that his head has a rock saying “HIRE PEOPLE FROM GOOD COLLEGES WITH LOTS OF EXPERIENCE” where his brain should be.

> But he hires good people for his company. So regardless of the soundness of his methods, he is providing value to that company.

He didn't provide any more value than a rock without a salary though.

> You could still replace the first visit with a rock or automated response.

Not really, because the doctor would have to receive a first visit before a second visit, and would evaluate how to deal with each one. This is an oversimplification and it doesn't work.

> He didn't provide any more value than a rock without a salary though.

He hired people. Rocks cannot hire people. So he absolutely did provide more value than a rock. In fact, a college degree is a pretty decent heuristic all around. It'll miss a lot of good candidates, but overall the candidates who pass his filter will be mostly sound (as was mentioned in the example).

The overarching problem is that these are oversimplified examples with oversimplified explanations to support his oversimplified conclusion. Real life doesn't work that way. Real life is why we need heuristics in the first place.

I think stones get an implicit bad rap here, I mean people should listen to them more. In Fukushima there were stones which said:do not build below this level; that was ignored.

https://www.smithsonianmag.com/smart-news/century-old-warnin...

Great article, really enjoyed it. I second the Talib connection.

But anyways... Does anyone else struggle with Substack's typeface, specifically it's width and spacing between characters? I'm a bit of a typeface nerd, and I genuinely like or enjoy most of our common fonts. Substack is the only site that I find the typeface to significantly affect the reading experience.

I tend to agree with you about the typeface they use, Spectral. I think it's a combination of factors: loose tracking designed to permit slightly over-emphasized differences in weight (which were a design goal for the font) and somewhat selective replacement of inktraps from the inspiration face (Elzévir) with angular serifs, which creates a bit of a visual discordance. I don't find it unreadable, personally, but it does call attention to itself.

Very well put. I agree with your analysis. There's a lot of things I like about Spectral, and in general I like the "genre" of faces that it belongs to. The loose tracking+over-emphasized font weights is the core of my challenge.

I see this more as intuition than a heuristic. If your are in a field where everything is OK 99.9% of the time, you can't help but have an intuition that there will never be any problems. Correct intuition needs to be fed positive and negative outcomes to be well balanced. If you are in a field where intuition is biaised, dont use it, rely only on data.

Heuristics are literally an evolutionary mechanism, the same reason we don’t stop and observe every blade of grass. If we did we’d be easy prey, and extinct. As would other species capable of the same contemplation.

They’re also very seldom so weighted. 99.9% confidence in anything is so uncommon that it’s not even a marketable confidence. How many nines? Not this many.

And many of the examples in the article are clearly strawman arguments where the exemplary person would never express the same level of confidence. Or if they would, they’d expose themselves as incompetent or a fraud.

I feel like I’ve spent more time and mental energy on this than it deserves, but to round it out: heuristics are intended to be hypotheses, and intended to be tested. They’re not rules. The only reason anyone would treat them as rules is if they’re 99.9% confident. If anything is that sure, you’re either dealing with business that cares about that probability margin or you’re gambling against yourself.

> But then you consult a bunch of experts, who all claim they have additional evidence that the thing won’t happen, and you raise your probability to 99.999%

I lose the line of reasoning here - 99.9 to 99.999 doesn't happen if you don't have new evidence, so why would you raise your probability? or maybe i'm being too literal?

You think the supposed experts have additional evidence, so you're treating their claim to have evidence as evidence in itself. You're unaware that they have no more evidence than you already did.

Reading through the comments shows that basic understanding of probability is crucial. If an outcome of an event is labeled with a probability of 1/10, what that really translates to is that if the event happens 10 times, you can expect the outcome to occur once.

The whole talk about N+1 gives the same probability as N is not right.

The security guy metaphore is false to a degree. The security guy is the reason there are no robberies as he himself reduces the win lose ratios. Given no security guard win ratio for robbers is very high (no witness, just take stuff and go). The company has several security level cameras, alarms, security agency nearby they can calibrate their response to further devalue the win possibility.

There is also society game level above all of that. You can persuade more and more people that stealing things just generates additional costs. Seems we have to allocate resources(private and public) on protecting property and its beneficial for all if we stop playing this game (where reasonable people have to spend money to continously fix unreasonable people's deeds).

In safety engineering, this is called normalization of deviance. I remember in one book I was reading about this, they said that the real problem wasn’t that “everything that can go wrong will” but that “everything that can go right will”. That latter position causes you to erode your safety margin until you explode.

I’ve noticed my behavior is drastically like this. Even if I rationally know I am mortal, I take ever larger risks and each escape gives me more certainty that I am truly lucky and increases the boundary of operation.

The problem is incorrect prior updating. If there is a 75% chance that I will die if I jump off this bridge and I jump and live, then I update the prior too generously. Clearly I didn’t die, so it has to be lower than 75%. Well, yes, but perhaps update more conservatively when the bet is big.

I think my main takeaway is how profitable it would be to be one of these naysayers. Good career move.

I think history has shown it is far more profitable to amplify fringe theories and capitalise on everyone’s fears that we are being deliberately mislead by corrupt authorities claiming false expertise.

Agree with the point the author is making, but here's an added question – is it wrong to always trust these heuristics regardless?

To use the doctor example, over-diagnosing symptoms is as harmful as under-diagnosing them. You cannot prescribe an MRI and other advanced tests to every patient who walks into your door, since (1) they are expensive and capacity is limited and (2) there is a chance you will get a false positive and end up in a worse condition than you started from. Maybe always sending patents home with an aspirin until the symptoms get worse is the right thing to do?

In more generic terms, betting on the 0.1% outcome is a risk, and one that you may not be able to afford to take.

Funnily enough, you could replace this article with a rock that said, "When people are confronted with a very skewed probability distribution, after a while they become complacent and default to the most probable outcome".

Hum... You mean you can replace some text with some other text?

This is the best summary of the article. How is this #1 on HN right now.

I really liked this part:

>You know what doesn’t need oxygen or water or food? A rock with the phrase “YOUR RIDICULOUS-SOUNDING CONTRARIAN IDEA IS WRONG” written on it.

> This is a great rock. You should cherish this rock. If you are often tempted to believe ridiculous-sounding contrarian ideas, the rock is your god. But it is a Protestant god. It does not need priests. If someone sets themselves up as a priest of the rock, you should politely tell them that they are not adding any value, and you prefer your rocks un-intermediated. If they make a bid to be some sort of thought leader, tell them you want your thought led by the rock directly.

Heuristics? For amateurs and lazy folk. We find metrics that reliably measure something, find or make a gauge that measures that metric, then make sure our gauge stays good over time by doing R&Rs or recalibrations.

It is almost rediculous how everything is a subtle bullshit with nice wording.

Yes, neither ivermectin nor hydroxychloroquine had an agent which is relevant at the level of a RNA viruses, a hurricane can be seen long before they are here (unlike earthquakes let's say) and every single social construction, be it a religion or anything of a lesser scale, has distinct patterns and social techniques, such as secrecy, etc.

The world has no miracles, except in Nature, and even those are bounded by the environment.

But, of course, one can get the HN front page these days with such a subtle bullshit.

A pedantic criticism of the title: ‘Heuristics that almost always work’ is a truism, or a tautology. If a heuristic worked 100% of the time, it would no longer be a heuristic, it would be a rule!

But if a heuristic worked 75% of the time, it wouldn't almost always work.

I wonder what the minimum % a heuristic could be said to have? When does a heuristic become just an educated guess?

The problem with articles like this is that they take one good clear simple point, "the existence of experts using heuristics causes predictable over-updates towards those heuristics", and pad it out with so many examples and allegories that, understandably, what the reader wants to talk about is the validity and accuracy of the examples. These, of course, aren't the point of the piece, but the point of the piece is so slight that it hardly bears discussing. Of course everyone looks to the actual content.

Lol, this article can be replaced exacty with a rock with same title.

This is why we have 2727272 self help books that I can't read past chapter 3 as they regurgitate the same idea in every sentence

This is where Bayesian inference really signs.

For the security guard, hearing a single noise is likely to be nothing. However, what if you heard two noises, and the sound of tires outside?

Same thing with the doctor. Most good doctor's I know have a sixth sense, about when something is off and needs further tests beyond just take an aspirin. So maybe the person had a stomach ache, and they had lost some weight, and they were looking a little yellow. All of a sudden the probabilities start looking a lot different.

You would think Bayesian inference is good at integrating multiple information sources but practically you have to model the dependencies between different information sources and even doing a good job of that doesn't save you away from logical fallacies such as "Explaining away". In real life people use Naive Bayes a lot because properly modelling a Bayesian network is hard and trying to learn the network gets you in all sorts of problems -- allow arbitrary dependencies between N inputs and you are talking eᴺ coefficients in your model and you'll never solve it.

This is one of the reasons why people got frustrated with Expert Systems as real-life reasoning requires reasoning with uncertainty and we don't have a satisfactory general way to do it.

The whole point of Bayesian networks is to have something that's asymptotically simpler than "arbitrary dependencies between N inputs" while still being able to model useful scenarios.

The flip side of this, is that if the rock is 99.9% right, and the high-minded rationalist is only 99% right, what does that say about the value of his rationality?

What if the rock's errors are all false negatives, the high-minded rationalist's errors are all false positives, and false negatives are 100 times worse than false positives?

Then I guess measuring raw accuracy was not a good idea in the first place.

The rationalist should still be able to provide unbiased probabilities. How you act on the given probabilities can be informed by the relative costliness of false positives vs. false negatives.

You have to ask questions about expected value rather than just accuracy to figure that out.

Great article and examples.

In the security guard story: I understand the point that the author is trying to make, but it falls short as the security guard adds value to the property owner by just being phsysically there.

By being present the chance of a robbery, vandalism etc is significantly reduced. In most countries the average guard is also not expected to actually intervene with a robbery and instead call police (the US might be an exception here in some cases).

I find it so comforting to believe in Heuristics That Almost Always Work even when I know I ought to employ more scrutiny. It's too easy to jump on board.

The argument in the first example is just wrong. 1) His value might be that it looks like there is a security guard on duty, and that a) encourages customers, b) discourages robbers. A rock cannot do that. 2) He could do that, but he probably won't, because it's boring to just sit there. Once in a while, he will walk around, looking, paying at least a little attention. It makes robbery riskier.

Another one is the "science can explain everything" heuristic, which is true in 99.99+% of cases.

But there's one important case where this is not true and many people are trying and failing to use the scientific approach, unable to realize that "solving" the problem requires a very fundamental change of thinking.

Try to guess what I'm talking about (it should be easy I think).

1. The guard: He is not even there to protect the store. He is there as a deterrent. Nothing more. The number of people stealing from a guarded store is small, but not 0. The number of people killing the guard to steal a store is really negligible. A store without a guard will get stolen, however.

2. The doctor: This is a true one. I did study medicine and I figured that for most of the time, it is fine. But the doctor here serves an important emotional job to support these people. Some people freak out especially after they did some Internet research or a relative went through the same symptoms.

3. The futurist: Not sure why this one was included. There is all kind of crap in the media. None of it is true. No one is also hiring such a guy unless he is looking at the other outcome (tech that will make it). In that case, I think the guy will be worth a lot more than a rock.

4. Same as 3

5. The interviewer: Most executives know this. That's the job and that's why credentials are a thing.

6. The queen: Should read some history I guess: https://en.wikipedia.org/wiki/1975_Australian_constitutional...

7. The Weatherman: Some are lucky and some have busy jobs. That's life I guess?

Someone who has cancer presumably would see the doc. again, and further tests would be run. Studies show that this delay costs minimal lives, and testing immediately would cost more lives due to possible false positives or complications.

And occasionally we get a stark reminder that the "impossible" is actually possible but improbable.

https://knowledge.wharton.upenn.edu/article/why-economists-f...

There was a woman who was not a licensed nurse but practiced as one for about 20 years without troubles until she got caught trying to subscribe to a mandatory course and they couldn't find her professional order ID.

I'd go as far as saying 90% the resources spent on learning and teaching are entirely wasted and could be done away with.

I heard an analyst say, "All models are wrong, some are useful."

After this article I've updated it to be, "All models are wrong, some are useful in certain circumstances."

I would say another heuristic should be something like, "Don't be so quick to dismiss criticism", or "Never over rely on a model."

Sounds like he's describing software architects. I feel software architects follow a slightly more complicated heuristic. They can't do the same thing every time when they draw the line connecting all the boxes. It has to be a little different every time with different boxes and a different set of lines.

Unimportant, but the dad of one of my friends worked his entire career as fire brigade on a military base. During that career, there were only two fire incidents. In both cases it was not during his shift.

A lifetime of providing zero value. But that's not the way I see it, he got paid and this supported a family.

This article uses a trope that is frustratingly common in rationalist articles: It sets up over simplified straw-man versions of real world scenarios and then knocks them out of the park.

In the real world, a doctor who ignores the complaints of every patients will quickly find themselves the subject of malpractice lawsuits. Not by every wronged patient, but it only takes one or two angry patients with lawyers in the family to cause huge problems. Malpractice insurance is expensive for a reason.

Real-world security guards do a lot more than just catch robbers in the act. I've had security guards catch employees trying to remove company assets late at night, catch doors left open, notice faulty security mechanisms that need to be repaired (e.g. door sticks open), and so on. Not to mention the presence of a security guard is a huge deterrent for getting robbed in the first place.

And so on. Yes, there are situations where you can get away with betting on the most common outcome for a while, but unless the people around you are all oblivious then eventually they'll notice.

I don't think there's the need to take the examples in the article hyper literally. You can call them strawmen if you'd like but simplification for the sake of presenting a point (which you seem to have ignored) is necessary.

aka exactly what straw man means

I'm not arguing it isn't, only that the comment does does something similar by ignoring the point of the article.

> You can call them strawmen if you'd like but simplification for the sake of presenting a point

If you simplify away the parts that make the argument invalid, that's literally the definition of a strawman argument.

I don't disagree, I'm simply questioning the benifit of nitpicking the strawmen instead of addressing the argument itself.

And for the robber, crime is a lot more common than he seems to think it is. Metal theft, for example, is extremely frequent: it is rare that an industrial facility would have so little theft that such a proposed heuristic would be reasonable.

If crime were that low, then the guard's position doesn't exist anyway.

So then we're just left with an empty thought exercise with no relation to reality, as an argument for how we should think about reality.

I'm confused. These heuristics don't almost always work. The security guard has a 0% chance of investigating. How is 0% almost always?

If you make a confusion matrix its precision and recall is 0. If it almost always worked then its precision and recall would be close to 1.

You're only counting the positives. When positives are rare, just guessing the result will be negative is usually a really good starting place.

>is usually a really good starting place

No, you are getting misleading results because you have an imbalanced dataset.

This is how you can throw out most of the COVID tests in the trash and say they all were negative and get away with it... for a while!

Does anyone else who is following the FLCCC/Ivermectin/Peter McCullough story find it ironic that they would use this as an example of something we can believe the "experts" on?

Japan, India, and a growing list of countries apparently never got that memo...

The heuristics are actually ok. The only thing is missed is not accounting for adverse events.

Risk = Odd x Impact

In the examples, the odd is very low but the impacts could be fatal. That’s where is NOT OK to ignore. But if the risk is low, you can follow whatever your momentum approach is.

The reasoning in this post is completely backwards: just because a job could be completely replaced with a rock without affecting the majority of cases doesn't mean that the actual practitioners of that job are either completely useless or are themselves on autopilot.

Siskind assumes the latter and reasons towards the former, which isn't aligned at all with what actually happens in the world: we do predict hurricanes and exploding volcanoes, and there's no particular evidence that the average doctor is ignoring their patients. We're all subject to biases and fatigue, but neither of those supports the claim that we're all phoning it in all the time.

Edit: I will also note that "when nothing happens at all, a person can be replaced with a note on a rock" is not an interesting statement to make. Dressing it up with eloquent prose (and he is indeed eloquent!) does not change this, and does not a poignant observation make.

His whole point is that the occasional person who IS "phoning it in all the time" will appear to be very good at their job, possibly better than the people who are really trying their best to get it right.

Where are you seeing the author suggest most people in the described jobs could actually be replaced by rocks?

> Where are you seeing the author suggest most people in the described jobs could actually be replaced by rocks?

I really think it's harder to get more literal than this:

> He could be losslessly replaced by a rock with the words “THERE ARE NO ROBBERS” on it.

I think the point is that the person who inappropriately uses a heuristic instead of doing their job, is in fact, not doing their job.

A guard who adopts the heuristic "there are no robbers" can be replaced with a rock, but has adopted a heuristic that almost always works.

Guards that do not adopt that heuristic cannot be replaced by that rock.

What's the most insidious form of this in software engineering? Some examples: "It's probably the users fault" "It's probably just a transient spike" "It's probably fine to do it the hacky way this time"

Young machine learning practitioners : That is why you should do not trust accuracy when doing a ML model on imbalanced datasets.

And in general, better use precision, recall, f1-score or confusion matrices

I take issue with the first example. It’s not necessarily true that the lazy security guard offers no value. Unlike most of the other examples he affects the likelihood of the outcome.

99.9%% correct heuristic/0.1% role value is a stretch. The author only gave an arbitrary number to make a point.

Everybody knows 80% correct heuristic/20% role value is more realistic.

- said the contrarian

The rationalists’ heuristic is: no one else has experience or facts I need to understand this problem. I can approach it from first principles and my conclusion will be the best.

The doctor example rings true to me, as someone who just graduated from nursing school. Lazy assessments are far more common than we'd like to admit, even to ourselves.

I am completely in love with how this article introduces itself.

Does he really not understand how Brier scores work? If you are unable to distinguish between more risky and non-risky instances (e.g. specify when it's only 99% not happening, vs. usually when it's 99.99% not happening) then of course you aren't adding any value above and beyond the heuristic.

Additionally, getting the base rate right is important when considering lifetime risk and the costs vs. benefits of taking action or engaging in further screening - e.g. missing two or three cancer patients might be worth the benefits of not subjecting large numbers of patients to secondary screening.

I agree with this, with the caveat that you don't want to fall into the opposite trap of believing the people who just use the contrarian heuristic.

The medicine example could have been summed up with the old saw: "when you hear hoofbeats, think horses, not zebras"

Train some machine learning model for anomaly detection, and send those to a human for final investigation/action.

I must be missing something, as this is highly upvoted, but I don't get the point of this submission

The article reminds me of this: https://xkcd.com/937/

Sure, relying too much on heuristics can be a bad idea in tail risk situations.

But other times, they make perfect sense and save a lot of time and effort.

This post reads like a series of straw men created to show that heuristics are dangerous. I’m not sure who is going to argue that heuristics are appropriate in those situations.

These heuristics seem be the same thing: The null hypothesis is the most likely.

This 2189 word essay can be replaced with a rock that says "ALARM FATIGUE" on it and a QR code pointing to this Wikipedia article:

https://en.wikipedia.org/wiki/Alarm_fatigue

No We are not going to die, hit by some asteroid. believe me.

The opposite to working heuristics is the black swan theory.

While the security guard doesn't actively catch criminals, he still may act as a deterrent. In that sense, he's still somewhat useful.

Interesting article nonetheless!

The security guard still provides value because he deters would-be robbers who have no way of being certain that he is using the wind heuristic

One way to solve this is that instead of asking for a yes - no answer, you ask for a ranking, and you disallow equally likely. Is bigfoot more or less likely than telepathy? Is telepathy more ore less likely than the vaccines being dangerous? Are the vaccines being dangeroues more or less likely than some guy achieving cold fusion in his garage?

A ranking forcibly brings the metric away from accuracy (which the heuristic can score well on) to something based around precision-recall (which it cannot).

TL:DR - you can't abstract away the need to make a judgement call every time.

Also if you think horses not zebras, make sure to at least glance the picture for stripes.

people don’t really think this way.

What is the impetus for this writing style that repeats different versions of the same analogy 10 times when one would have sufficed? Surely Substack doesn't have a word count minimum.

I actually found the ramp-up from security guard to sceptic pretty clever. Demonstrate the principle on an easy, constructed case; verify on a real-world example; then present the applications you care about and have somewhat more controversial content. Although I agree that the number of repetitions is higher than optimal here.

A long article is a sign of effort expended, which rationalists value as a sign that a lot of thought has gone into the ideas the article puts forward. (They have a rock in their heads saying "DON'T BOTHER YOURSELF WITH BRIEF EXPOSITIONS.")

I'm not sure, but I see a lot of this in marketing and management books

Repackaged straw men, in the form of several narratives that describe the exact same thing.

I stopped reading the examples after the second one and scrolled straight to the end, to confirm my suspicion that there was actually no useful information in the article at all.

Specifically around the security guard example, the mere presence of a security guard should deter thieves (in theory), so I think the analysis is a little more nuanced than "security guards investigate weird noises".

Unless this all went over my head and that's all sort-of the point of what he's getting at . . ?

He's not talking to you, he's talking to the other people in his pod.

Reply on news.ycombinator.com