> Why is it wrong?
If you look at 200 rivers, it would not be surprising to find something that naturally occurs 0.5% of the time. It is not correct to say that there is a 99.5% chance that this is due to non-natural causes.
If you look at 200 rivers, it would not be surprising to find something that naturally occurs 0.5% of the time. It is not correct to say that there is a 99.5% chance that this is due to non-natural causes.
We're not going to get more information here without someone looking into where the .5% number comes from.
The method of Roe et al. is summarized as follows. Let 1L be the change in glacier length over the past 130 years (∼1.9 km), and let σL be the standard deviation of glacier length due to stochastic fluctuations in mass balance, b, from natural, interannual climate variability. The signal-to-noise ratio is defined by sL =1L/σL . Likewise, sb =1b/σb . Ref. 12 demonstrates that the two are related via sL =γ sb , where γ is an amplification factor that depends only on the duration of the trend and the glacier response time, τ . The probability density function (PDF) for sb is generated by combining the signal-to-noise ratios of the observed melt-season temperature and annual-mean precipitation trends, normalized by the summer (bs) and winter (bw ) mass-balance variability (Supplementary Fig. 1a,b,c), respectively. We take σbw = 0.3 myr−1 and σbs =0.5 myr−1 , based on the observed mass-balance variability at Gulkana Glacier and the analysis of the global datasets of glacier mass balance33. The glacier response time is given by τ =−H/bt , where H is a characteristic glacier thickness, and bt is the (negative) net mass balance at the terminus. We set H=590 m, based on the scaling relationship for glacier geometry suggested by Haeberli and Hoelzle34 and measured cross-sections35; and we set bt =−7 myr−1 , estimated by extrapolating the vertical mass-balance profiles calculated by Flowers et al.8 , thus giving a central estimate for τ of ∼80 years. A PDF is estimated assuming τ follows a gamma distribution incorporating a broad uncertainty of στ =τ /4 (Supplementary Fig. 1d). The PDFs for γ and sb are combined to give a PDF for σL from the relation σL =γ 1L| obssb . This, in turn, is used to evaluate the null hypothesis that 1L| obs occurred due to natural variability. Supplementary Fig. 1e shows our estimate that there is only a 0.5% chance that the observed retreat of Kaskawulsh Glacier happened in the absence of a climate trend.
It's essentially like saying that a pair of dice that rolled a 12 has a 97% chance of being loaded, because there's only a 1 in 36 chance of rolling that high.
Let's assume that when a river gets redirected, a scientist goes and investigates it (which seems reasonable), and is asked what is the chance that it happened by natural causes, or is due to global warming. Based on the analysis that @acover shared with us, we can see that out of a thousand such dispatches, 995 will be due to global warming and 5 will be due to natural variability. So it seems correct for that scientist to conclude "It's 99.5% due to global warming.". In fact, saying that it's .5% due to natural variability is equivalent.
You are probably thinking of the jelly bean situation, where a test is performed to detect something within a sample population. The test has a failure rate, and the population has a base rate of occurrence.
In this case (thanks @acover), they are directly calculating the base rate.
Now, you could also ask "How many times in a year will a scientist be dispatched and be wrong", which is a again a different question (and is closer to the scenario you described).
The paper appears to create a model for a single glacier's retreat. Selects the parameters of this model by analyzing all glaciers. Then determines the retreat of this glacier is unlikely under this model.
The issue is that there is a sampling bias. This glacier is not like most glaciers [I assume]. It was not randomly sampled from all glaciers. It was selected because of a river diversion due to glacial retreat.
What did they do to compensate for this sampling bias? I don't see it.
This is a faulty assumption and is what leads to the wrong conclusion. The probability of 0.5% is for a randomly selected river. That is, if you went and examined 200 randomly selected rivers, 1 of them (on average) would be redirected due to natural variability.
That does not imply that the remaining 199 were redirected due to global warming. It does not even imply that the remaining 199 were redirected at all!
What is needed is the percentage of rivers that have undergone this redirection. Here's a simplified example: If it's ~0.5%, you conclude it's just natural variation. If it's >0.5%, you conclude that something (possibly global warming) is increasing the number of rivers that are being redirected. If it's <0.5%, you conclude that something is decreasing the number of rivers that are being redirected.
Sorry, what is the probability .5% for exactly? The probability a river is redirected under global warming conditions? I didn't think that's what they computed. If it is, then my bad. :) I thought they had computed given that the river was redirected, what is the likelihood it happened due to global warming. Ah well, like I said, the details are the hard part. :)
"shows our estimate that there is only a 0.5% chance that the observed retreat of Kaskawulsh Glacier happened in the absence of a climate trend"
The 0.5% has nothing to do with the river. It is their confidence that the retreat of the glacier could occur in the absence of a climate trend based on their model.
What is the properly worded complement?
If there is not a climate trend, we would expect this to happen with a .05 chance.
If they meant the latter, my confusion is resolved.
Fundamental truth: bayes theorem.
P(evidence | null hypothesis) = P(null hypothesis | evidence) * P (evidence) / P (null hypothesis)
The P-value test determines:
P(evidence | null hypothesis) = 0.5%
= there is a 0.5% chance of the observed evidence given the null hypothesis
The statement "We estimate that there is a 99.5 percent chance that the observed retreat did not happen in the absence of a climate trend."
translates to P(!null hypothesis | evidence) = 99.5%
By Bayes theorem:
P(!null hypothesis | evidence) = P(evidence | !null hypothesis) * P (!null hypothesis) / P (evidence)
We know almost none of these terms. The answer is not as simple as 99.5.
The possible error you identified is with the original paper and not due to a lack of basic statistics education.