Impossibly Hungry Judges (2017)
m.nautil.us
m.nautil.us
The data is easily explained as judges taking clear case of leniency first, then doing the tough ones (that could go either way), and ruling on the clear cases of parole denials later in the session. The judges have hopefully done their homework in advance and have read the case files and petitions, so they come in with at least a rough sense for which cases are in each bucket. There's likely a significant degree of agreement on that metric among the members of the panel, so it's not surprising that cases are heard in that order: clear grants, ones that require discussion, and clear denials.
The lesson learned seems to me to be when the evidence shows a result this large, take a really hard look at your study before you publish it.
A related problem here is that even though the original study was bad for many reasons, it continues to be cited much more than the refutation.
Edit: just the scheduling, people. I'm not claiming the effect isn't due to that sort of thing, just that judges can't easily arrange their calendars so that "good" defendants are in the morning and "bad" ones are in the afternoon.
The daily court schedules were arranged to minimize the amount of time lawyers and prisoners spent waiting in court. Each lawyer would usually represent several prisoners, and would go through all of their clients consecutively. The court would schedule the lawyers and clients starting after meal breaks. After all lawyers and their clients that were scheduled for a block of time had gone through, then they would fill the remaining time until the next meal with cases by prisoners without lawyers.
Cases with lawyers had a 35% favorable outcome rate, while cases without lawyers would have a 15% favorable outcome rate. In addition, each lawyer would present their best case first and worst cases last.
So yes, the daily calendars were arranged such that the first prisoner after a meal break was extremely likely to get a favorable outcome, and the prisoner right before a meal break had only a 15% chance of a favorable outcome.
Not that this is universally true, but it's what I found to be pretty common over the years.
Most really good science got done before statistics was even a thing (starting in the 1930s-1940s). Stats has very little to do with being a good scientist.
Or perhaps science has become so institutionalized that any truly novel ideas are suppressed either directly or indirectly (eg the funding for that type of thing is already maxed out).
Or maybe the truly novel ideas are getting classified: https://slate.com/technology/2018/05/the-thousands-of-secret...
What does random mean? It just means you didn't include every possible influence in your mathematical model of whats going on. Instead you used some distribution of "noise" as an approximation to whatever was really going on.
Eg, if you know everything about airflow, gravitational, electrical, etc forces on a coin throughout a flip, you can predict which side it will land on with near 100% accuracy. Most of the time we don't have info about that available so use the binomial distribution approximation.
>"where effect sizes are gigantic"
Most science has nothing to do with "effect sizes", in fact I'd say concern about "effect sizes" is indicative of very rudimentary science. Advanced science is concerned with making precise and accurate predictions.
In physics, it is fundamentally impossible to predict the outcome of wave function collapses/pick your favorite interpretation's verb. CERN doesn't do statistics on their histograms because they don't have good models...
That aside, does the difference between a lack of knowledge and a lack of predictability really matter? Either way all you can do is calculate the odds.
>Most science has nothing to do with "effect sizes",
True, that language is not used everywhere. However if your department deals with signals that are hard to distinguish from background then you have some way of expressing that idea in your vernacular, whatever it is.
This is one interpretation of quantum mechanics. Plenty of physics is done without any consideration of this.
>"CERN doesn't do statistics on their histograms because they don't have good models..."
I believe the histograms you are referring to were done on the data, not on the predictions. And afaik they actually do lack a good model of the higgs boson mass, since they never predicted an exact mass.
>"However if your department deals with signals that are hard to distinguish from background then you have some way of expressing that idea in your vernacular, whatever it is."
Distinguishing signal from background is the same thing, its rudimentary stuff you do when you can't predict exactly what you are looking for. The key difference is that in advanced science you check for deviations from the model you believe is correct. No one at CERN believed the standard model without a higgs boson they used for the background reflected reality.
If you are in a field where you don't need statistics, it is not because they aren't at play, but instead because they all have -9999 in the exponent and you have implicitly decided not to worry about them.
There is no fundamental uncertainty in GR predictions. There is none in the stefan-boltzmann law, etc. In many cases whatever QM uncertainty exists will be negligible next to measurement error and so is not considered. Acting like QM uncertainty has anything to do with 99% of physics is nonsense.
If you are in a field where you don't need statistics, it is not because they aren't at play, but instead because they all have -9999 in the exponent and you have implicitly decided not to worry about them.
I feel some topics are getting confused here.
Now you seem to be saying "a field that doesn't need statistics" only doesn't need it because the error due to QM is negligible compared to other sources. How do you account for all the advances before QM and statistics?
Discoveries made before the introduction of error analysis turned out to either be lucky or wrong - a pattern repeated every time we all realize that something was being overlooked. It's worth pointing out that the astronomers of yore probably knew that their measurements weren't perfect and that averaging them was a good idea, which counts as statistics.
It makes no sense to be concerned about sources of prediction error that are orders of magnitude smaller than your measurement error. Such concerns would only waste time and money for no benefit. This has nothing to do with "straggling" or expected "effect size".
No one would do this, its once again a "rudimentary science" behavior. People have figured out what we need to know to get the exact relationship between voltage and position.
It's primarily due to "a statistical artifact resulting from favorable rulings taking longer than unfavorable ones".
My idea: our ancestors were opportunistic savanna hunters. Why would we be unable to maintain homeostasis with respect to cognition even without food? Our ancestors would have needed their best cognition when they lacked food!
To maintain a consistent energy level, 5+ small snacks is better than 3 large meals.
Anecdotally, that's not been my experience, except in the time periods when I've been transitioning between feeding on that short schedule and feeding once a day.
I recognize that this experience is wholly anecdotal, but the only time that I've felt "paralyzed with starvation" is not when I am actually in starvation mode (that is, 48 or so hours of not eating, which I do every couple of months), but rather when I've been eating often and switch to eating less often.
That is to say, when I used to have candy and donuts and juice as part of my diet, then I would occasionally crash and lots of smaller meals prevented that crash. In that case, your advice is good.
In other cases, it doesn't fit my experience.
When I am in a period where I am eating a single large meal every 24 hours or so (mostly rice and beans and some other plants with a lot of fat), then I don't feel those crashes at all. I don't eat outside of that meal except for a short period of time before or after. I occasionally feel hungry, but I generally on-point mentally, and certainly not "paralyzed with starvation".
If I get out of that pattern (usually because I am traveling or around folks who prevent that kind of eating from happening) I will often need to go through a longer fast to get back into that daily eating schedule.
So my hypothesis (untested, of course) is that most folks have never gone more than 24 hours without eating, so if they are crashing after 3-5h then that's specific to their diet and not a universal human constant.
> It's interesting to look at the history of measurements of the charge of an electron, after Millikan. If you plot them as a function of time, you find that one is a little bit bigger than Millikan's, and the next one's a little bit bigger than that, and the next one's a little bit bigger than that, until finally they settle down to a number which is higher.
> Why didn't they discover the new number was higher right away? It's a thing that scientists are ashamed of—this history—because it's apparent that people did things like this: When they got a number that was too high above Millikan's, they thought something must be wrong—and they would look for and find a reason why something might be wrong. When they got a number close to Millikan's value they didn't look so hard. And so they eliminated the numbers that were too far off, and did other things like that...
https://hsm.stackexchange.com/questions/264/timeline-of-meas...
A more theoretical argument is as follows. Suppose we systematically remove from analysis any data point which are more than 1.5 times the IQR below the first quartile or about the third quartile. (https://en.wikipedia.org/wiki/Outlier#Tukey's_fences) Points censured by such a rule are often called "outliers," but because we do not yet know the true underlying probability distribution this terminology is misleading. Then our estimates for skew, kurtosis, and all higher moments will be biased downward, and furthermore the censured sample is more likely to "pass" normality tests such as Shapiro-Wilks which can lead to us applying inappropriate models. However, what is the interpretation of a model when applied to a new observation? If the data point is well within the Tukey fences, then we can use the model to make a valid prediction. However, if it is outside the fences, we can say nothing. What then is the population mean of the predicted value? It is the weighted mean of the values for those data points for which our model is valid and those which lie outside of it. But those that lie outside of it can be enormously influential.
An example can make this concrete. Suppose an actuary estimates the amount of an insurance payouts for house insurance. 99% of the time, these effects are small, a few hundred or a few thousand dollars. But 1% of the time they are much larger, perhaps $500,000 or more. If the actuary applies Tukey's fences to his data and fits a model, he may infer that 1% of claims payout each year, and those payouts average $1,000 dollars. So he sets the price of the insurance at $20/year and makes a 100% profit margin. Until the first house burns down, and his company goes bankrupt.
The point of this is merely to illustrate the dangers of ignoring any effect which simply seems "too large." The beneficial effects of penicillin are "impossibly large" - would we reject the evidence on those grounds? The change in resistance of a semiconductor in the presence of an electric field seem "impossibly large" until the solid state physics is carefully analyzed - would we reject this clear and highly reproducible evidence of a qualitative change in behavior because it is too evident?
A much better approach is to scrutinize the methodology for, say, omitted variables. This is what the more detailed criticisms he cites in the third paragraph do. Or to attempt to reproduce the study, which is perhaps the strongest approach.
Looking at the study on judge's sentences before and after meals, you wouldn't expect to see a giant effect, but seeing some kind of an effect is a priori plausible, since we know many people are irritable when hungry.
When you see an unusually large effect size, it's often the case that there is a smaller effect, but you happened to get some extreme data. You definitely shouldn't jump to the conclusion that the effect size is large if you think it is a priori unlikely to be so large (unless you have very high statistical power). As you say, the results shouldn't be discarded, though.
edit: If an effect is incredibly improbable (like a relation between country music and suicides), evidence can still sway the belief against it. However, extraordinary claims require extraordinary evidence.
I think you miss the point -- the efficacy of penicillin is obvious to anyone, whether they've taken statistics or not.
The authors point, as I understand it, is that if the effects are so very large they should be obvious to a=even a casual observer. If the effect is so large it's impossible for them to be surprising.
The exhortation to look at the mechanism is also critical, and ignoring the mechanism is exactly how you end up with the worst inferences… if something is “impossible” or even just “surprising” according to your model you should probably reëvaluate your model.
The effects of penicillin are overwhelmingly obvious: for some set of diseases, penicillin produces rapid and complete recovery even in otherwise-terminal patients. I don't know the d for "penicillin treats staph", but in pharmacology d=5 isn't considered unreasonable for precisely these reasons.
The average d in psychology, though, is around 0.4 - and that was before many of the 'strongest' results fell apart in recent years. This study's effect size wasn't inherently insane, but it was claiming confidence in the realm of "antibiotics kill bacteria" for a subtle and previously unobserved force on judge's decisionmaking. It absolutely should have raised questions like "if this effect holds, why aren't trial lawyers acutely aware of it and working to manipulate it?"
> The point being made isn't that large effects are impossible, or that high-d results should be dismissed outright.
But what the author actually says in the concluding paragraph is that certain results are to be dismissed outright and moreover the burden of proposing a plausible mechanism is wholly on the experimenter:
> Implausibility is not a reason to completely dismiss empirical findings, but impossibility is. It is up to authors to interpret the effect size in their study, and to show the mechanism through which an effect that is impossibly large, becomes plausible. Without such an explanation, the finding should simply be dismissed.
He does not say, "other experimenters should attempt to reproduce the result" or "theoreticians should explore the phenomena mathematically" but only "the finding should simply be dismissed." Dismissed! Note the stark contrast: while you emphasize that such results should not be "dismissed outright," that is exactly what the author calls for! No follow up studies, no novel theoretical models proposed, just blanket dismissal; furthermore, in the authors own words, the test used to divide those results which are dismissed is "implausible" vs. "impossible."
But who can so finely demarcate the implausible from the impossible? Certainly not the contemporary scientists, who in every age we find railing against newly discovered theories and inconvenient facts. And why would the experimenter who first notices a phenomenon also be required to at once to provide a satisfying theoretical explanation? The role of the experimenter is careful lab work. Was the mechanism behind penicillin understood immediately? Was the quantum physics of the depletion zone fully understood when the strange electrical properties of doped silicon were first observed? When half of the neutrinos from the sun go missing, the experimenter must simply attend to his instruments to ensure he has not made an error, and then throw up his hands and let the theoreticians get to the bottom of the thing. This is not the only way science can proceed, but it is an important way: surprising empirical observations, outside the realm of current theory or even directly in contradiction to it are announced decades, or centuries, or in a few cases (such as ancient supernovas) millennia before they are fully understood.
https://en.wikipedia.org/wiki/Solar_neutrino_problem
https://en.wikipedia.org/wiki/History_of_supernova_observati...
If the author had been content with making a specific criticism of this one study, I would not have objected. The study is almost certainly flawed, and various good reasons why it is flawed have been discussed. But the author seeks to elevates this principle to that of a general maxim (as the concluding paragraph makes clear), recommending it to one and all as as a valid and useful form of inference. But it is not (in the general case) a valid "form" of inference, as we see by swapping out the details of this one flawed psychological studies for several others, such as penicillin, the field effect, or neutrinos. In those cases, arguments of the exact same form would have resulted in the dismissal of legitimate data. It is often not a good idea to reject experimental results just because they seem "impossible." Concretely, we may analyze the authors proposed method as a three step process:
1. Before the experiment, decide what is possible and what is impossible. (We fix a hypothesis space. Anything outside of this space is given zero a priori probability.)
2. We conduct the experiment and observe the outcome.
3. If the outcome of the experiment is impossible, we "dismiss" the result. (We update no posterior probabilities.)
It is difficult to see how science could have progressed if such a rule were universally applied. For example, Rutherford was absolutely shocked when a certain percentage of alpha particles were reflected from a thin sheet of gold foil back at an angle near 180 degrees:
> "It was quite the most incredible event that has ever happened to me in my life. It was almost as incredible as if you fired a 15-inch shell at a piece of tissue paper and it came back and hit you."
Yet, because he did not follow the author's advice and dismiss these incredible observations, the nucleus was discovered.
Kuhn argues that when this situation occurs in the history of science, what actually happens is that the old theories are discarded and new ones are proposed and validated until even the "impossible" observations are now explained. (A paradigm shift.) I would suggest, and I think Kuhn would agree, that in step #3 we should not discard the observation, but rather discard the theory used in step #1 that led us to conclude that it was "impossible."
I can only add an anecdote to this:
I read a book where a planet would blink out of existence occasionally, but long went undiscovered because all the high tech scanners and computers kept throwing out the "impossible" readings as corrupt data.
I'm glad I read that book so I could grasp what you're saying here.
Rutherford was able to come up with such a mechanism in his work with Niels Bohr. Atomic theory was quite new when Rutherford made his discovery: in a sense this was low-hanging fruit. The article's point is that this effect size is so big that if it was real, any half-decent lawyer would aready know about it. Without an explanation for why everyone somehow missed this point, the claim is pretty much invalid.
I think the right analogy is if someone comes up with evidence for a speed of light that does not agree with the standard measurement. If the claim is to be taken seriously, not only do they have to be rigorous in their experiment - they also need to explain why all other measurements were incorrect. This stuff happens all the time in science. Experiments are sloppy, and results are difficult to replicate. The go-to reaction should be to throw out a result, unless it can be replicated time and time again.
Kuhn's description of a paradigm shift included room for 'problems' that can't be explained with the confines of normal science. You only start throwing things out when the evidence against you becomes unassailable.
Indeed it can. Why, if there were really effects as large as you claim -- where several hundred thousands of dollars of value are lost in a single incident -- then society would recognize that fact. We would have formed "Fire Departments" consisting of professionals skilled at minimizing the damage. We would have created "Homeowner's Insurance" as a way of distributing the harmful effects throughout society instead of concentrating it on a few. It is precisely because we DO observe these behaviors that we can nod and say "Yep, those 'outliers' are to be expected.
On the other hand, if you had never heard a story about an entire house being destroyed in a single incident and someone tried to tell you that the statistics showed it was possible, even likely... wouldn't you stop to ask questions? Wouldn't you feel like a well-done paper on the subject needed to address possible mechanisms by which the information might be less well-known?
On the one hand, shoudn't we? There are a lot of supposed facts that need to be checked. Also, the size of the supposed effect everybody knows still needs to be measured, if, for instance, comparisons between different populations can be made at all.
On the other hand, and personally, I find it to be common sense never to have difficult conversations on an empty stomach.
Previous comments: https://news.ycombinator.com/item?id=14701328
It's not that the data was poor, but that it and the analysis was incomplete and ill considered.
In psychology in particular, large effects will seep into everyday life well beyond the phenomena being studied, which gives a very useful check on astronomical effects like those originally reported on judges and time of day.
The original study was disproved, and TFA includes multiple such links near the top.
The critical part of science is the actual data and methodology. If you do X you will see Y is what constrains new theories. Conclusions on the other hand have very little weight.
PS: Sorry pet peeve.
If your conclusion is impossible, look for the mistake before publishing or trying to replicate it.
a) The act of ruling over criminal proceedings places one in a very unique position of power over complete strangers' lives which only judges routinely experience. What subtle effects on empathy, vindictiveness, leniency do minor fluctuations in mood have that are exacerbated by the unique conditions of this profession?
b) Judges are members of an extremely privileged professional class and universally come from extremely privileged backgrounds. The mechanism of the hedonic treadmill might explain how working class folks cope with regular day-to-day stresses with almost no impact to their mood where being peckish at midday is likely the harshest reality of life that a financially secure judge making $100 an hour has to deal with and may disproportionately sour their temper to a comparable degree that someone making minimum wage worries about paying their rent.
Etc etc. There is no compelling proof of this argument.