The Irreproducibility Crisis of Modern Science
nas.org
nas.org
1. No result is 100% reproducible because you can never completely reproduce the conditions of any experiment. The best you can hope to do is to reproduce the conditions that matter, but enumerating those has to be part of the theory you are testing, and so you can never be 100% sure that you have a complete list.
2. Even a completely non-reproducible result can be scientifically significant. For example, celestial events are almost never reproducible. Our understanding of celestial mechanics nonetheless rests on solid science.
3. The end-product of science is not truth, it is explanations of observations. Those observations can (indeed must) include non-reproducible ones. Sometimes the explanation of non-reproducible results is "experimental error" or "delusion" or "we just don't know." But non-reproducible events are nonetheless within the purview of science.
On the other hand...
4. The statistical tests currently in widespread use as a criterion for publication in peer-reviewed journals guarantee that at least one result in 20 will be due to chance and not because the hypothesis being tested is actually true. That, combined with the suppression of negative results, guarantees that the results published in scientific journals that adhere to those standards will be unreliable. But that doesn't really have anything to do with reproducibility per se, it has to do with the fact that the journals use a weak criterion for defining positive results. That, combined with our human predilection to value positive results over negative ones and the understandable desire of scientists to advance their careers, all but guarantees that journals will contain many defensible but false results. But this is not because of a lack of reproducibility per se, it's because of poor policy choices.
You mean that papers are published with P<0.05?
When did this become a "standard"? Most papers I have seen in respectable journals and other sources would not strive for such a high Pvalue, it would be much lower like 0.001.
I'm guessing you meant to say that no papers are published with P<0.05. And of course they are, but that's irrelevant. What matters it that many journals, especially in "softer" sciences (i.e. not physics) use p<0.05 as their threshold, which guarantees that at least one paper in 20 will be "wrong". Then add self-suppression of negative results and the actual percentage of "wrong" papers becomes higher, potentially much higher if real positive results are rare.
> When did this become a "standard"?
I have no idea. You'll have a ask a historian.
It is not, in any way a measure of reality. It is a guess at how accurately you've modeled the system, how accurate your priors are, and so forth.
> It does mean that ... 1 in 20 would be explainable by the null hypothesis.
That seems like splitting a pretty fine hair to me, particularly since I haven't actually committed to any of the many possible interpretations of the word "wrong".
While not everyone in a particular field has been a party to this myopic view of statistical significance, enough have been that the American Statistical Association released a statement on the topic not long ago. That may not seem like a big deal but, given the organization's historical reticence to officially weigh in on such issues, it really is.
ASA statement on p-Values: https://www.amstat.org/asa/files/pdfs/P-ValueStatement.pdf
Most people would say in the 20s and point at Fischer's publications, though you can have arguments for later and earlier. Some of that is mentioned in the Wiki page, and you can see 0.05 all over the article [0], and the other related wiki pages[1]. You can find some key quotes there:
"The significance level for a study is chosen before data collection, and typically set to 5% or much lower, depending on the field of study"
"In 1925, Ronald Fisher advanced the idea of statistical hypothesis testing, which he called "tests of significance", in his publication Statistical Methods for Research Workers. Fisher suggested a probability of one in twenty (0.05) as a convenient cutoff level to reject the null hypothesis.In a 1933 paper, Jerzy Neyman and Egon Pearson called this cutoff the significance level, which they named α."
If you mostly see lower p-values, that most likely means that you usually read non-social science papers. I guarantee that most social science (and related) fields use 0.05 regardless of the respectability of the journal.
As an example of fields with lower p-values - 0.001 is common in medicine, and physics commonly has even lower values.
0. https://en.wikipedia.org/wiki/P-value#History 1. https://en.wikipedia.org/wiki/Statistical_significance#Histo...
I think this depends on the field. In some areas of neuroscience for example, brain data can have extremely high variability and is very expensive to collect a lot of, so p<0.05 is not uncommon.
Some of the best advice I got was to spend a lot more time than one might want to really come up with an experimental design that avoided a statistical outcome and instead probed at an either/or mechanism. It's not always possible, but when it is, it can be a powerful reply to fighting statistical battles (see the Michelson-Morely experiment for a classic example).
Further careers, families, reputations, humans are also part of that scientific process, which together with the above require and admit some deviation from perfect efficiency. Which we as a society should demand be okay.
I think news organizations make this worse. Whenever a new paper comes out they'll champion it as fact if doing so leads to clicks.
Peer reviewers can only hope to validate that the science was done well, not that it is inarguably correct. It requires a body of work around a topic before it's fair to say at all that something is "for sure" proven.
University PR is far more guilty. Buzzfeed isn't scanning the current issue of "Physical Review Letters" for the latest scoop.
That is the idea behind Rutherford's famous motto, If your experiment needs statistics, you ought to have done a better experiment.
So long as the researchers aren't outright inept or fraudulent, what's there to be ashamed about? Those would be some of the more interesting papers to read, in my opinion.
No, that won't help. You'd end up with the Journal of Creation Science and things like that.
What we need is pre-registration: https://cos.io/prereg/
We also need to change the culture so that positive results are not disproportionately rewarded.
So we choose to spend our time reviewing and evaluating those experiments that do seem to have at least some chance of saying something interesting. Even then, though, depending on the discipline, if the researcher did everything perfectly, they're still saying that there's a 5% or some% chance that they just "got lucky" and there is no underlying effect at all.
http://www.jnr-eeb.org/index.php/jnr/index
I would actually prefer that each and every journal devote at least half of their space to negative results. I have seen some sublime experimental setups that produced negative results. I can't begin to imagine how instructive it would be to see more such experiments.
I am not sure this is correct. The rules that govern celestial bodies is same as the ones that govern objects on earth. So why are they not reproducible? It does not require to measure forces between celestial bodies to measure the value of G. Measuring the forces between two masses on earth is enough.
So, for example, we can predict with ridiculous accuracy where the next total solar eclipse is going to happen. But once it happens, we can't roll back the clock and make it happen again.
I am finding a hard time to grasp what you are telling here. If you deduce a set of laws A from an event X, you don't need X to repeat to check the validity of A. You can check it against another event Y, where you predict its characteristics using A. If the laws are valid, the prediction ll match the actual observation..
That is exactly what I wanted to ask you originally when you cited the celestial mechanics example. It is not relevant in cases we are deriving a theory underlying the behavior, rather than axiomizing the very specific behavior itself...
Deriving an underlying theory would be deducing the force of gravity from it.
Axiomizing the very specific behavior would be making a rule that says. "If the object A (and object A only), while moving in a straight line, comes at so and so coordinates with respective to B, will end up in an orbit around it"
So these are two completely orthogonal issues. The question of what you do with the result of an experiment is completely independent of whether or not you can reproduce that result. The latter is what is under discussion here.
So you can, for example, observe that there was a solar eclipse at a particular time and a particular place. But you cannot repeat that observation. (By way of contrast, you can drop and object from a particular height and observe that it takes a particular time to fall. You can repeat that observation.)
What you do with those observations has (almost) nothing to do with whether or not you can repeat them.
I have a pedantic correction, but a relevant one. If a journal makes a rule that results must have p < .05, and then scientists go off and do a bunch of well-managed science they will get results with a range of p-values. .05 will be a ceiling, not a floor, on the possible p-values. More papers have a p-values right around .05 than they would if results were being produced fairly, so that's actually evidence of bad practices.
Now, it turns out that the floor is far lower than researchers believe and the field would be better off by slower incremental progress built atop "obvious" results, but still, I assert that publishability is an adequate explanation for high p values and obscures the evidence for bad practices.
Not that I have any doubt that bad practices abound; I have an undergraduate degree in psychology and am well aware of how things are done. It's not even malicious; people don't understand statistics, don't understand significance, don't understand models, and are mostly just doing whatever they can to make themselves feel good about their own intuitions and biases.
That's wildly overly pessimistic. That would only be the case if scientists just went around, looking at the world, and came up with null-hypotheses willy nilly. That's generally not the case. There is often a _reason_ for conducting the test and a plausible cause of action. There are two possible reasons for a significant result:
* The null hypothesis is correct but they got "unlucky" data (5% chance or p% chance)
* There is a real effect (and the null hypothesis is actually wrong)
This becomes more problematic in reproducibility tests, though, since that biases my prior towards a correct null hypothesis and now you must be very careful about pre-registration and the numbers of folks worldwide that are trying to reproduce a given experiment.
> That's wildly overly pessimistic
Why? You yourself said:
> The null hypothesis is correct but they got "unlucky" data (5% chance or p% chance)
How is that different from what I said? 5% is just another way of writing 1 in 20.
They are showing the _likelihood_ of generating the data were there no "effect." You make a "null hypothesis" — that is, you assume there is no effect — and you construct a distribution of the results you might see in such a case. If your results are relatively unlikely, then you say the null hypothesis is rejected at a p-value level, leading support to the fact that there is such an effect.
But there's another bullet in my comment. You could _also_ see a low p-value if the null hypothesis is wrong! That would _also_ lead to a low p-value, but for a completely different reasons — reasons that are typically elucidated in the entire rest of the paper. Thus, you should be most suspicious of papers that do not have a reasonable cause of action, as that should increase your prior that the null hypothesis is actually correct.
If I were to edit that post, I'd add a third bullet — that they cheated or did bad science. My objection is to you stating that the statistical tests alone guarantee at least a 1 in 20 failure rate with a p-value of .05. There are indeed many other reasons to be suspect, but the statistical tests don't guarantee this.
The on top of that baseline rate of false results you also have self-selection and occasional outright cheating that drives the failure rate higher than 1 in 20. 1 in 20 is what you get when you use a 0.05 threshold and everything else in the system works flawlessly.
If what you mean to say is that "1 out of every 20 statistical tests of a false finding will demonstrate a 'significant' p-value and may subsequently get published," then we're in agreement.
> while a low P value indicates that your data are unlikely
> assuming a true null, it can’t evaluate which of two
> competing cases is more likely:
> * The null is true but your sample was unusual.
> * The null is false.
http://blog.minitab.com/blog/adventures-in-statistics-2/how-...
At the same time, though, converting p-values to a bayesian posterior or conditional probability that says something about the likelihood of the _hypothesis_ (like you're trying to do) is hard and generally way more pessimistic than 1 in 20 — it's more like 1 in 3.
http://www.dcscience.net/Sellke-Bayarri-Berger-calibration-o...
Of course it can. In fact, it can tell you exactly how much more likely one case is versus the other. P=0.05 means that there is a 5% chance that the null is true and hence a 95% chance the the null is false.
> says something about the likelihood of the _hypothesis_ (like you're trying to do)
No. I am simply saying that there is a lower bound on the rate of false positives caused by the choice of the threshold p value. If you chose P<=0.05 then you will have an error rate of at least 5%. The actual error rate is almost certainly higher (that's what "lower bound" means) for a host of other reasons.
Were I to run another 100 independently designed experiments that all tested big real effects, then a p-value of 0.05 could result in all 100 accurately reflecting reality.
Were I to run 100 independently designed experiments that all tested false effects, then at a p-value level of 0.05, I should see about 5 erroneously significant results.
Now I submit all these significant results to a journal and they all get published. Of the 135 results, only 5 are coming to the wrong conclusion.
Look at that, an error rate lower than your lower bound. We simply don't know the sizes of these populations. It could be better or worse than my example, and it could be better or worse than 5%. There simply is no lower bound.
Of course you're right that there is a problem here but there isn't a lower bound guaranteed by the stats tests. Quite a bit of the problem stems from misinterpretations of p-values. For someone who likes to condescendingly explain concepts, you might want to make sure you have the harder concept down before assuming others don't understand "ratios" and "lower bounds." At this point you're arguing against the sources I cited. You should check them out. :)
That's actually not true. The problem is that you cannot define what is a "real effect" without begging the question. Let me illustrate with an example: Suppose I do what appears to be a legitimate experiment to test a well-accepted law of nature. Unbeknownst to me, my instrument is faulty (or has been tampered with) and is in fact returning essentially random results. If I choose p<=0.05 then I will get a positive result 1 time in 20, i.e. I will "erroneously find no result" 19 times out of 20. Except that I'm not really erroneously finding no result. I am correctly finding the results of a different experiment than the one I think I'm running.
And, of course, an instrument that returns random numbers is an extreme example. See:
https://arstechnica.com/science/2018/04/new-measurements-set...
for a current real-world example.
Now, you do have a valid point in that I made a tacit assumption when I claimed that statistics put a lower bound on the error rate. That assumption is that most experiments that are done test hypotheses that ultimately turn out to be false. I believe this assumption is actually true. This is one of the reasons making scientific discoveries is hard. But I didn't actually state this assumption, so I'll cop to that.
> At this point you're arguing against the sources I cited.
That's not true either. The first source says that the error rate is much higher than the p-value, which agrees with me. (I'm actually a little skeptical that the error rate is as high as your source says -- "At least 23% (and typically close to 50%" for p=0.05, but that's certainly not refuting my position!) I don't have time to go through your second citation right now.
That's precisely my point. We don't know the sizes of these populations, so you really can't say anything quantitative about how the p-value affects the proportion of bad results in any given journal. My example was intentionally contrived.
You just have to be really careful when you talk about p-values because _so many people_ have this misunderstanding and it's actively harmful to getting the correct interpretation. That's why we keep going back and forth here — I'm not disagreeing that the situation is bad, I just want folks to recognize what the stats say and what they don't.
I think we can make some pretty reasonable guesses.
In any case, I certainly agree with you about the big picture: it's confusing, it's important to get it right, and a lot of people don't understand it. I may even be one of those people. But at the moment I believe that our disagreement is really over something else, namely, whether it's reasonable to have a prior on the null hypothesis.
No. It simply does not mean that. It would be very convenient if it did, but it does not and can not.
p=0.05 means that if the null hypothesis were true and you ran your experiment, then you would only have a 5% chance of getting the results that you did.
The distinction is not at all obvious, but you'll need to understand it before you can make sense of statistics.
The p-value is not the number you want; what you want is the probability that the null is false, but you can't have that unless you know the actual probability distribution. Which generally isn't something you could possibly know or figure out.
(You can make educated guesses about that distribution, by making a guess and then updating it based on observed results. That's Bayesian statistics, which is great because it does give you a full probability distribution and all you have to feed into it is... uh... a probability distribution.)
If there's no pre-registration, my prior of a paper actually being true would be much lower than statistics would say simply because the incentives are aligned so that there's a motivation to exaggerate significance.
The reason this is being called a 'crisis' is because it's looking like the vast majority of science, particularly in the social and human physiological sciences, is junk.
There are definitely other reasons that lead us to such horrifying results — especially in fields where experimenters _do_ conduct experiments seemingly at random and without a reasonable pathway of action (I'm looking at you, social psychology and friends).
On top of this, the more interesting results are the most surprising. And therefore the least likely to be true a priori. So the chance of an interesting study being true is even lower.
Yes and no. I think you have to have predictions in there somewhere. That's why your celestial mechanics is science: whatever the movement is, you were able to say what it was going to be beforehand, even if it only happened once.
As for explanation, that is not necessarily so simple, either. Sure, a lot of things seem to make more sense with a theory, but not every theory makes sense. A lot of stuff in fundamental physics is predicted by the equations, but in what sense are the equations explanations? "Shut up and calculate" comes to mind.
And also, shut up and come up with something falsifiable (in our lifetime).
That sounds a whole lot like truth to me, or at least, like any sane construal of truth.
And in fact, General Relativity is the best explanation science has given us at the moment, which seems to fit what I said.
Until this moment, I never suspected how insightful the following is:
It's worth noting that this is part of the goal of reproducing results. If two identical experiments produce the same result, you're only providing proof for the very narrow theory the experiment tests. Slightly different experiments allow you to demonstrate that the theory is applicable. Imagine if evolution only occurred in the Galapagos, or if gravity only worked in an apple orchard in Cambridge. The only reason we know that evolution and gravity aren't local oddities is that they have both been tested in a wide variety of locations, with a wide variety of experiments.
The term 'science' is used in this way. Perhaps it should not be. Perhaps semantics drive the issue.
In the field in which I am familiar, we have 'expert witnesses' that will use reproducible scientific principles to analyze, for instance, blood scatter to determine the angle in which a shotgun was held when it discharged and blew off a particular person's head. The principles -- based on physics, chemistry, etc -- are sound and scientific.
The problem is, we will invariably have TWO very scientific expert witnesses who BOTH use reproducible and accepted methods. Yet, using their scientific tools, they will come to two utterly different conclusions.
It's left to a jury to decide which interpretation is correct.
While the tools used by the experts are scientific, their opinions are not. While their scientific tools are reproducible (for instance, the speed at which coagulating blood and brain tissue will slide down a stucco wall given all the variables such as temperature, humidity, friction coefficients, etc.) their ultimate reconstruction of what happened is not reproducible, and it is left to lay people to draw their conclusions.
There should be more of a distinction between Science and History. That is, science -- strictly defined -- is all about reproducibility. History -- by definition -- is not. Using scientific tools to reconstruct a past event does not result in a 'scientific' theory. Rather, it results in a historical theory arrived at with scientific tools.
Whether it is blood scatter from a murder (or suicide, depending on whom you ask), the formation of the Grand Canyon, or the development of finch beaks on Galapagos, the ultimate theories are inherently non-scientific as they can not be tested. The methods and tools used to derive the theories can be... but not the conclusion itself.
In short, perhaps the term 'science' is being used for things outside its domain. (Or, alternatively, if we wish to include such things inside the domain, we should broaden the strict definition of science. Like I said, there are some issues of semantics that may be driving some of the issues in the article... other issues, of course, pertain to sloppiness, errors, and the like.)
If the original study was under-powered, the estimated effect size in that study will be inflated and any replication attempt that uses this inflated effect size estimate will be severely underpowered.
Plus, two independently conducted studies that are both powered at 80% to detect a true effect will both be positive results only 64% of the time (assuming absolutely nothing fishy going on, e.g. p-hacking).
This is why you include the error in your results. We don't care if your two experiments result in 100% the same results. We care about if your results line up within the error of your experiment.
> 2. Even a completely non-reproducible result can be scientifically significant. For example, celestial events are almost never reproducible. Our understanding of celestial mechanics nonetheless rests on solid science.
This is why we share data. That is in essence our observation. A celestial event might be caught by only one instrument, but several people can develop several different models. We wait and watch for similar events though, to check if models are consistent (within error).
> 3. The end-product of science is not truth, it is explanations of observations.
I just wanted to repeat this because it can never be stated enough.
> 4. The statistical tests currently in widespread use as a criterion for publication in peer-reviewed journals guarantee that at least one result in 20 will be due to chance and not because the hypothesis being tested is actually true.
While I don't like p values, because of hacking, (especially 0.05) that's not how stats work. Flipping 10 heads doesn't guarantee 5 tails, not even 1. I wouldn't use as strong as a word as guarantee.
But this is also an argument FOR reproducing. If multiple experiments are consistent with one another (within error) than that strengthens the argument.
TLDR: More brains that look at a problem helps solve the problem.
I will add my own statements about the reproducability problem in science. One is because there is less funding for it. Reproducing an experiment isn't sexy. Another problem just stems from that data isn't always open. It is hard to review work if you don't know everything about the experiment. Data can even have simple mistakes that just weren't caught. But it is also embarrassing to share data.
I never said it did. But in the long run, flipping a fair coin will generate pretty close to 50% heads and 50% tails. That's what it means to be a fair coin. Likewise, in the long run, using a p threshold of 0.05 (which many journals do) will generate 5% false positive results (that's that the 0.05 means), i.e. on average 1 false positive result for every 20 experiments you do.
Flipping an unfair coin twice in a row and discarding the HH and TT doubles will generate close to 50% HT pairs and 50% TH pairs. This only presumes that subsequent flips are independent events.
It's still not going to be exactly 50%, because random events are random. You can get as small an error margin as you need by increasing the number of trials. And in some cases, this property of statistical analysis makes it easier to increase N than reduce the randomness and uncertainty in the experiment.
Example: I want to test if evolution is true. I hypothesize that if evolution is not true, then God will give me some sort of sign, e.g. I pray to God to make a coin that I flip come up heads if evolution is false. I flip the coin and it comes up tails. I conclude that evolution is true. That conclusion would be wrong even if evolution is true.
It is if P(A) is 1, which is what you stipulated when you asked "How can u get a wrong positive result if you only test correct H1s?"
> "... using a p threshold of 0.05 ... will generate 5% false positive results" is just plain wrong
No, it's not wrong, it just makes a tacit assumption that most hypotheses that get experimentally tested are incorrect. Which is a reasonable assumption. If it were not true, making scientific progress would be a lot easier.
[0] https://en.wikipedia.org/wiki/National_Association_of_Schola...
[1] https://en.wikipedia.org/wiki/National_Academy_of_Sciences
Thanks for the link to the PNAS -- looks like a whole special issue, not just an article, in fact. Great. I'm gonna ignore the fake-NAS one, and read the real-PNAS one.
I wonder what National Association of Scholars' motivation is here exactly.
However, this paper isn't a study. It's just an essay. And while, yes, the paper in question does highlight numerous issues in reasonable and rational ways, it also uses this as the breakout quote on page 36:
> Scientists readily accept results that confirm liberal political arguments, and frequently reject contrary results out of hand.
Now, there's a couple problems with this statement. The most obvious is that it implies blame on liberal politics. On the same page they detail that the percentage of self-identified liberal psychologists at a conference (~800 of 1,000) wildly outnumbers the number of self-identified conservative psychologists at the same conference (3 of 1,000). Well, this conference is the Society for Personality and Social Psychology, which is based in southern California. Would that bias attendance? However, even if we assume that distribution is correct, the quoted statement doesn't say anything about the conservative scientists. Would it also be correct that conservative scientists readily accept results that confirm their political arguments? We don't know. Maybe it would be more correct to say this:
> Scientists readily accept results that confirm pre-existing personal political arguments, and frequently reject contrary results out of hand.
Instead, this paper chooses to imply that scientists who are more liberal are somehow inherently more biased. If we assert that good scientists are those who don't allow political arguments to influence their acceptance of their results, then the natural conclusion to draw from that is that being liberal is incompatible with being a good scientist.
Don't get me wrong, this paper makes a lot of legitimately good points with good examples, but the whole section on groupthink comes across a little bit biased itself. However, it's existence means I can't shake the feeling that maybe I'm missing something.
So having show that there's a reproducibility problem, the paper concludes. Then very end of the paper, the afterword lists these points introduced seemingly out of the blue:
> Symptoms of Pathological Science:
> 1. The maximum effect that is observed is produced by a causative agent of barely detectable intensity, and the magnitude of the effect is substantially independent of the intensity of the cause.
> 2. The effect is of a magnitude that remains close to the limit of detectability; or, many measurements are necessary because of the very low statistical significance of the results.
> 3. Claims of great accuracy.
> 4. Fantastic theories contrary to experience.
> 5. Criticisms are met by ad hoc excuses thought up on the spur of the moment.
> 6. Ratio of supporters to critics rises up to somewhere near 50% and then falls gradually to oblivion.
These are from Irving Langmuir (https://en.wikipedia.org/wiki/Irving_Langmuir) and his criticisms on research conducted with unconscious biases or subjective elements. He meant it as a criticism of research like cold fusion.
Now, there's not any real one-for-one discussion of these items or how they relate to what came before. They're just presented basically as-is as the final portion of the essay with few notes that they correlate to what came before. You'll notice, however, they they all call into question the credibility of the scientist as well as the credibility of the scientific community, and not the quality of their research or data.
However, let's say you're politically conservative, and you're reading this essay. Maybe you don't believe in climate change. You get to the end of this paper, and it tells you that in spite of the claims of accuracy of the data, in spite of the consensus of the community, in spite of all the evidence, if it's contrary to [your] experience or if the defenses to criticism feel invented on the spot or if you keep hearing claims that support is waning, and apparently especially if it's liberal, then you've found a legitimate case of bad science.
I don't think that's necessarily the point they're trying to make here, but I really don't like the way this afterword is presented. It's introducing new concepts and arguments and then handwaves away having to justify them or explain their relevance.
To eliminate any questions about social context or bias or agenda? Totally not possible, social context is an inherent part of science, as in all human endeavors.
I think the purpose of reproducibility is to well, make science _possible_, it is the fundamental bedrock of science -- to determine what and how we can make reliable predictions about the world. It doesn't mean bias and effects of social context no longer exists though. Science exists anyway, in a social context -- as evidenced by the past couple hundred years!
In general, I don't think science exists to eliminate _any_ questions, science exists to ask questions. Eliminating questions may be a goal of ideology or religion, but not science. (and in the real world, they certainly influence each other. science is a goal to strive for, not a fait accompli. Scientific understandings of the world are always changing; science is an approach to understanding the world, not a set of settled conclusions).
Maybe I'm wrong. There are probably ways to investigate it. This essay isn't one of them.
There are certainly non-Science ways of forming an opinion and deciding policy, in many cases totally legitimate. But once science is claimed, then of course it has to be subject to science rigors.
From the journalism side I think you end up with the problem the wide spread field of journalism has, you have to be read by your audience. That means something with a wider audience is probably going to be more on the ELI5 side which means both the reader and the journalist have to take any reported findings more lightly.
The sliding scale, I believe, comes in with the rigor something is reported. That is both in the practice and the journalistic side. More details can give a more granular understanding showing limitations but some times broad strokes are necessary to get some information out to a wider audience or more quickly.
Take climate change, no doubt a controversial issue. To say it is "controversial" in the political sense would mean Left and Right (in the USA alone) have vociferously different stances on the issue. However, if viewed as a scientific controversy, we are now talking about detailed methodological concerns, like methods of data collection, analysis, kinds of statistical bias, and subtle changes in arbitrary parameters. Any scientist can tweak this or that in their model to make it conform more easily to their preconceived notions about climate change. Unfortunately, we also have some oil industry shills out there who got trotted out as an equally weighted side B to the side A of the dozens of scientists who would generally disagree. Then, you also have anti-science proponents who use the legitimate self-criticism of scientists to attack science as a whole.
It's a sad state of affairs. Science should be reported in the press, but it should also be reported much better than it is. In the USA in particular, STEM education is lagging behind: the average person can't delineate the good science coverage from the bad, and we have ridiculous notions and conspiracies that fail to become filtered out (anti vax, climate change deniers, flat earthers, moon landing was faked, etc.)
For me, reproducibility is one problem in a broader ecosystem of scientific problems, including science education generally, as well as misuse of statistics, and a saddening drive for incremental results at the expense of more broad-based thinking which might lead to fundamental breakthroughs. Our education systems must be reformed to deal with these problems, that's the only way out that I see.
1. assumes the validity of the scientific method
2. relies on the scientific method as its critical lens
Whereas those who critique science as a whole:
1. assume that the scientific method does not work and does not arrive at "truth"
2. then use scientists being self critical to prove #1.
Such a "proof" does not work as there is its uses the assumption "the scientific method arrives at truth" to derive the contradiction "the scientific method does not arrive at truth". See for instance comment: https://news.ycombinator.com/item?id=16859200
In reality, work on reproducibility is about improving the practice of science overall. It does not in itself show that science is inherently untrustworthy. What it does show is that scientific discovery is difficult and it takes a lot of effort and new findings should be treated critically. What does critically mean in this context? It means with in the boundaries of science analyzing the theoretical basis, hypothesis, method, and experimental results for potential flaws. It does not mean to be skeptical as a default because science "doesn't work."
What you should have said is that some critics start with the premise "the scientific method does not arrive at truth", and then use other people's arguments that depend on the premise "the scientific method arrives at truth" to support their claim, which is indeed logically invalid.
I think what happening here is a bit more serious. They are showing a widespread crisis. It is not just some minor feedback to improve the process.
>It does not in itself show that science is inherently untrustworthy.
I think when statistics is involved, the results are inherently untrustworthy. This is not really surprising because there is a whole bunch of ways these studies that involve statistics could go wrong. And we are still finding new ways on how this could go wrong.
Then there are things like publication bias, that takes this to a whole new level. Things like that means that a biased body of journals can project any consensus that it favors just by selecting studies that fit its narrative. The inherent issues with statistics means that you can find studies that shows any possible outcome.
Ummm...are you kidding? Statistically vetted results are inherently UNCERTAIN, but how could they possibly be inherently untrustworthy?
Even if a mechanistic effect is observed, its relationship to a particular cause or influence is only established statistically. In fact, the very observation is often performed u Dee the umbrella of statistical calibration of appropriate instruments.
As Pearson said, "Statistics is the grammar of science". The level of uncertainty in most (perhaps all) experiments involving particles in a vacuum is far lower than experiments involving human behavior. Statistics doesn't create the uncertainty, it communicates the uncertainty.
I can use propositional logic to arrive at the conclusion that "Unicorns are real" yet I don't encounter many people throwing logic itself under the bus because of that. Statistics has been a whipping boy for a long time by people who don't know much about statistics or just fail to think clearly about what is being said.I would recommend reading the article. The executive summary summarizes things. The issue is not statistics in and of themselves, but how they are used in the overwhelming number of studies - particularly those in the social sciences or related with human physiology.
This is the whole point. It is the difficulty in doing so, or the difficulty in deciding if this was done appropriately or not is the only thing that it untrustworthy..
>People lie while speaking English every day yet I don't encounter many people claiming that English is inherently untrustworthy.
No one expect a statement to be true simply because it was uttered in English.
Another very real issue here is that malicious use of statistics can be used to show nearly anything in ways that can be extremely difficult to detect, even when the maliciousness is hidden in plain sight. And then going a step beyond that there's plain old number fudging which is almost impossible to prove since variance works as sufficient plausible deniability. And finally there is of course plain old ineptitude. Like you mention even when trying to do things completely by the book, statistics are incredibly difficult to get right.
Something that comes to mind here is the recent MIT study stating that Uber drivers earned $3.37/hour. That study was completely broken. [1] It's debatable whether the cause was maliciousness or ineptitude, but the point is that these problems arise, with a disturbing regularity, even when the most reputable of names are attached to them.
[1] - https://qz.com/1222744/mits-uber-study-couldnt-possibly-have...
There can be malice, sure. But there can also be desire to believe. And, hey, here's a statistical analysis that shows what the investigator is already biased to believe anyway...
The executive summary lists 40 points on how to improve the reproducibility of science. A bit over half of them are addressed to the sources of capital such as private organizations, universities, and governments. I think many of those points are good. However, I don't think the other points, the ones that recommend doing science in different ways, have a good punch. Even if you fix the problems there are today, so long as science is a rat race of trying to get grant money to stay afloat while burning out grad student after grad student, I think other pathological practices will creep in as a completely rational response on the parts of scientists to a hostile ecosystem. There's just a very big gap between how science should be done and what capital owners want from science.
Maybe the biggest problem is what you mention, but in another direction. For whatever reason there seems to be extremely little interest in the private direct funding of science. In times past the aristocracy would often fund scientific research on all sorts of topics. Today the practice seems to have all but disappeared, certainly if we measure relative ratios of the practice.
Eliminating fossil fuel use is also simple. But there's another dimension to this problem, called "political viability", which I feel you're overlooking.
Too much income equality. You can't have Rockefeller University without Rockefeller.
It was "a team from Bayer Healthcare" who "tried to replicate the results of basic cancer studies," failed, "and kicked off a media storm questioning the legitimacy of cancer science—and science in general" [1]. The "capital owners" looking out for their own buck are performing more effectively, narrowly speaking, than academia.
[1] https://www.wired.com/2017/01/fighting-cancers-crisis-confid...
It's also quite obviously the case that profit seeking research has incentives to be bad science in other ways.
I agree. My point is the Marxist framework is a bad one for modern science.
We have endowed academics and ones working for private institutions and getting grants from public bodies run by scientists. Privately-financed biotech companies founded by academics taking moon shots and getting acquired by marketing and distribution powerhouses. Even delineating who is "capital" and who is "labour" quickly becomes pedantic more than useful.
I will concede that it's reasonable to think it's an incomplete and limited framework, but I have to take issue with calling it bad. For example, I think this starting point is more productive than the premises of this report.
> Privately-financed biotech companies founded by academics taking moon shots and getting acquired by marketing and distribution powerhouses. Even delineating who is "capital" and who is "labour" quickly becomes pedantic more than useful.
In this example it's plain to me that VCs are capital and entrepreneurs are labor. That's usually the game in such moonshots. Here we can analyze what sort of science is being done through this relationship through a Marxian lens too. VC's want to make big bets with potentially astronomical upside. That means only certain kinds of science will ever be done in this way. Yea, this kind of science is probably more replicable on average, but what about something like Theranos? This sort of science funding has its own problems with inefficient and incomplete allocation of resources.
I'm not a hardcore Marxist; I don't think you can reduce the world to one grand theory based on hot takes inspired by Hegel, but I think it's a more powerful tool than most care to admit.
- she is exploited, i.e., forced to act in a profit-maximizing way (in particular, short-term profit). Each other behavior will be penalized (no career, no tenure, ...).
- she is exploiting, i.e., forces her workforce to act in way that they maximize her profit.
Just admit that the marxist analysis doesn't hold much water in this case. That doesn't mean it is never useful, just that you might need to think a bit before applying it.
You're really set off by the word M__x, aren't you?
The very nature of capitalism is that people who produce monetizable results should be rewarded for that work, and that people who don't shouldn't. This is fundamentally incompatible with doing science in a reproducible way: paying for positive results and not negative ones gets you p-hacking as an emergent effect. Of course the capitalists don't want to pay for non-reproducible positive results, hence the firestorm. But they more strongly don't want to pay for negative results.
The capital owners in the science industries will probably end up doing something sort of like what the capital owners in the software engineering industries have done: begrudgingly accepted that a non-capitalist approach (open source development across multiple companies without proprietary advantage) produces better results for everyone, and figured out a way to kind of support it. Much like the capital owners in the software engineering industries, they still won't be very effective at it. Their nature will be to still try to extract proprietary advantage when possible, to cut costs per the tragedy of the comments, etc., and so this model will only work well when funded by companies rich enough that they can allow themselves not to show an immediate return on their sponsorship.
Issues with reproducibility have been discovered independently in a number of different scientific disciplines going back to the 1990s at least. This was then covered in a New Yorker article by Jonah Lehrer, published in 2010 [1]. It included the phrase "decline effect", coined by a researcher studying ESP in the 1930s.
A great deal of the research described in that article is academic in nature. The article also alludes to the contributory influence of funding as one of the causes of this phenomenon:
"The bias was first identified by the statistician Theodore Sterling, in 1959, after he noticed that ninety-seven per cent of all published psychological studies with statistically significant data found the effect they were looking for." ... "In recent years, publication bias has mostly been seen as a problem for clinical trials, since pharmaceutical companies are less interested in publishing results that aren’t favorable. But it’s becoming increasingly clear that publication bias also produces major distortions in fields without large corporate incentives, such as psychology and ecology."
[1]: https://www.newyorker.com/magazine/2010/12/13/the-truth-wear...
The book is set around 1660, as the center of the scientific universe shifted from Italy to England. Interestingly, the formation of the Royal Society occurred in the midst of the English Civil war. Not only economic forces, but military forces and political forces had a dramatic impact on shaping what became the formalization of of the scientific method, and peer review process.
Consider that the very existence of Oxford (and other universities) hung on the arbitrary conceptions of military generals and the political implications of their decisions.
It's a nice story you've got there, but the reason there is a replication crisis in the softer sciences is not due to the evil capitalists of the marxist imagination. In fact, much of it is due to cultural marxist insistence on outcomes conforming to political correctness, as well as uncritical acceptance of politically conforming work. See the ongoing fight against the idea of a high genetic basis for adult IQ.
All of this is obvious enough: capitalists aren't interested in non-reproducable results unless it is obviating them from blame for a given externality. But that sort of research is a small fraction of overall research (and should absolutely be done by impartial third parties.)
It’d be cool if your and the other poster’s ideological spat could try to not use science as the battleground, or at least eliminate more likely issues before going there. Social sciences are hardly even science from the perspective of a chemist or physicist, there’s no need to delve into petty ideology. Methodology is a much richer source of hurdles to overcome than old and sad debates.
I don't disagree. A big part of the problem is the social sciences claiming the mantel of science (often just statistics layered over tiny samples, as you point out), in order to imply some sort of impartiality and precision on par with the hard (i.e. real) sciences.
Unfortunately, and I say this sincerely, I believe the ideological nature of the social sciences since the 60s has been responsible for a huge amount of uncritical acceptance of bad (false) work as well as preventing a lot of unpopular good (true) work. I don't know how to deal with that fact other than stating so forthrightly, which, unfortunately from the perspective of science, carries necessary ideological freight with it.
A chemist or a physicist still have hard time dealing with particle assemblies of say 100-100000 particles. For several last centuries they learnt to deal with [aggregate properties of] assemblies on the scale of 10e20+ particles. Only during last century they came to pretty good understanding of 1 particle.
In social sciences this is Facebook and Cambridge Analytica (ie. they are real social science today, not some professor in an office). This is the scale where aggregate properties appear, and they start to be able to drill down from it.
https://en.wikipedia.org/wiki/Bogdanov_affair
> It's a nice story you've got there, but the reason there is a replication crisis in the softer sciences is not due to the evil capitalists of the marxist imagination.
You understand literally nothing about Marxism. For example, "capitalist" isn't a moral designation, it is a description of an economic agent's role in an economic relationship. Capital owners have access to resources and they give permission to laborers to use those resources. That's it.
> cultural marxist
You can't be serious.
> See the ongoing fight against the idea of a high genetic basis for adult IQ.
This plays out more in the public sphere than in scientific communities. It's also the case that a few popular intellectuals try to sell psychometric science as saying more than it actually does about how social policies should be designed. This, again, happens moreso outside of scientific circles.
Studying economic factors underlying social activity isnt by itself Marxist. This methodology is adopted in support of different political positions. In fact, many free-market economists also do this and get accused of 'economics imperialism' by social science faculty who want to study other factors.
Another point made in [1] is the blame isnt just on companies but people being influenced.
[1]https://medium.com/@drjasonfung/the-corruption-of-evidence-b...
If you see everything through a such a strong lens, you see very little.
Getting triggered happens to everyone but is not an act of intellectual curiosity. https://news.ycombinator.com/newsguidelines.html
That said, they should be well-informed on the subject. So it probably does contain a lot of good information.
Also, when experiments depend on source code, etc. we need real engineering tools/principles applied. (Something like: “you can’t publish paper X if you aren’t including a public repository with build/run instructions”.) Unfortunately, there are all kinds of reasons why scripts/builds could fail just a few months or years later so they would have to be checked too.
I think it would be cool if document-generation caught on in the publication of papers, i.e. the paper itself is generated by running actual experiment scripts and producing charts, etc. from plain text source.
That IMO is a far more dangerous stance. Any study can have hidden flaws, none should be trusted without some form of replication.
How are you going to reproduce a study about a cancer drug? Synthesizing the drug itself is a huge endeavor, never mind finding a group of participants with the exact cancer you need. How would you reproduce the Large Hadron Collider experiments? You'd need to rebuild one of the most technically challenging, most expensive scientific instruments ever made.
Our problem isn't lack of resources. There's 7.5 billion people in the world, I can't believe we've ran out of control groups and experiments to do. Maybe it's a little more difficult with medical edge cases, but there's plenty of other things to research. It's just hard to do it politically, especially with people that are "anti-science", meaning against funding research that might show outcomes that aren't good for them economically, or against their religion.
You must have a totally different definition of "the scientific model" than I do. I suspect (although I could be wrong) is that you have an oversimplified view of just what the scientific process is.
Berkeley University does a good job of providing visualizations on this topic:
Back in early science days, this used to be a valuable section because the pool of "scientists" was huge and they often crossed disciplines. Now you have all sorts of specialties such that the pool of colleagues who are able to understand and assess your work might be, like, 8 people.
There was a Japanese mathematician who recently put out a huge proof that claimed to solve a major math problem (I am forgetting all of the important details here, so forgive me). The news wasn't so much that he created the proof, but that they couldn't find anyone to verify it!
You can't trust blindly the previous studies because they may be wrong, or have hidden assumptions. A well known case is the use of a control group. If you want to study a new drug, you must compare the results of the patients with a similar group that is taking the previous drug and/or a group that is taking no drug. You can't just trust that the cure rate of the previous study is 23.573% and compare it with your results. (And there are some trick to get more reliable results, like using a placebo instead of no drug, and doing a double blind study.)
> Improper use of statistics, arbitrary research techniques, lack of accountability, political groupthink, and a scientific culture biased toward producing positive results
And really science need not move quickly as long as it continues to converge on accurate answers that's plenty useful.
EX: Nutrition research often get's dumped on, but it's far from useless. There are 41 things we know you need or issues will show up: (http://www.nutrientsreview.com/glossary/essential-nutrients + energy + water) Getting the specifics right is complex and we are working on it. Thiamine deficiency from white rice is the kind of thing you need to get right or die, 'optimal health' is a harder target.
Each study is a weak classifier and would have a 'reproducibility crisis' if retested on new data. However after lots of studies of similar phenomena, a strong classifier emerges. In the field, we call this converging lines of evidence.
It can be very hard for many reasons to stand up and say "this prior work is wrong".
If the replication crisis had been as well-known back then, she might have fared better.
To me it brings up an interesting point. If we view experiments as classifiers, then how would a machine learning expert set science policy and practice?
Also viewed this way, it makes me wonder about p-hacking, which increases sensitivity while increasing false positives. Since negative results are not generally reported, I wonder whether p-hacking diminishes replicability at the study level for efficiency at the aggregate level. This is an empirical question, as ethically it is of course a dubious practice under current understanding.
What you describe sometimes happens in a tradeoff between very different ML models (e.g. an ensemble of decision trees versus a single deep neural network has the properties you describe), but within any single paradigm (an ensemble of neural networks versus one NN with more training time) it's the other way around.
edit: When I read about ensemble theory, you receive support for the increased computational cost.
Evaluating the prediction of an ensemble typically requires more computation than evaluating the prediction of a single model, so ensembles may be thought of as a way to compensate for poor learning algorithms by performing a lot of extra computation. Fast algorithms such as decision trees are commonly used in ensemble methods (for example Random Forest), although slower algorithms can benefit from ensemble techniques as well.
But I found this interesting:
Empirically, ensembles tend to yield better results when there is a significant diversity among the models.[4][5] Many ensemble methods, therefore, seek to promote diversity among the models they combine.[6][7] Although perhaps non-intuitive, more random algorithms (like random decision trees) can be used to produce a stronger ensemble than very deliberate algorithms (like entropy-reducing decision trees).[8] Using a variety of strong learning algorithms, however, has been shown to be more effective than using techniques that attempt to dumb-down the models in order to promote diversity.[9]
..which under my analogy would seem to suggest that exact replication is less productive than having a diversity of study designs.
Then there is a problem that an ensemble of biased classifiers (mostly in the same ditection in this case - positivity bias) will magnify the bias.
This is a reason why metaanalyses have to get at the actual raw data to pool and analyse as well as correct for multiple biases. Even then the process is not perfect.
It's so bad now that for any article/study, I look at the sample size first. If it's too small (especially < 100) or they don't say I just ignore it. And if you don't publish or give some estimate of your effect size I just think about it directionally but don't give it much weight mentally.
The current "standard" p-value in many fields, of 5%, is arguably too high. consider throwing a double six in backgammon has probability of 3%. That means that, on some level, throwing 6-6 would be a valid scientific "proof" of ESP. (p-value<0.05) This of course is even before p-hacking.
A high p-value is basically externalizing part of the research cost onto other scientists and in the process creates a lot of false-positive "noise".
https://www.npr.org/sections/money/2018/03/07/591213302/epis...
What does that even mean?
You have one point "it's a conservative think tank" - which does not mean anything - coupled with another empty statement.
Now I sure don't mind criticism of that entity, but I find it ironic that it is done with such an empty statement that's just a not very cleverly disguised type of ad hominem attack.
May I suggest you take some time actually reading the paper and then coming back with criticism of their actual points?
While the report itself may not push an obvious partisan agenda, a critical reader should consider the conservative agenda of the NAS in relation to why they might publish such a report. In the conservative wing's long-standing opposition to science, it's not far-fetched to view this report as being published tactically -- even if all of its points are well-founded -- as something to point to when future efforts attempt to undermine science-based policy on issues like climate change, stem-cell research, abortion, etc.
Especially when the President of the NAS is a public "climate skeptic".
https://www.chronicle.com/blogs/innovations/bottling-up-glob...
Also from the intro: The National Association of Scholars (NAS) has long been interested in the politicization of science.
It's better than presumptively assuming that "Science" is an infallible always black-and-white.
This is such an astute observation that reproduction studies can be used to find a placebo control group for bad scuence. Parapsychology was one such group, but now we can find many others. Brilliant.
Three is NO way to do that. There are ways to misuse scientific results. Premature generalisation, assumptions of safety. Cherry picking to support preconceptions. Attributing strong support to tentative and/or unchallenged results. And more...
Of course science is imperfect, but it tends towards truth over time, whereas dogma and populism have no way of correcting mistakes that isn't just as likely to introduce new ones.
The scientific method is important to be used, but don't trust something just because another person believes it to be science. You need the hypothesis and the verification method. Others (all of us) have the obligation to challenge both the hypothesis as well as the verification methods.
I think it's really dangerous to tie the wrongness of the supposedly scientific facts they spouted to the wrongness of their deeds. Dangerous because if it turns out they were right about some point, then you look like an idiot and have to change your tune. And dangerous because it warps subsequent science, to avoid such embarrassment.
They believed in breeding a master race. In response to this, the acceptable opinion in the West ever since seems to have been that this is impossible, that humans are unique among creatures on this planet in that they are immune to the pressures of selective breeding. We can double the size of our chickens (and breed tame foxes, and smart dogs) but humans are apparently immutable. This is creationist nonsense, and it becomes more and more obviously nonsense every year. We need not have painted ourselves into such a stupid corner.
Frankly, genocide, or at least dehumanization and extreme indifference, seems like the logical endpoint of sincere belief in the concept of a "master race" which is inherently good and is hamstrung by perfidious inferior races. Once you've determined that a member of one race is worth less than another, it's natural to judge that the death of a member of this inferior race would be justifiable to save the race of a member of the superior race, and in this way the concept of genocide becomes, in your mind, a "defensive" action.
If I'm not mistaken, the Brahmins tick most of your boxes. Are they alright? Some would argue that they go in for a bit of dehumanization... but nobody is proposing a world war to stop them.
Actually I'm alright with people believing lots of crazy shit, in private. Because I think intellectual monoculture is pretty dangerous.
I do find it a little strange how many people want to turn the concept of "diversity" on its head and tell me the real diversity problem is that we don't let enough people with racist views into power.
My objection is to conflating them with scientific matters, i.e. empirical claims, especially about human biology. I worry that doing so actually weakens our arguments against such evil. We shouldn't build into our case against them scientific claims on which we may ourselves be proven wrong. It's not necessary.
(And, much as I enjoy using words like diversity in their older meanings, in the hope of tripping people into thinking, here I resisted the temptation. Intellectual monoculture was however a feature of most of the evil regimes of recent times, too, and one of their weaknesses.)
By way of analogy, when someone tells you he believes global warming is impossible because it would go against the Bible, you don't have to wonder that much what he would do about carbon emissions.
EDIT: Krugman had this to say about the R&R austerity paper "What the Reinhart-Rogoff affair shows is the extent to which austerity has been sold on false pretenses. For three years, the turn to austerity has been presented not as a choice but as a necessity. Economic research, austerity advocates insisted, showed that terrible things happen once debt exceeds 90 percent of G.D.P. But "economic research" showed no such thing; a couple of economists made that assertion, while many others disagreed. Policy makers abandoned the unemployed and turned to austerity because they wanted to, not because they had to."
https://en.wikipedia.org/wiki/Growth_in_a_Time_of_Debt
Particularly telling: "The paper was published in an annual 'Papers and Proceedings' edition of The American Economic Review that was not subject to the same peer-review standards that other editions use before publication"
What was said above was "one of the most horrific and important examples of widely accepted science". Important words here are "widely accepted science". So science didn't go wrong, the LABEL of science was attached to something that didn't meet the scientific method. That something was then sold to people in power, and the science LABEL was used to convince people of the correctness of that something.
People are really quick to say this, but how many people have actually studied the 'racial science' literature of the 20s and 30s and figured out to what extent it meets the methodological standards of today? Yes, we know that they drew incorrect (and reprehensible) conclusions. It doesn't necessarily follow that their methods were any worse (epistemically speaking) than methods that currently pass muster.
Just the premise of the "racial science" studies would render them invalid, how could they even have controlled for the myriad variables that we have no idea how to control today?
I don't need a specific study to know that. Any sociology/economic/psychology research should be taken with a grain of salt.
So how is that not an example of "science going wrong"? Unless you are proposing that science is right by definition, or that governments should ignore most of the branches of science that are actually relevant to government.
There are plenty of branches of physics that are at least as speculative as sociology, economics and psychology (e.g. cosmology). The only difference is that these branches of physics have few political implications.
Governments (and people) should evaluate the merits of the information they receive. Properly conducted and repeated experiments with noted mechanisms of actions (e.g. vaccines) should have more weight than studies about trickle down wealth or whatever. Obviously, decisions have to be made, and some will be wrong unfortunately.
This is a good example of why it's not so simple as you're making out. Earlier, you put biology on your list of unreliable branches of science, and you claimed that you could know that the racial science of the 30s wasn't "real science" simply because it consisted of "studies about people". But vaccine science falls into both those categories!
In reality you're deciding whether or not any given branch of science is methodologically sound not by examining any actual studies in the relevant disciplines, but simply on the assumption that any study which has arrived at the "wrong" conclusion must have had a bad methodology.
Just stating that all of soft science is dodgy doesn't really distinguish us from them, does it? We do a lot of soft science!
https://en.wikipedia.org/wiki/Replication_crisis
The point is that some research is weaker than others, and the reasons why haven't changed since WW2, so if it's a problem now, then it was a problem then.
Soft science has many questions to which we'd love to have solid answers. So (perhaps unfortunately) it's not going away, and I think we need to get better at sorting out what it tells us.
More importantly, the example of the Nazis is useful in an ethical argument, because they're an almost universally acknowledged example of evil. Much like we aim to test our programs with extreme inputs, we can use the "extreme input" of Nazism to test our ethical arguments. If I posit some universal ethical standard, and then find that it doesn't seem to work if we take the Nazis, we've proven by counterexample that my standard is not universal after all.
More seriously, it wasn't just this group that believed in this stuff, it was the many many institutions in the western world, as emodendroket points out. Certain departments at Yale, most notably, supported eugenics.
https://embryo.asu.edu/pages/american-eugenics-society-1926-...
edited portion in case people think I meant all of Yale, bad generalization, my bad.
Additionally, using science and other expert judgments can easily tend toward simply choosing experts and studies that align with whatever policy you want to push for unrelated reasons, or outright seeking to manipulate research for political ends.
Of course it can, but that's equally true for any other way of making decisions. So it's not a downside of science in particular.
Talk about science creating a mind boggling ongoing health crisis by telling people to limit their fat intake and replace it with cereals and bread.
Current scientific consensus is that it is caloric surplus - confounded by a bunch of poorly researched factors. For weight gain source does not seem to matter too much. This does not say anything about dietary specifics or how they affect intake and metabolism.
Technically nutrition science does not provide strong support for such statements as this carb vs fat one based on interventional studies.
Epidemiology alone does not tell you "therefore this change will be beneficial"...
I've looked into that and all I can say is the history is disputed - some people say there really was a nutritionist consensus, others that this was a popular misunderstanding based on overzealous popularisation of a small number of papers (similar to the myth that the consensus among climate scientists in the '70s predicted an ice age) and that working scientists never thought this.
Either way it's a failing of the process, but the appropriate response is quite different; in the first case all we can say is mistakes do happen, in the second we need to get better at ensuring that the real scientific consensus gets translated into appropriate policy actions.
Why? Do we believe that individuals are likely to be better informed than governments? (why?)
Uncertainty is a fact of life. Any decision-making process will lead to some mistakes. That's not a reason to give up on trying to make things better. If the government believes, based on the best evidence available at the time, that a particular fat tax policy (say) will, overall, be good for society, surely it's better to implement that policy than not.
I certainly believe that some individuals will be better informed and better able to translate that information into the best course of action for themselves given their goals than governments, especially "governments" as represented by a patchwork of laws and regulations the impact a complex thing like individual health. This is true even at the time that new policy is enacted and tends to become more true over time due to the inertia of government policy that goes down the wrong path.
Some individuals, sure. Most individuals, probably not.
I agree that government policy has inertia, and that overconfidence on uncertain issues can lead to imposing well-intentioned policy that ends up being harmful. But it's equally possible to make the opposite mistake, and hold off introducing policy changes for too long because of perceived uncertainty. E.g. car seatbelt laws and tobacco advertising laws should have been introduced earlier than they were. I don't think there's any general principle we can draw here; some government policymakers could do with more humility, others could do with more confidence, but all any one of them can do "from the inside" is try to figure out how strong the evidence actually is on a given issue, and use their own best judgement in determining whether there's a clear enough scientific/medical consensus to warrant policy changes, or not.
I thought the parent comment made it clear: the alternative we are supposed to use is "democracy". That is, the people who are ruled should determine the rules that are imposed on them.
Now, democracy does not preclude the use of science to make decisions. The voters may rely on science to inform their decisions (when the science is available). It does however preclude a rule by scientific experts/bureaucrats.
But I also think it's foolish to frame the decision as solely one of "science" vs. "gut instinct". The are many phenomena, especially when it comes to human society, that are too causally dense for us to sufficiently tease out the causal factors to make accurate predictions.
In those conditions, the goal should be to minimize the consequences of being wrong. We do that by decentralizing decisions to the greater extent possible.
I would expect the result to be strongly biased due to societal norms, much less direct manipulation.
Suppose a majority wants strong social programmes and basic income. The tiny problem is the economy is nowhere near big enough to afford it. Satisficing (!) executive branch puts some of the policies starving innovation and education budgets. Causing long term harm and stagnation.
Where is the minimax decision in here? Over what timeframe?
(P.s. I could've as easily picked liberal, fascist, military or conservative example. This one is relatively easy to understand.)
The second part of my response was to your framing of "science vs. gut-instinct". My point is that sometimes ignorance is thrust upon us by the complexity of the circumstances. In that case, my point was that decentralized policy making (separate from any notion of voting) was essential to reducing the worst case.
In other words, I am not claiming that democratization "results in minimax optimal decisions". I am claiming that decentralized governance structures do "result in minimax optimal decisions". Does that change your question?
edit: fixed spelling
In some cases yes. In others this doesn't seem to be the case - e.g. decentralised housing/planning/zoning policy seems to have lead to worse outcomes than more centralised approaches.
I haven't surveyed zoning policies, so I don't know whether I agree with your conclusion, but I'd like to clarify a couple things.
First, we must ask ourselves what the goal of optimal zoning policy is. Is it to maximize the number of people who live there? Achieve an ideal rent/wage ratio? Maximize the efficiency of shared infrastructure?
Perhaps it is to maximize the desirability of living in an area. If that is the case, then if people are complaining about the house prices that may mean you're doing something right.
In any case, assuming we settle on a shared metric (which may involve a combination of these various factors), we need to distinguish between comparing the average case and the worst case performance of centralized vs. decentralized zoning policy.
I would say that my claim is that decentralized zoning policy will have a significantly better worst-case than centralized zoning policy, and that crafting a good zoning policy is enough of a gamble that I'd rather settle for a less-than-ideal decentralized policy than risk a horrendous centralized one.
Now, to tie back to the original debate of scientific expertise in particular: would adding a group of economists and social scientists to the central planning committee tilt the situation in favor of central planning? I very much doubt it.
"At the core […] is the idea that people should design for themselves their own houses, streets and communities. This idea […] comes simply from the observation that most of the wonderful places of the world were not made by architects but by the people."
— Christopher Alexander et al., A Pattern Language, front bookflap[1]
Because the costs of building new housing are local but the benefits are global, or at least more widely distributed than the costs. If one borough builds denser housing, that borough is the one that has to worry about building work, plumbing, schools and all the rest of it for those residents, while businesses from all its neighbours benefit from more customers and cheaper employees. For a starker example of the same effect think about taxation and public services: if we relied on local taxation being spent locally then rich areas would have low tax rates and high-quality public services (because even at low rates they can still raise a lot of money) while poor areas would have both higher taxes and worse services.
Localised government is rather like privatisation: it works well when the costs and benefits are on the same scale, but rather less well when there are externalities.
And even if we had perfect predictions, we would still have to choose what to do.
For example, if we had a perfect economics, we could predict exactly how many people would be made unemployed compared to how many made how much richer by raising the minimum wage (or trade barriers, or taxes). How should we weight these?
I think it's easy to greatly overstate how useful science is to policy-making. There are narrow areas where they should (and do) call a specialist in. But the big debates are not over factual, empirical, matters.
One less drastic solution may be to require a full disclosure of the process for all experimental results (to allow reproduction) and place much higher value in funding decisions on reproducibility.
One particularly egregious approach reproducibility requirement can address is doing enough "independent" studies until you like the result, then report that one study only (e.g., toss a coin 10 times; repeat as a new study until you find 10 tails; report a new discovery). My 2c.
This happens not only in which studies get submitted to the public, but also which submitted studies get chosen for publication. Journals are proving to be a poor way to handle the amount of science humanity is doing.
Environment has changed enough - especially number of people - that attempting to use traditional methods en masse could be very risky.
(Much less in nutrition or medicine where we have made major gains on the average.)
But science tells us nothing about what should be. That’s the realm of morals, ethics, and its applied (mis)application: politics.
But again, getting back to agreeing, scientific findings should not be used blindly, they should be used to inform. And as reproduction of findings occurs, the the topic that is being informed gets re-enforced or course corrected. It is very similar to policy, you have to try things. Some times the policy does what you want, other times it doesn't, and the process should continue and be correcting.
Science is the only tool specifically designed to weed out falsehoods and reach an objective consensus based on how nature works.
How is this "subject to falsehoods just like any other human endeavor"?
There is no substitute for science.
We all seem to agree that democracy only works when the population is informed. Well, are we going to inform ourselves with the beliefs and tweets of those which we grant moral authority? Or science?
"Science, however, cannot create ends and, even less, instill them in human beings; science, at most, can supply the means by which to attain certain ends. But the ends themselves are conceived by personalities with lofty ethical ideals and—if these ends are not stillborn, but vital and vigorous—are adopted and carried forward by those many human beings who, half unconsciously, determine the slow evolution of society.
For these reasons, we should be on our guard not to overestimate science and scientific methods when it is a question of human problems; and we should not assume that experts are the only ones who have a right to express themselves on questions affecting the organization of society."
So my less dystopian future goes like this. Train lots and lots of physicists, chemists, engineers, programmers, materials scientists, biologists, mathematicians. And (shock horror) through higher taxes employ them. Employ them on promising ideas, and on impossible ideas. Employ them on the problems of the day, and on arcane research that will almost certainly never bear fruit. And sure, right at the top in the Apples and Googles of the world, you'll have the best and brightest pulling together all of the discoveries made without all these reproducibility issues.
Or we could continue down the dystopian path. The low tax path where there really are only meaningful jobs for the best and the brightest. Where we rely on Apple and Google to decide what ideas are worth pursuing. Where the skills and abilities of a great many are not used. And most of us fight for the crumbs from the oligarch's tables. And the oligarchs? For every Elon Musk, there will be 10 who just want the biggest yacht and the prettiest mistress.
Let me say that again. In a low tax environment, the skills and abilities of a great many will not be used. And we will all be poorer for that. Genuinely poorer, because great discoveries will not be made. And it is happening now.
And I'll go one step further. We are also wasting the talents of our best writers and artists. The BBC and other national broadcasters are squeezed. Squeezed so that the idea of something as original and amazing as Monty Python appearing now is laughable. And again, we are all the poorer for that. Not just the lucky ones who get to make amazing shows or works of art, but all of us who never get to revel in them.
The low tax, small government world is a step backwards. If you wish to sum it up succinctly, you may say it is a world where, in order to placate the very rich, we take away opportunities from many, and impoverish the world as a whole.
That doesn't seem intuitive at all.
>Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something.
Sure, we will undoubtedly get better society-wide results if we identify a subset of elite students (notably defined primarily in terms of STEM fields) and only "bother" educating them, leaving everyone else to Just Trust Them™ and underutilizing the long tail of talent.
And then there's the premise that society somehow isn't looking to utilize the existing population of elite students, which of course it is, no shortage of organizations looking for outliers.
About the only good point there is that our system for allocating the best/brightest to certain problems could use improvement.
More public funding for basic research (including reproduction studies) would probably deliver major long term benefits.
I do suspect that profit motive isn't enough to drive effective R&D, though, I just don't know exactly how to articulate a model for why except at the basic level that that truly searching R&D has a higher risk profile than other opportunities for profit.
I support the idea of more tax-supported public funding for a variety of things. I'd also support reshaping the tax code to incentivize R&D. But I'm also not sure that's a going to give everybody their opportunity -- I'd imagine there'd still be lucky ones there too. And we'd still have a lot of work to do to address incentives that drive reproducibility issues.
The scientifically minded are not the ones who will be deciding what's worthy of research. Politicians and bureaucrats will, and there are plenty of them who believe in all kinds of pseudo-scientific nonsense. How does your system prevent them from shoveling tax dollars into research on homeopathy and young-Earth creationism? And how does it prevent them from using funding to incentivize scientists to find the "correct" results?
To phrase my question more generally: I would like to know how your system would avoid making science merely a tool of political interests instead of a tool of moneyed interests.