History is invaluable and the lessons learned can provide some degree of predictive power. Yet it would be unreasonable to try to argue that something ought be enacted as, as a matter of fact (or at least heavily supported hypothesis), because of results derived from the study of history.
The social sciences in general seem to be want to be allowed to behave as a "historical and descriptive" field, yet be received with the same credulity as, for instance, experimental physics. Ultimately, I think the field needs to choose to go one way or the other.
See: Stereotype Threat, Implicit Association Test and Implicit Bias, Power Posing, et al.
Maybe my point is off the mark given your "qualitative" qualifier, but about the social sciences: behavior is a phenomenon. It's complex and can be examined at multiple levels, but it is a phenomenon. We have two choices in understanding it: to try to approach it through quantitative, methodical, logical, observation and prediction or to leave it to other approaches. If you do the former, it's extremely difficult and you end up struggling even to quantify anything. But if you do the latter, you are leaving something extremely important to wild speculation, unmoored opinion, and fairy tales. The quantitative social scientist faces a dilemma of being ridiculed for trying to approach something scientifically and rigorously when the alternative is to leave it to forces that are much worse. It's simply false that there are no provable metrics, unless you literally mean in the sense of proof, but that stands to be true of many scientific disciplines. And the reductionistic reply that everything is neuroscience is akin to telling computer scientists to abandon algorithmic theory because it's not about electrons, or the astronomer that it's all chemistry and physics anyway, or the evolutionary biologist that it's all molecular biology.
In any event, the replication crisis is not about statistics, and it's not unique to the social sciences. Studies have shown the same problems exist in many fields, although they do appear to be worse as the field increases in complexity of subject matter. Oncology is also plagued with such problems, for example, although no one would say that cancer is a soft topic. The social sciences have a tendency to look inward (appropriately enough), and so in many ways, are bearing the brunt of the negative publicity they are shining on themselves and everyone else.
The problem I see extremely complex, and won't be fixed by any statistical solution. There are many causes, too many to name (I could probably write a book on the topic and it still wouldn't be enough). In general, they can be summed up by saying that there's been a fundamental shift away from soundness, and toward celebrity and novelty, at a time when the justification for that celebrity is weakest, if it were ever strong. Tenure protections have eroded, accelerating the problems, and funding has been cut substantially from universities, even as there's an oversupply of new scientists entering the field. It's as if we decided as a society to honor war veterans by celebrating only those few survivors with the wildest, most glamorous tales of adventure, ignoring the dead who made the actual contributions. To make this analogy more accurate, imagine that we've grossly romanticized war publicly, lavished attention on these survivors, even while we've dried up job opportunities for our best and brightest young citizens in society at large; we've all but cut public funding from the military in practice, and replaced it with a thinly veiled system of contracted private mercenaries, who are told their obligation to the country is to raise funds in whatever way possible.
I'm exaggerating and being dramatic to make a point, but I think the parallels are pretty close.
I'd go so far as to say the problems aren't really unique to science, as is reflected in comments here about software projects and the like. We have serious problems in society (at least in the US) with overhype, and gross distortions in how we compensate for work and contributions to society in general. This is true in business, politics, healthcare, everything. I think science is just one more casualty of this, but it's been hidden for awhile due to the gradual nature of the disease, and the cracks are starting to show.
Perhaps you should write a book!
I could probably write a book on the topic and it still wouldn't be enough
Then perhaps you should write two books.
I think this is a story that needs to be told its full complexity, and you seem to be in a good position to do so.
It's as if we decided as a society to honor war veterans by celebrating only those few survivors with the wildest, most glamorous tales of adventure
An excellent analogy.
I think science is just one more casualty of this, but it's been hidden for awhile due to the gradual nature of the disease, and the cracks are starting to show.
This is one of the parts that confuses me: what sets the pace for the "race to the bottom"? If the incentive structures haven't changed, why did it take so long to reach where we are, and why are some fields still relatively immune?
William James (paraphrased): "James acknowledges that in our scientific age, there is something dubious about the voluntaristic view that, in some circumstances, we can legitimately choose to believe in the absence of any objective justification. However, he claims we naturally do so all the time, our moral and political ideas being obvious examples. When you believe that your mother loves you or in the sincerity of your best friend, you have no conclusively objective evidence. In addition, you will never be able to secure such evidence. Yet it often seems unreasonable to refuse to commit to believing such matters; if we did so, the pragmatic consequences would be a more impoverished social life. Indeed, in some cases, believing and acting on that belief can help increase the chances of the belief being true." http://www.iep.utm.edu/james-o/#SH4a
Please distinguish proof and evidence.
> Not all that counts can be counted.
How do you know that?
> However, he claims we naturally do so all the time, our moral and political ideas being obvious examples.
Yeah, and so incredibly successful, too, how people believe that black people are inferior or homosexuality is an abomination, without any interest in objective evidence.
> When you believe that your mother loves you or in the sincerity of your best friend, you have no conclusively objective evidence.
Yeah, it's absolutely impossible to know whether your mother or your best friend is objectively treating you well or terribly, so what can you do? You just have to assume that they love you!
> In addition, you will never be able to secure such evidence.
WTF? Next you tell me it's absolutely impossible to know whether animals are aggressive and you will never be able to secure evidence?
> Yet it often seems unreasonable to refuse to commit to believing such matters; if we did so, the pragmatic consequences would be a more impoverished social life.
WTF?! Yeah, believing and acting as if someone loves you without any evidence is so socially enriching there is even a term for it: It's called stalking.
I don't think the comment you were replying to was a model of clarity; the quote is a paraphrase of a paraphrase of an argument in James' The Will To Believe. I'm familiar with that essay (not one of his most impressive) so maybe it was easier for me to hear what they were trying to say. But "WTF?!" etc isn't helpful. ... Objecting to every sentence like that doesn't even begin to address what they were trying to say. Anyway, all of this seems obvious, I think you know this, I'll stop here. Your comment read like you didn't care about what they were trying to say, just to knock them out of the water with any available weapon, which you seem to think you had done.
Well, yes, I guess I was objecting to everything they said. But not because they said it, but because it's a bunch of terrible arguments.
> and tiresomely missing the point at every turn.
Such as?
> From the initial pedantic "Please distinguish proof and evidence."
Well, except that it's probably not pedantic, as their argument probably hinges on that fallacious equivocation.
> Do you talk to your friends like that?
Arguing in public is not talking to your friends.
> It's already headmasterish and snarky.
Which invalidates the argument how?
> It's not clear what you're asking for.
If you think that I am misunderstanding your(/their) argument or you see a flaw in mine, I am asking for you to explain the error in my reasoning.
> But you move on after 5 words to dispatch the next offending sentence
The offense is something that you choose to see, not something I said.
> like they're a succession of pesky bad guys in a kung fu movie waiting to be swatted down by our hero.
Are they not?
> I don't think the comment you were replying to was a model of clarity; the quote is a paraphrase of a paraphrase of an argument in James' The Will To Believe. I'm familiar with that essay (not one of his most impressive) so maybe it was easier for me to hear what they were trying to say.
Well, I don't know that essay, so it's possible that I am completely missing the point, in which case, feel free to explain. But then again, I have heard arguments of that form more than once, and so far whenever I looked any closer, it boiled down to the same flaw in the reasoning.
> But "WTF?!" etc isn't helpful.
Helpful for what?
> Objecting to every sentence like that doesn't even begin to address what they were trying to say.
Well, OK, what were they trying to say, then?
> Anyway, all of this seems obvious, I think you know this, I'll stop here.
You think that is helpful?
> Your comment read like you didn't care about what they were trying to say, just to knock them out of the water with any available weapon, which you seem to think you had done.
So, my goal was to knock some person I don't know out of the water regardless what their argument was? I'm not sure that makes sense ...