Academic success is either a crapshoot or a scam
grasshoppermouse.github.io
grasshoppermouse.github.io
All of this, though, shows how hard those disciplines are. Relative to that, hard sciences are easy, because they study precisely defined things pretty closely entangled with laws of physics, and you can't easily bullshit mother nature. But as you move away from physics, complexity rapidly increases, the questions themselves become fuzzier, and the answers less clear.
So IMO, the probability that academic success is crapshoot or scam increases with (Kolmogorov) complexity of the subject domain.
Physicsts absolutely could be performing experiments with poorer statistics; it'd be cheaper, more advanced things would be done sooner, and we'd be having a replication crisis.
If the social sciences were willing to go a century without discovering anything exciting, they could achieve whatever level of certianty they wanted on their current budget. (At the expense of new discoveries.)
https://cds.cern.ch/record/2282544/files/arXiv:1709.05260.pd...
Your statement about huge numbers of authors (as well as the great number of theoretical papers that have to wait a very long time before being confirmed or denied) is true.
The downside is we end up building simplified models well Y is correlated with bad things so we should treat Y as a bad thing. Often this actually works really well, but also opens the floodgates for all sorts of issues.
That's not to say a less practical approach that simply looked for the details would be more effective. But, it would look very different.
I'm always baffled by this. I can understand why we should move these drugs into practice. But, if the way you conduct research stops at X works - you should have a good look at the way you conduct research. I can completely understand short term gains but ignoring long term gains is very unscientific and immature.
You can't just take a person apart to study them while they're alive, and you can't easily reconfigure them to account for variables either.
Besides, there's plenty of Y where they were sure it was going to work, it fit all the models and theories, it did the right thing in mice, and then in a trial Y fails catastrophically.
no, medicine really is that much more complex (which is not to say physics isn't complex and that physicists aren't very intelligent). a model of a single-celled organism, or even of a protein structure, is orders of magnitude more complex than, say, the three-body problem or the navier-stokes equations.
we're at the very beginning of understanding truly complex systems situated in truly complex environments, which is what medicine faces. we have millenia of exciting medical discoveries in front of us.
Since you bring up CPU design, the appropriate comparison would be that medicine is to chemistry what CPU design is to physics, if by "CPU design" we mean reverse-engineering alien technology that operates on the edge of our understanding of physics and way beyond our ability to manipulate the physical world.
Medicine is practical and accepts not understanding the details because it has no other choice. In that regard, it's similar to social studies because the latter can't get into details without fully reverse-engineering a human brain either.
I'm not sure a researcher enjoys the same freedom to put together medical experiments as physicists and chemists do.
Moreover you comment sounds very disingenuous. It's like claiming that electronics is a realtively simple research domain because it still operates on well understood electromagnetics.
Meanwhile do keep in perspective that even today's state of the art in medicine involves various forms of mutilation.
Consider we generally do blood not lymph tests. That's a mix of practicality and tradition, but one one of many areas of basic research mostly left alone. Sure, doctors physically examine for enlarged lymph nodes, but that's because the immune system is largely based around the lymphatic system because that's mostly how diseases move around the body not through the blood.
Getting back to what I said, we can do a lot of high quality medical research that may or may not be useful in the long term, but we don't approach it as a research problem we approach it as an engineering problem looking for solutions to real world problems.
PS: As to CPU vs Medicine vs Physics. We are comparing numbers do large it's hardly meaningful, 10^20 (CPU) vs 10^30 (Medicine) vs infinity (Physics) all round to bigger than you can comprehend.
Psychologists don't have a medical degree, psychiatrists do and they could hardly be called social scientists.
Hard sciences also don't have a very high replication rate, they have the same incentives which the article talks about, and the same publishing bias problem.
In some areas there isn't even validation at all, it's just "we put this interesting thing in the expensive and exclusive machine, and these are the results you guys". For instance testing different fuel mixtures in a rapid compression machine, testing prototypes in a wind tunnel/flow rig/anechoic chamber, etc.
Successfully building your biological proof-of-concept is lovely and remarkable but depends on a whole bunch of research done with p-values.
For instance, in the fields of electrical engineering and computer science, you can "prove" your results by either building/fabricating a prototype or writing a piece of software.
The studies under question here don't really do that. Physicists often test their hypothesis (eg this satellite should undergo x seconds of time dilation), but in fields like biomedicine or psychology that is extremely rare. Instead they usually test a "null hypothesis" that there is no difference (eg the population mean of the treatment and control groups is the same) when their hypothesis predicts some vague but non-zero difference.
1) Scientific method (what you originally described)
2) NHST (where some default hypothesis is tested)
3) Engineering (where you perform some kind of feat to demonstrate capability)
The problems seem to be with people doing #2 rather than the others.
One: at this state in my career I need to publish a lot. There is kind of a speed limit on how fast you can publish empirical work, but with theory the upper bound on your publication rate is how hard you work and how creative you are.
Two: I feel more honest about my theoretical work. My models and simulations are so obviously contrived that there is no danger of anyone mistaking the map for the territory. By contrast, if you come up with some effect of size X in an empirical paper it might be reported in the media as some kind of golden truth like the charge of an electron.
I think if I spent like 12 or 15 months on an empirical project, I also would be so desperate to get a result out of it as to fudge the result. Even among intuitive relationships, the chance you can honestly and provably identify an interesting relationship is really low. For this reason, I read theory and description from all kinds of researchers, but I tend to only be interested in the empirical work from the best-of-the-best. In my field, I think some of the best work is from Rebecca Diamond at Stanford, and one of her most recent papers had a negative result.
I also really discount job market papers: these people spent years on a single project and their whole future is riding on finding something cool, so the incentive to fudge is just extraordinary. It's kind of like trusting a judge to try a case in which his son is the defendant. No matter what he told me about how fair he'll be, I wouldn't believe him.
It's just sophistry all the way down.
This is typically not the case. Academics will have a number of students working for them and many academics push them incredibly hard while doing very little of the actual research themselves. Instead they spend their time chasing funding. Projects are often built on prior work so many new papers explore new areas of a previous discovery or measure said discovery in a new way. So you’re not discovering three new things every year. Some academics will even be bold enough to publish the same data in multiple journals with minor changes in analysis.
With academic research it’s often a case of two steps forward one step back. You spend years working towards deep expertise in a field through postgraduate studies and then a postdoc or two. Then when you become academic staff you suddenly become an inexperienced manager and teacher. So you pass the work to inexperienced students who spend years relearning everything and often suffering from over aggressive supervision or complete abandonment. The funding bodies, as is their right, are increasingly looking for commercial outcomes to research so the scope moves from Research to r&D with a little ‘r’. So we see research on battery technology making ludicrous claims or graphene in all the things rather than real progress.
This stunts the potential of scientific discovery in favour of pushing revenue generating students through universities and chasing funding for short term gain.
What you're describing is research in fields with lots of grant funding. In some fields, you don't spend your time writing grants, and you don't hire grad students to do your research for you. In those fields, teaching is important because you do a lot of it. In grant-heavy fields, it wouldn't matter if the university shut down their undergraduate programs, because the emphasis would still be on bringing in grant dollars.
Perhaps in a field like philosophy or math at a very small school it's possible with amazing individual results, but it's uncommon in my experiences which include watching numerous friends at many different schools (>10) and fields (bio, chem, phys, geo, cs, math, philosophy) both public and private.
Maybe you mean outside the US? I'd believe that. Europe hasn't gone as far down the path of non-tenured teaching faculty as the US, but due to money they seem to be starting to.
I think you ignored his point altogether. I was at a top university. While it was common for an engineering professor to have between 5-10 students and 2 post docs, the folks in the social sciences like anthropology have much less (2-5 students is what I would guess). Someone has to pay them and those disciplines have little grant money.
Just checked my school's anthropology department: 25 faculty members and 75 graduate students - so roughly 3 per faculty member. Their Electrical engineering department? Almost 6 per faculty.
And that feeds the birth/death demographics problem where the reason why its hard to get a tenure position is the previous generation had to produce 10, 20, 30 qualified grads in order for them to get their previous generation tenure position, so 10, 20, 30 minus one person are going to get training and education for a position they will never be allowed to fill. I'm sure they'll find something to do with their lives, although to get position XYZ they never trained for, they'll be competing with people who actually trained and studied for XYZ work, and in the end it just means a different name is on the unemployment lists.
Academia is a weird pyramid scheme of human suffering.
I avoid these type of "research" articles like plague. There probably exists a "study" that proves and disproves virtually anything. For example, study suggests coffee prevents cancer and vice versa. Such pseudo (and often badly done) statistical voting type of work shouldn't even be called "research" and people doing these shouldn't be called "scientists" (the word "surveyors" seems more appropriate). There is now a cottage industry of pushing out absurd correlations that would be immediately be picked up by media because, you know, so called "scientists" did that.
The fact that you’re making overly broad claims accusing people of making overly broad claims doesn’t instill confidence that you know what you’re saying. If you’re such a champion of science, why not state your claims minimally concrete terms. And ideally make falsifiable claims.
The idea was to look at unfruitful research as something useful for the community and give some credit to time and resources spent doing something that did not work (and should not repeated by others). I still wonder if that would create the wrong incentive though and if that could be feasible at all (who likes to talk about money and time wasted?)
EDIT: fixed wrong sentence
A bit like the Journal of Negative Results in Biomedicine, the Journal of Negative Results — Ecology and Evolutionary Biology and the psychology Journal of Articles in Support of the Null Hypothesis?
> several phd undergraduate friends
People doing PhDs who are simultaneously undergraduates?
http://www.journals.elsevier.com/new-negatives-in-plant-scie...
The problem is the following: If you seriously want to fix science you don't need one such journal. It would have to be the norm.
It's simply not plausible that the majority of empirical research leads to positive effects. Even if you publish a few negative results (or "failures") - that doesn't even begin to address the problem of publication bias seriously.
As long as the majority of scientific publications are not "failures" we should assume that the majority are false positives due to p-hacking or publication bias.
http://www.journals.elsevier.com/new-negatives-in-plant-scie...
On a slightly different note, I would love to see a section in journals devoted to publishing the results of duplicated experiments (performed by graduate students and interns, probably).
It’s as if the most important thing about software is that other people are talking about it.
Time - do it as a hobby so it's limited to people who are passionate about discovering something, not just trying to rack up points.
Access to human subjects - get people off the street. Universities can make access to student subjects easier but they're not representative of the world's people anyway so it's not that valuable.
Money to pay the human subjects - save up? People who're passionate about cars, for example, somehow find the money to build modified cars and don't need funding.
Ability - With the oversupply of researchers, maybe just leave it to those who can do it under their own steam? Hopefully undergrad degrees taught them enough to get started. This article says that university researchers aren't doing it with skill anyway but with luck or cheating.
Motivation - if nobody can be bothered investigating some effect, then maybe it's not really that interesting anyway and there's not much value in studying it?
What am I missing?
On the other hand, outside of institutions there are cases like https://www.gwern.net/zeo/Vitamin-D and I hope there'll be more.
You're really talking about small scale interviews or qualitative work. That's an aspect of social science but it's not all of it.
That’s the money quote. This is mainly applicable to “empirical social science”, which many already know as a big scam.
Once I found a major bug in some code that was the basis of a significant chunk of his major book and an important article. (Basically an optimization process that didn't work but gave some nice results he built a story around.) When I wrote him a report on what was wrong and why, he didn't respond. I was supposed to write a paper based on the same code and all he would say was along the lines of "it should work". Over time I realized that the implication was that I should "make" it work. Of course, he never said those words but over time the message was clear. After beating my head against a wall for a long time, I eventually walked away from the project. Technically bad for my career, but I honestly couldn't see the point in being at a top university essentially selling snake oil.
That being said, there are fields of social science that are much better: demography, political science, economics and even psychology (which is very difficult at the best of times!) are trying to clean up their act. Basically, any field of study with consistent data available openly is probably pretty decent.
There are other fields like business/management where the pay is generally good (i.e. motivation for a nice job with good perks is there), many of these journals don't require data to be released, and the samples being studied tend to be convenience samples or impossible to reproduce. I generally don't trust the work in this area.
Sadly, policy makers are often willing to take questionable academic studies that support their preconceived notions. When these things don't work out, the general public becomes jaded that their taxes were wasted... it makes all social science research look bad.
William James (paraphrased): "James acknowledges that in our scientific age, there is something dubious about the voluntaristic view that, in some circumstances, we can legitimately choose to believe in the absence of any objective justification. However, he claims we naturally do so all the time, our moral and political ideas being obvious examples. When you believe that your mother loves you or in the sincerity of your best friend, you have no conclusively objective evidence. In addition, you will never be able to secure such evidence. Yet it often seems unreasonable to refuse to commit to believing such matters; if we did so, the pragmatic consequences would be a more impoverished social life. Indeed, in some cases, believing and acting on that belief can help increase the chances of the belief being true." http://www.iep.utm.edu/james-o/#SH4a
Please distinguish proof and evidence.
> Not all that counts can be counted.
How do you know that?
> However, he claims we naturally do so all the time, our moral and political ideas being obvious examples.
Yeah, and so incredibly successful, too, how people believe that black people are inferior or homosexuality is an abomination, without any interest in objective evidence.
> When you believe that your mother loves you or in the sincerity of your best friend, you have no conclusively objective evidence.
Yeah, it's absolutely impossible to know whether your mother or your best friend is objectively treating you well or terribly, so what can you do? You just have to assume that they love you!
> In addition, you will never be able to secure such evidence.
WTF? Next you tell me it's absolutely impossible to know whether animals are aggressive and you will never be able to secure evidence?
> Yet it often seems unreasonable to refuse to commit to believing such matters; if we did so, the pragmatic consequences would be a more impoverished social life.
WTF?! Yeah, believing and acting as if someone loves you without any evidence is so socially enriching there is even a term for it: It's called stalking.
I don't think the comment you were replying to was a model of clarity; the quote is a paraphrase of a paraphrase of an argument in James' The Will To Believe. I'm familiar with that essay (not one of his most impressive) so maybe it was easier for me to hear what they were trying to say. But "WTF?!" etc isn't helpful. ... Objecting to every sentence like that doesn't even begin to address what they were trying to say. Anyway, all of this seems obvious, I think you know this, I'll stop here. Your comment read like you didn't care about what they were trying to say, just to knock them out of the water with any available weapon, which you seem to think you had done.
Well, yes, I guess I was objecting to everything they said. But not because they said it, but because it's a bunch of terrible arguments.
> and tiresomely missing the point at every turn.
Such as?
> From the initial pedantic "Please distinguish proof and evidence."
Well, except that it's probably not pedantic, as their argument probably hinges on that fallacious equivocation.
> Do you talk to your friends like that?
Arguing in public is not talking to your friends.
> It's already headmasterish and snarky.
Which invalidates the argument how?
> It's not clear what you're asking for.
If you think that I am misunderstanding your(/their) argument or you see a flaw in mine, I am asking for you to explain the error in my reasoning.
> But you move on after 5 words to dispatch the next offending sentence
The offense is something that you choose to see, not something I said.
> like they're a succession of pesky bad guys in a kung fu movie waiting to be swatted down by our hero.
Are they not?
> I don't think the comment you were replying to was a model of clarity; the quote is a paraphrase of a paraphrase of an argument in James' The Will To Believe. I'm familiar with that essay (not one of his most impressive) so maybe it was easier for me to hear what they were trying to say.
Well, I don't know that essay, so it's possible that I am completely missing the point, in which case, feel free to explain. But then again, I have heard arguments of that form more than once, and so far whenever I looked any closer, it boiled down to the same flaw in the reasoning.
> But "WTF?!" etc isn't helpful.
Helpful for what?
> Objecting to every sentence like that doesn't even begin to address what they were trying to say.
Well, OK, what were they trying to say, then?
> Anyway, all of this seems obvious, I think you know this, I'll stop here.
You think that is helpful?
> Your comment read like you didn't care about what they were trying to say, just to knock them out of the water with any available weapon, which you seem to think you had done.
So, my goal was to knock some person I don't know out of the water regardless what their argument was? I'm not sure that makes sense ...
History is invaluable and the lessons learned can provide some degree of predictive power. Yet it would be unreasonable to try to argue that something ought be enacted as, as a matter of fact (or at least heavily supported hypothesis), because of results derived from the study of history.
The social sciences in general seem to be want to be allowed to behave as a "historical and descriptive" field, yet be received with the same credulity as, for instance, experimental physics. Ultimately, I think the field needs to choose to go one way or the other.
See: Stereotype Threat, Implicit Association Test and Implicit Bias, Power Posing, et al.
Maybe my point is off the mark given your "qualitative" qualifier, but about the social sciences: behavior is a phenomenon. It's complex and can be examined at multiple levels, but it is a phenomenon. We have two choices in understanding it: to try to approach it through quantitative, methodical, logical, observation and prediction or to leave it to other approaches. If you do the former, it's extremely difficult and you end up struggling even to quantify anything. But if you do the latter, you are leaving something extremely important to wild speculation, unmoored opinion, and fairy tales. The quantitative social scientist faces a dilemma of being ridiculed for trying to approach something scientifically and rigorously when the alternative is to leave it to forces that are much worse. It's simply false that there are no provable metrics, unless you literally mean in the sense of proof, but that stands to be true of many scientific disciplines. And the reductionistic reply that everything is neuroscience is akin to telling computer scientists to abandon algorithmic theory because it's not about electrons, or the astronomer that it's all chemistry and physics anyway, or the evolutionary biologist that it's all molecular biology.
In any event, the replication crisis is not about statistics, and it's not unique to the social sciences. Studies have shown the same problems exist in many fields, although they do appear to be worse as the field increases in complexity of subject matter. Oncology is also plagued with such problems, for example, although no one would say that cancer is a soft topic. The social sciences have a tendency to look inward (appropriately enough), and so in many ways, are bearing the brunt of the negative publicity they are shining on themselves and everyone else.
The problem I see extremely complex, and won't be fixed by any statistical solution. There are many causes, too many to name (I could probably write a book on the topic and it still wouldn't be enough). In general, they can be summed up by saying that there's been a fundamental shift away from soundness, and toward celebrity and novelty, at a time when the justification for that celebrity is weakest, if it were ever strong. Tenure protections have eroded, accelerating the problems, and funding has been cut substantially from universities, even as there's an oversupply of new scientists entering the field. It's as if we decided as a society to honor war veterans by celebrating only those few survivors with the wildest, most glamorous tales of adventure, ignoring the dead who made the actual contributions. To make this analogy more accurate, imagine that we've grossly romanticized war publicly, lavished attention on these survivors, even while we've dried up job opportunities for our best and brightest young citizens in society at large; we've all but cut public funding from the military in practice, and replaced it with a thinly veiled system of contracted private mercenaries, who are told their obligation to the country is to raise funds in whatever way possible.
I'm exaggerating and being dramatic to make a point, but I think the parallels are pretty close.
I'd go so far as to say the problems aren't really unique to science, as is reflected in comments here about software projects and the like. We have serious problems in society (at least in the US) with overhype, and gross distortions in how we compensate for work and contributions to society in general. This is true in business, politics, healthcare, everything. I think science is just one more casualty of this, but it's been hidden for awhile due to the gradual nature of the disease, and the cracks are starting to show.
Perhaps you should write a book!
I could probably write a book on the topic and it still wouldn't be enough
Then perhaps you should write two books.
I think this is a story that needs to be told its full complexity, and you seem to be in a good position to do so.
It's as if we decided as a society to honor war veterans by celebrating only those few survivors with the wildest, most glamorous tales of adventure
An excellent analogy.
I think science is just one more casualty of this, but it's been hidden for awhile due to the gradual nature of the disease, and the cracks are starting to show.
This is one of the parts that confuses me: what sets the pace for the "race to the bottom"? If the incentive structures haven't changed, why did it take so long to reach where we are, and why are some fields still relatively immune?
What you're proposing is mass unemployment.
In any case, if we can't expect and even demand high-quality research from professional scientists, I fear we are doomed as a species.
I'd also said that many may be _capable_ of high-quality work, but merely have limited incentives to produce it currently.
The claim? That the biggest challenge facing remote poverty- and alcoholism-stricken communities of indigenous peoples were not enough tablets to be able to "Netflix and chill". Because surely that will solve all the world's problems.
I could have just said "What the linked article says is correct", but I figured this anecdote would add more value. But yeah, whether you want to believe it or not, the linked article is pretty accurate. Successful academics either get lucky or they are frauds.