The incentives of academia naturally select for less reliable results (2016)
theatlantic.com
theatlantic.com
No it isn't. It is a simple one but it isn't obvious. Low statistical power is just one of many indicators of research quality. Qualitative research with sample sizes in the single digits can be done well and research with large statistical power can be done poorly. The measure of quality in research is much more conceptual and cannot be reduced to rule based measures without failing completely.
But it is currently a useful indicator. This is a symptom of a problem by itself.
The solution is to go back to the old way of science (pre-1940):
1) Collect data and figure out what is repeatable and consistent (eg come up with "natural laws").
2) Come up with theories that explain the observations from #1 that also make precise predictions about some different type of data.
Reward researchers who figure out methods that lead to reliable, repeatable observations and those that make precise predictions (if your theory predicts nothing more than a positive/negative correlation between two variables it is too vague).
Indeed, everything in the article or in discussions of this that I see is always treated symptoms far from the causes. Let's do a quick five whys:
Why are publications not reproducible? Because positive results are being published as fast as possible with little checking on them.
Why are they publishing so? Because the people involved need publications to get out of their current transient position and secure another transient position, with hopes of someday being able to settle down somewhere.
Why do they need to secure their next transient position? Because they have been inculcated that this is the career to be in by their milieu and that is the only way to stay in it.
Why is that the only way to stay in it? Because there are no permanent positions that aren't at the very end of this gauntlet.
Why are there no permanent positions except at the end of the gauntlet? Because this is a pyramid scheme built on an apprenticeship system.
Don't get distracted by all the other stuff. It's the fundamental structure of academic science that leads to the rest of this.
There's a lot of "BS" in these power structures that keep people from being able to dedicate resources and a proper amount of time to generate quality research.
I'm fairly "green" in my research career, but I've been in the game for a few years now in the Biology/Bioinformatics realm. I've seen and heard stories from plenty of colleagues about how when they first got started and well into their first P.I. positions (and even until this day when they're 10-20 years into their work) where they get abused by their institutions. Aside from "publish or perish" there's also your institution skimming 50-60% of your hard won grant funding to let you "rent" their facilities where you're fighting over shelf space with other P.I.s Absolutely insane.
Seeing this I decided I can bootstrap my own lab for much less and have true autonomy. For example a rough breakdown of my costs for setting up a bio/bioinformatics lab for myself and a few friends:
Land Zoned for Light Industrial and Scientific Research? about $1000/acre or less in my area. (hint: buy tax liened property)
CO2 incubators? Don't pay $4k-8k new. I can buy refurbed or liquidated lab equipment from the unsustainable institution labs for literally pennies on the dollar. Try $100-200 for this equipment.
High RPM centrifuges? Once again instead of $2k-5k I can spend $300 for a 15k RPM centrifuge. and something like $20 for desktop centrifuges.
PCR machines? Try < $100 each. And I've even gotten "extras" with them in that price range.
Compute power/I.T.? I can get 256 core, 1tb ram blade centers for about $1k. Modern Core i5/Core i7 desktops for $80 each.
By my estimates I can buy the land, build a building(admittedly a very utilitarian one), and equip a new lab for sub $30k. Now my only costs are re-agents(if I'm doing research that requires it), power, and cheap land taxes.
It goes without saying that the idea is just like any other business venture, with ways to succeed and to fail.
But a useful thing is to try and get a survey of what kinds of businesses are out there, what they are doing, and how well they are doing. Next time you're at a conference that has a trade show, go and look at the booths in the "low rent district," because it will give you an idea of what people are doing.
Oddly enough I work for a F500, and we buy second hand gear. One reason is that if somebody is advertising a particular instrument, it means that they actually have it, and you can get it right away. With new gear, sometimes it can take weeks or even months. Also, if it's under a couple grand, I can buy it on my purchasing card without management approval.
I'll definitely take a look at what some other small businesses are doing. I've seen some up close, but the trade show run through is a great idea.
And yea. I buy stuff at auction within 300 miles of my area and drive the same day to pick up any gear I buy. What kind of department do you work within inside your F500?
I'd prefer not to share the info on a public forum to keep mass buyers out of the market(more than there already are)
Shoot me a message at ethan at dancecardrx dot-com and let me know what you're looking for and I can point you in the right direction.
There's a lot of consulting work available in traditional labs because of bureaucracy around hiring new employees full-time. So I can apply my skillsets in those labs as an outside consultant.
Ideally though this would be a pure-research endeavor. Which requires time and building up credentials and a funding base. So basically eventually income for the entire operation would be a mixture public/private grants and leasing out technology developed within the organization to others.
The B.S. is amplified however when you're at a big institution. There's a lot more red-tape even if you have private financiers.
One way would be to yes, keep burn low. The idea there right would be that you're asking for less money, you're lowering expectations of those giving you money. If someone gives you $1000 they're going to micromanage you a lot less on how that money is used versus if they give you $10,000, $100,000, etc. The other idea is that if you have less stakeholders there's also less back and forth that needs to be maintained with them.
My idea would be that you'd avoid the incentive-skewing over the long term by doing some research that can be converted into revenue generators to cover the base costs of running a lab. Reinvesting profits into traditional finance products to get a yield on the lab's money. Even running a small scale cryptocurrency mining operation as a single part of revenue generation. If you combine this with low-overhead by being smart about what would typically be big ticket purchases that only the most well funded of labs can afford and finding people with a like mind who are willing to take a liveable(but smaller wage) to work in a more open environment then it should be sustainable. For me in particular I can balance research time with programming consulting time and make 80% of what I would make in Biology/BINF or at an academic institution without breaking a sweat.
Like I mentioned before about grant skimming by institutions.. If you're a PI at a state lab or university they will take a "generous" portion of your grant as soon as it arrives. I'm specifically thinking of one guy I know who recently had HALF of his grant taken for facility rental and upkeep. It was a large grant mind you. In another case a friend who had finished his postdoc and was a first time PI landed his first grant. As soon as it was in the lab managers tried to say "awesome, now we can pay your salary out of the grant" He had to play hardball with HR to state that his salary was guaranteed and not to be paid out of grant money intended for his research. Now I can understand why this happens. It's not free to run a research enterprise, and it's not particularly cheap when you're buying equipment at MSRP. Research equipment in the second hand market before it goes to professional refurbishers is literally cents on the dollar. It's so specialized, there's a small market for it, and big players are not buying this stuff because of the inherent risk! In my experience though the defect rate is so absurdly low that it's not an actual risk buying this equipment. But it's not something an institution with a multi-million dollar grant to setup a research center/core is going to waste their time on.
As for specifics of what I'm aiming to do. I intentionally picked a research area I'm going to delve into that has low costs. An operation like I'm talking about in my OP would not be feasible for high energy physics or high cost sequencing projects. I'm focusing on photobioreactor design and algae cultivation yields for animal feed, biodiesel refining, etc. For that I need CO2 incubators, centrifuges, a 3d printer, algae samples, a sample freezer, turbidimeters, photospectrometers, pH meters, and then a revolving supply of plant fertilizers, CO2, and Nitrogen. But before I get deep into actual PBR design and cultivation techniques I'm going to work on computational models for PBR design, algae growth rates, and PBR monitoring software for data collection.
You do have to be careful about the fixed costs for them administering the grant however. The time and effort of reviewing and approving / rejecting a $1000 request is not going to be a fraction of a percent of a much larger request. Lower perhaps but proportionally higher. And by promising much less, you might not really make it more appealing.
If you get a $250,000 a year NIH R01 (the largest you can get without PITA budget details) and your institution has negotiated a 50% rate, the NIH pays your institution $375,000 a year. Your $250K doesn't get reduced.
The only time I've ever had something like what you described happen was when a private funder had X total amount they wanted to spend, but hadn't bothered pre-specifying that they'd only pay a particular overhead rate. Which is mainly just a company wanting to get something for nothing, and doesn't leave me particularly sympathetic.
It's not only private funders who do this. It's the biggest funding agencies in the United States. NSF, DARPA, Army, etc. In fact, I feel most grants operate this way. I'm not sure about NIH since I've only been Co-I on those.
You are correct that there are grants from agencies with a cap including indirect costs - I confess I have the opposite experience of you, I'm usually the Co-I on NSF-style grants, so I don't look at them much.
In that case, yes, if indirects were lower you could up your direct costs. I still contest the idea that the indirect cost system is somehow stealing "your" grant money though (where you = lab).
Maybe here is another perspective. NSF "Smalls" in my program are 500K, and Army "YIPs" are 360K. If I get a Small or a YIP, and my friend at another university with half the overhead gets the same Small or YIP, my friend will get more spending money than me.
I get 300K on the Small, and my friend gets 380K. To do the essentially same unit quantity of research. A university with a higher overhead takes 80K more from the grant that would have gone to the lab.
Is certainly a creative way to describe indirect costs.
For standard NIH/NSF/most Federal grants, they don't come out of "your" grant funding - they're a pre-negotiated rate, and the award amounts are based on indirect costs. If an institution cut their indirect rate to 25%, you wouldn't see that money back, it would just never be there.
I've also never been at an institution that's not willing to at least work with a grant giving agency/firm/etc. on lowering the overhead if it's a sticking point.
But I will say in my experience that I get a ton of value for my overhead rate - and that it's known in the institution to actually be slightly lower in most cases than the actual cost of doing research.
I don't think I can argue that the overhead at traditional research institutions is super inflated. It makes sense when you take into account the level of bureaucracy and growing amount of "extra" administration :). But I think it's disingenuous to say that if the costs were cut back to 25% that you'd NEVER see that money. If indirect costs were able to be lowered at these institutions there would be more money in the funding pipeline for direct costs to begin with right?
It's fair that you think you get a ton of value for your overhead rate. Different setups are better for different people. My ideal situation is definitely not ideal for most people. I just don't think I can stomach supporting what I see as administrative waste and needless redtape(not that I believe there should be NO redtape, no one sane would believe that?).
Also in your other comment about private donors not wanting to cover any indirect costs.. completely agree.
One thing to check on, though, is the requirements for waste disposal from your lab in your area. You may find yourself having to pay for special disposal.
>"Because positive results are being published as fast as possible with little checking on them."
Why do you think this is so? It is because they think the stats are telling them the probability their theory is true, etc. A significant p-value means the result is "real", etc.
Innovation is overrated; maybe we should just pay scientists more to do inglorious but immensely valuable work. I was certainly repulsed from lab sciences because I had no desire to do the work that's in demand—pulling results out of your ass to get more funding.
Something I'd be very interested in is paying independent labs to replicate studies. This removes the incentive issues surrounding the journals and researchers, leaving the funding issue.
If these were the case you'd have the opposite problem. People would be hugely incentivised to reproduce highly cited papers again, when what you want is to reproduce papers with low levels of confirmation.
You can't rely on anything in the current half-assed, no replication environment. In many areas it is literally not worth coming up with an mathematical model to explain the data because it is all so questionable.
You're saying it's better overall if more replications are published. I'm talking about the problems that are stopping us from getting to that point. The current incentives do not align well with the desire to have more replications done, and simple changes could easily backfire.
> (not vague crap like "this drug makes that disease happen less")
I simply do not agree that if you think this and find some results that point in that direction you should not publish. I see absolutely no reason to save electrons to improve some average of papers, I'd much rather work isn't re-done repeatedly. Perhaps publishing something vague with some backing (e.g. we think X does Y and the data is at least consistent with this, and we can't think of what else makes sense) gives enough for someone else to build on and do a more rigorous investigation.
Explaining the observations is a separate step from coming up with a reliable way to generate a pattern in the data.
The Nobel Prize goes to the people who discovered something. Not the people who confirmed that they were right.
Having someone else run the experiments is also required to get past something we have to deal with a lot in the tech world. The "it works on my machine" result.
You may have run the experiment twice, but can anyone else? Does your setup depend on something not specified in your paper? In the tech world, how many times has the setup documentation you wrote turned out to be missing some detail when a new starter actually tries it?
No setup will ever be perfect, but we can look at reducing common points of error.
Can nobody publish these results? Is it not a sensible building block on the way to precise predictions?
correlation is not too vague, but implying causality from correlation without further tests is
conversely...
assuming that a paper that only reports correlation or reports correlation as well as further evidence of causality (i.e., prior work on causal mechanisms, qualitative evidence, etc.) is faulty simply because it has correlation is equally problematic.
The major problem of the article is that it displays the same flaws as many of the underlying problems in science itself: It makes massive generalizations without nuance or appreciation of context. The problems are paradigmatic (i.e., Kuhnian) and the fact that they appear in the research AND in the critiques of the research are far more problematic than a p of .05.
But either way, why not publish? Why not allow others to build from your work and reasoning up to now?
Edit:
If all that gets published is a correlation coefficient and not the distribution, scatterplots, etc. Then no, that paper is worthless.