How to fix?
There needs to be less emphasis on quantitative metrics like citation counts, h-indices and other trivially gameable nonsense, and more emphasis on human judgment along with a set of basic criteria to provide a floor to research quality across the board (things that every good scientist should do, as opposed to things that you should relentlessly maximize). This would require the non-scientific managerial class to abrogate some of the power they now wield in the academia and relevant funding institutions, therefore is very unlikely to happen.
And - probably, most annoyingly to the HN crowd - there is nothing meaningful that technological or business disruption can accomplish here. These are systemic social problems that have been continuously and systemically building up since (approximately) WW2. At this point the only entities large enough to make a difference are probably the NIH and NSF in the U.S. and the relevant EU funding agencies in Europe. In my judgment, these are precisely the agencies least likely to institute such change, so here we are.
So I would say that fundamentally the problem is with study sections and how they think. They are incredibly conservative, but in all the wrong ways. They highly value institution of origin, preliminary data from pilots, no matter how shitty, and the sort of hypothesis that seems reasonable from prior literature. There are also a lot of political games on study sections.
This means that if you repeat a lie often enough in the literature, and you know people, then voila, you now have support for the same hypothesis in future grants.
Peer review is not working for funding. I would say the NIH (in my case) needs to hire completely independent full-time reviewers, and needs to place a real, high emphasis on reproducibility. It is in the NIH's interest to do so, because as I said, its funding will be cut if this continues.
There is a bit of a difference between working directly for a university and working for a research nonprofit, like Dana Farber or MD Anderson or whatever.
If you work for a university, and you don't pull in a lot of money, what will happen is that you will be expected to teach more hours to "make up for it". You are less likely to be fired. At a nonprofit, if you don't cover your own salary and expenses through grants, you will quickly become unemployed.
I m really not sure about that as the publishing bubble coincides with the rise in funding by public bodies [1]. Funding safe projects is a way to disperse funds to as many people as possible, and that's not necessarily good. At least in EU it feels that way. The politics of funding seem to naturally have gravitated to the current situation. I doubt the model of funding can change with the same stakeholders.
[1] https://phoenixrising.me/wp-content/uploads/NIH_Total-Fundin... , https://www.researchgate.net/publication/327917254/figure/fi...
My radical proposal is to separate the research institutions from the universities -- especially the public universities. If you want public research institutes, that's fine -- but they should be their own entities.
The university system, with its mix of research and instruction, and its tenure-vs-non-tenure system creates terrible incentives.
Top researchers are rarely also the best instructors -- and so it makes little sense for the two occupations to be so intimately tied.
The university funding model is terribly destructive to top science labs as well -- the university sees them as a source of prestige, sure, but more importantly as a source of that all-too-critical grant money (entirely too much of which is bogarted by the rest of the university) -- putting many of the labs in this place where they have to keep securing grants at an ever-increasing pace no matter what, to keep the rest of the university flush with cash to spend elsewhere.
There's really no good reason why research needs to be done at a university, or why a research institution needs a body of lecturers and students.
My wife did a PhD at an elite private university, undergrad tuition north of $50k/year not including room and board and other expenses. At one point three senior professors in her department went to the department chair and the dean with a proposal to revamp a notoriously unpopular intro class in their department. Rather than having it taught by adjuncts or postdocs, the three of them would team teach it. They would change the syllabus to bring it up to date with the current methods in the field--it had hardly been touched in two decades.
They were told to forget about it. The University said it was a waste of time for them to teach undergrads.
Presumably the kids applying to these types of universities are not aware of how little concern the institution has for the quality of their education (the perceived quality, on the other hand, is extremely important--but only loosely connected to the actual quality).
It turns out that prestige has very little to do with undergraduate education quality.
This used to make lots of sense -- for centuries, college was more important as a place to meet other people as it was a place to actually learn useful skills -- and so you were primarily trying to go to the highest-class place you could get yourself in to.
In this day and age, though, I don't think that's the right way to approach choosing a school.
The gap between elites and non-elites is growing, and as a result, the value of breaking into the social circles of the wealthy and "legacy" students who dominate elite schools is also growing.
Meanwhile, high-quality educational content is becoming more widely available - often published for free by those same elite schools - so the value of the education itself is approaching zero. (For some programs, notably elite MBA programs, the value of the education is basically zero already and has been for some time.)
If a course is taught by a TA or adjunct who barely speaks the language in which it's conducted, rather than by the distinguished professors whose names are on the department website, that's an annoyance, but all it costs you is some time on Khan Academy or Coursera to learn the material on your own. Your well-connected classmate who introduces you to a hiring manager or angel investor will have a much larger impact on your life.
You could of course lower compensation for academics to incentivize other professionals to support research but you'll find academics fleeing to other opportunities in many cases.
Accreditation could be performed for research institutions just as it is for university departments today -- PhD candidates could work in research institutions just as well as they do in university departments. The few classes they take are all within the department that they're researching in anyway, and are also all separate from classes that undergrads take -- so moving that education out of the university and in to the research institute seems fine too.
That's not to say that instituting my plan would be easy. I did call it 'radical', after all
There are millions of dollars on the line for researchers involved. It is the difference between a well paid career and a life of destitution while being a slave to huge student debt.
Professorship and research grants should be given out based on criteria that are only incidental to research results.
Evaluate profs and grants based on:
1. Domain knowledge (test the applicants) 2. Math skills (test the applicants, makes sure they know about how to avoid p-hacking, preregistration etc.) 3. Motivation and leadership. 4. Prior and current research _proposals_ (but without looking at the results or whether they have been published). 5. Other skills such as communication, interpersonal skills, outstanding achievements etc.
Universities should not rely on journals to evaluate their professors. This corrupts the whole system. Journals have different goals. They want to publish well done research with interesting results. Universities should hire researchers that do good quality research with interesting _questions_ regardless of the results.
If universities keep giving out jobs based on having generated interesting publishable results, they are going to keep getting researchers that ignore biases and and ignore bad science practices in order to generate publishable (but unreliable and often false) results.
One page (https://www.kn-x.com/static/PWFeb17forum.pdf), two page (https://ssrn.com/abstract=2835131), and three page (https://ssrn.com/abstract=2713998) versions of this are available. Further details are provided at http://kn-x.com.
I doubt anyone on this thread is going to like this solution ... but I think most on this thread agree that the underlying problem is the incentives, that fixing the problem requires changing the incentives, and that changing the incentives within the existing system is very hard.
(i) Reproducibility is not a criterion for tenure evaluation, therefore it bears little relevance for career advancement for young scientists. This is an overworked and underpaid group in academia. (In the bay area, they earn 1/3--1/2 the salary of an entry level software engineer but probably work 50% more hours.) They simply don't have the luxury to take time off to reproduce someone else's work.
(ii) No major funding agency would be willing to support the kind of work that reproduces published studies. When something is published, it is considered "done" and not worth spending more money on it.
There needs to be a sea change. But sadly, the academia is at best paying lip services to the structural problems in the "reproducibility crisis". I'd expect it to continue like this for most fields.
It's the good scientist who produces economic value but cannot capture it that we have to support but maybe that person cannot easily be found. The state just has to overfund and accept that some large percent goes to nonsense.
In my mind, that's okay, though, since the guys who can capture economic value can do a lot with what we have.