So let me ask the research community on HN something I see discussed less often. What alternative incentives exist today, or could be created, that would push more scientists to try higher risk / higher reward research?
This causes two problems:
1) It's easier to polish something which is incremental.
2) It's easier to achieve the look of polish by purposefully hiding issues and smoothing over edges.
I'd personally like every paper about a technique or algorithm to include as many problems where it does terribly as where it does well -- this is annoyingly uncommon. Also, people shouldn't be afraid to discuss all the nasty unfinished bits but in practice if you mention them reviews say "well, go fix that before acceptance".
When I left academia one of my most significant thoughts was, "at least I won't have to argue over the 21st draft of a 3 page paper ever again".
With less time spent per referee report, easily appreciated evidence of polish matters more where papers that pay attention to careful issues of methodology, tackle deep problems, etc., require time on the part of the referee that they are unwilling to spend.
https://www.theatlantic.com/science/archive/2019/07/we-need-...
And an EconTalk episode on the same subject (which goes into more detail than the article)
https://www.econtalk.org/patrick-collison-on-innovation-and-...
Relatedly, in this lecture [0] he expresses it by contrasting the "positivist" model where knowledge is piled on, linearly expanding vs the model of discourse, where research is a conversation, where participants must know what the others take for granted, assume, doubt etc and new contributions argue that there are better ways to do or conceptualize things. Works are forgotten, left behind, ignored etc. A work can be valuable in one context at one stage in the discourse, while worthless at another time. It's not like a neutral map of some terrain that we can just file and store forever.
He's in the social sciences, where this may be more obvious, but it's also true of sufficiently developed technical sciences too. When the low hanging fruit has been picked, the game becomes closer to zero sum. I mean new value can still be created, but not mainly through bringing in "new characters into the story" but by letting the story unfold using the same characters (main ideas). Only a true paradigm shift may break this equilibrium.
The way I see it, the easy problems that can be solved by one or two individuals are mostly gone. You need large diverse teams to tackle interesting problems.
To translate this into policy, you need opportunities for many more first authors, and you need to try to incentivize completely different departments to work together.
This ‘corporate’ view of science is just not true, kind of like adding “man-hours” to software projects doesn’t necessarily account for better software, speeds up or enables those projects.
Large diverse teams just produce more mediocre research, if recent experience with massive “deep learning” industry is of any indication, or fail completely like that “blue brain” project with multi billion funding and thousands of participants.
Bigger is not always better and it has never been necessary for good science.
On the other hand, there are pretty hard limits to the sorts of projects that our standard model of a single full-time person[0] can do, even with some part-time collaborators (i.e., middle authors). This is especially true for projects that involve a combination of very different approaches: bioinformatics, HTS, in vivo validation, etc.
I've noticed more and more papers with multiple (sometimes 5!) co-first authors, but that's a kludge around the fact that we can't figure out how to allocate credit to teams.
[0] And, almost invariably, a trainee
A good example is the Minneola, a wonderful citrus fruit. The plant patent is owned by three parties: the University of Minnesota and the two inventors.
Pfft. I've been through this, my ex-supervisor attempted to file a patent which included a bunch of figures and illustrations I drew single-handedly while writing up my PhD and which were lifted from my thesis without my consent.
My PhD was funded by a government research council.
My ex-supervisor was filing a patent on behalf of a private company he set up to run in parallel out of his university laboratory.
There are some people who just don't care about rules and figure they can just ask for forgiveness later. I'm still cross about this, over 20 years later.
On this note, and admittedly a bit of a (relevant) plug: I work at a startup called scite that's trying to improve this -- https://scite.ai
Citations are the primary mechanism by which scientific papers "talk" to one another, and one of the systemic issues we see in the status quo is a sort of numerical reductionism where a lot of emphasis is placed on "how many times someone is cited" without any indication of whether those citation counts are from papers that support or dispute someone's findings.
One of the things we do at scite is let you see, for a paper, how it has been cited (i.e. not just a count, but the surrounding textual context around the citation, and a classification from our model as to whether the citing work provides supporting or disputing evidence for the cited paper, or just mentions it).
That information is also aggregated to the author level, or journal, and so on.
The hope is that by providing (and improving) this service so that researchers can see how papers are cited, we're able to promote more reliable science (in addition to letting someone explore new subject areas / doing lit reviews much faster, and other things).
I just wanted to share that because a lot of the other comments were more abstract and it might help to include something more tangible that's being actively worked on now. There's a lot of interesting work in general in this space.
If you have any thoughts / feedback / questions, feel free to reply here (I check occassionally) or write to me at the email in my profile.
These are two parallel worlds being discussed as if they were one.
To answer your question, I believe we need to look outside of academia to relieve the pressures on most people so that fewer are tempted to add to the science glut. I'd venture to mention UBI but that's a whole can of worms
One is to accept only mini grant proposals, say 10 pages at most, and screen these to meet some minimum threshold, say, aiming for the top 20th percentile. Then a random subset of these get funded. This helps diversify the funding somewhat, and hopefully catch more "risky, but interesting" projects.
Another approach has been universal funding. The most basic is "everyone gets X research dollars", and any additional funding would require submitting grants on top of that. More complex schemes propose "everyone gets 2X dollars, but they must donate X to other researchers", which allows some people to accumulate the funds necessary to conduct big, expensive work. But all of these proposals aim to minimize grant writing and review, and hopefully to diversify the kinds of work being done.
Longer windows of evaluation could also be useful. Big interesting ideas take time to develop and support, but the competitive academic environment demands quick evaluations, which make ot difficult to sustain long projects. Evaluating a research project after 5 years, instead of 1 or 2, could help alleviate some of the pressure on researchers so they can pursue longer agendas.
Every researcher can come up with their pet theory and solution. Ultimately, I think that all of these approaches, and many others, have merit. But we need to experiment with different policies to see what works.
Federal grant funding needs to be increased.
Indirect funds need to be eliminated.
The funding structures need to be reorganized. Some ideas I think have merit are basing grant award on a lottery system, or funding individual researchers based on previous research for a limited nonrenewable period of time.
Grant review panels need to be mixed up dramatically. Currently they consist largely of people who have received grants in the past. One of the biggest predictors of proposal rating is being a co-author of people on the grant panel. I think grant panels need to be based on some lottery system too, like everyone with an ORCID ID gets thrown into a pool somehow, like jury duty.
Universities need to be funded in a way that's separated from grant receipts.
1. Publications in prestigious journals have become a measure, not a way to communicate. This leads to perverse behavior.
Any kind of project that would involve deep work is very scary, unless it has early low-hanging fruit called easy publishable papers as intermediate steps that are easily published. I personally felt I was encouraged to find such fruits without too much coherence in what I was doing. Preferentially one does not have to learn or study anything challenging, because that makes the fruit potentially much more difficult to pick.
My proposed solution: do not judge researchers by length of their publication list to evaluate them to hire/fund. Judge them by a selection of their recent work instead and maybe their presentation what they are doing now.
2. This is connected to another problem, corrective feedback comes quite late into play, when the most work has been done and it is very costly to change anything. I feel better results could be had of reviewers entered the stage when study was being drafted.
Currently, submit a manuscript into a journal, get it rejected after a round of substantive, critical but correct reviewer comments? Optimal thing is not to review the work as suggested in the light of new ideas, but try to find a less prestigious journal likely to accept it with minimal work.
3. This has effects on conferences. While their primary purpose has always been presenting ones research and hearing about others, they kind of turn too much into a platform for advertising your research so that you get cited. Thus the general feeling can be quite one-way, and there is less knowledge enhancing communication by scientific discussion. Some conferences are better than others (amount genuine curiosity and interest helps), but sometimes I felt people would come to deliver their own little advert, maybe as chore between meeting their personal friends, instead of coming to talk, both receive and offer meaningful updates.
4. PI-centered networks and death of university department. I believe originally university departments were formed because it was thought beneficial and reasonable to have scholars with similar interests nearby. Social networks are probably quite important part of science, and is easier with people who are located often in the same building. Research group culture is a fascinating way to have groups of people nearby who don't communicate. It appears that to PI, each unsanctioned contact from their underlings to outside the group is a distraction at best (if it leads to underling working on publishing something PI is not involved in) or an unwelcome threat at worst, because it risks changing their plans, whether they concern the projects underlings should working on, the author order on planned manuscripts, or something else. All presentations to group outsiders is about signalling importance, prestige and coolness, but no research communication within department to help with unfinished work.
Solution would involve stopping funding faux institutions within institutions, and start paying researchers in a way that encourages them actually collaborate within their physical departments when it is useful. (Salary paid by university/dept., no project grants.)
5. I suppose science moving very large projects involving lots of contributors can not be helped, but it is incompatible with publication authorship as the currency of value in academia. It leads to authorship beimg traded as a currency within projects. One is "paid" by official authorship in some papers; sometimes a person can show their magnanimity by bestowing it upon some people who participated. Sometimes technical or statistical help does not get acknowledged.
IDK what to do about that. Moving out of evaluating publication lists to evaluating personal contribution/capability might help. In addition to big publications that are meant to communicate the big findings publicly but leave outsiders to interpret author order like tea leaves, record publicly some documentation what exactly each individual did, in their own writing, and that documentation is what the contributors should refer to in their CVs?
1. National research funding agencies and universities need to stop doing their assessments based on publication counting. Of course, publications need to be taken into account somehow, but at the very least these indicators should be weighed by curated rankings of journals.
2. Existing publications can be used as an indicator for how much output is to be expected in the future, but the goals for research output should be kept reasonably low. You cannot control the output of researchers, you can only control which ones you hire and (to some extent) the type of research they conduct.
3. Hiring in academia needs to be based more often on screening candidates by committees of outside experts in the field instead of local staff and people who don't know the area well. Projects also always need to be evaluated by experts on project's topic. (As strange as this may sound, this is often not the case!) It might even help to prune away irrelevant indicators, e.g. ask candidates to submit only 3-5 of their best publications and completely ignore the rest. And by "completely" I really mean completely. The reason is that if you have two researchers and one of them has 20 publications with five really good ones, and the other has 40 mediocre publications and not a single good one, then it is very hard for a normal assessment committee to justify taking the first one, but it's always easy to converge on the second one, even if that is exactly the wrong choice.
4. Assessment committees need to be told to evaluate the quality and originality of the research only. If you really want or need a certain output quantity, then make it part of the formal hiring criteria, not part of the scientific evaluation.
5. Increase funding for risky projects and risky individual grants, elimination of criteria that exclude unusual CVs (e.g. allow long time after PhD, people from other areas, people with time spent in business, unusual research suggestions). Originality should be one of the highest ranking criteria. There should also be a focus on versatility and hard skills. Even in the humanities, never hire anyone who says anything disparaging about mathematical methods and statistics, and never leverage people who don't know the tools of their trade into positions of power.
Basically, you cannot push scientists to anything. Once someone is hired at the postdoc level, you cannot steer much, micromanagement and constant evaluation are highly counter-productive. You need to hire scientists that do interesting research. Treat them like an investment in a startup: Most of them will fail but some of them will succeed. Give them a second chance, maybe even a third one, but not indefinitely many. The hiring policies and processes at universities are often bad. Candidates are not evaluated by experts, there is plenty of favoritism, boring high producers are favored over interesting researchers who want to built something up, because the local staff doesn't like scientists who "shake things up", and so on. There is a lot of inertia to overcome in Academia, being a good scientist can sometimes even be harmful to your career. Funding agencies need to steer against this.
Edit: Much of what I mention can also be achieved by getting an absolute top researcher in an area, give him or her an institute or research unit and ton of money, and let them do their thing. They know what to do and whom to hire. But it's expensive.
1. Funding agencies and universities don't make decisions. Only individuals in those institutions do. And I don't know any individuals who admit to doing assessments based on publication counts. They'd be openly mocked for such an archaic way of thinking. The only people who seem to openly obsess over publication counts are Ph.D. students.
2. Agreed, but output can be measured in different ways besides traditional academic publications.
3. It seems like this is already done, through recommendation letters when people apply, and in external letters during the tenure process. I see the opposite outcomes, whenever I've seen someone with fewer but better publications compared against someone with more but weaker publications, the first is always preferred. Because it's easy for someone to make the case "Person 1 has fewer paper but they're more impactful and they choose riskier problems, an excellent trait" versus someone to make the case for "Person 2 has more papers in quantity so are more productive." Unless of course it's not clear that person 1's papers are actually higher quality, in which case the original premise doesn't hold.
4. This is already the primary factor from what I've seen. That's the whole point of job talks, publications listed on CVs, etc. I've never seen a tenure-track hire without people reading some of the papers listed on the CV, and discussing the presented work in the job talk.
5. Half of the funding agencies now say they're looking for risky ideas, the word "transformative" has been a common part of NSF vocabulary for decades. Originality is one of the NIH core criteria factors, as is criteria about the Investigator themselves. How panelists interpret the criteria is more social and political than a technical thing.
Anyways, maybe we probably just live in different academic worlds, but I don't see as much blatant expression of the biases you are describing, and feel like the major goals of your ideas have already been implemented which has led us to the current situation. Maybe not everyone buys into them, but as you say there isn't a way to get a bunch of individuals to follow orders, so the only change that's possible is changes to process which has already happened. I don't agree that we need to add additional bureaucracy like rating applicants on "did they disparage mathematics", which I find extremely problematic.
Even the last idea in your edit is already done in every way possible. The MacArthur award, Turing Award, Nobel Prize, various prizes, etc. all give an absolutely top researcher money and prestige, so they can do their thing.
I think the best way forward are social solutions, rather than procedural. Convince minds, rather than add bureaucracy.
I have to insist about what I wrote about candidates who disparage mathematical methods in the humanities. These people can destroy whole departments if you let them roam freely.