Negative incentives in academic research
lemire.me
lemire.me
On the subject of incentives in academia - reflecting on my academia run, I've noticed that I was the most creatively productive when I had a longer planning horizon in front of me. It was either at the beginning of a several-years position. Or at the very end when I knew that I'm leaving and didn't care anymore. The least productive was the sequence of one-year postdocs - when I was constantly worrying about the next one.
Do you believe the average academic research paper or project written/done today is of the same quality or better as the average in 1980, or 1950?
On the flip side, those early papers contain the most fundamental discoveries in our field: you're only going to invent RSA or blind signatures or zero knowledge once. It's possible all those researchers were much smarter than we are now. (I grant this!) But there are a lot of absolutely brilliant people I know today. Alternatively, the lower-hanging fruit is all gone and the problems have become much harder.
Reminds me of a patent officer who said in 1899 that "everything that can be invented has been invented."
Well, maybe. Sometimes things that are discovered are lost or go completely unnoticed, so progress is not always monotonic. For example, for a long time it was believed that John William Strutt first documented dynamic soaring [1] in birds in the 1800s, but just recently (2018) it was found that actually none other than Leonardo DaVinci documented the phenomena in his notebooks centuries before [2]. People just didn't notice or forgot, despite those documents being some of the most poured over in history.
[1] https://en.wikipedia.org/wiki/Dynamic_soaring
[2] https://royalsocietypublishing.org/doi/10.1098/rsnr.2018.002...
What do you mean by that? Good writing? Good rigor? Good research work? Have a fundamental discovery? There are not as correlated as one expects.
This one's tricky to measure. If you go by the number of words published, they're a lot more productive than they used to be. Ten papers won't even get you tenure. In terms of metrics, the game is a lot more competitive.
I would argue that the bigger problem is that academic science is now consumptive. Think of all those tuitions dollars the universities get because they run a protection racket over (what's left of) the middle class job market. Now consider how terrible the academic job market has been for the past 30 years, and that professors who want even a chance of getting respect basically need to self-fund by getting grants. Lots of money is going in, and how much is coming out? Something, for sure, but is it worth it? Or is it like U.S. healthcare, where we're paying 4x for a not terrible but merely acceptable product?
I was the most creatively productive when I had a longer planning horizon in front of me.
Right, and this kind of opportunity has gone extinct, due to the hypercompetitive culture you get as our society disinvests itself in research and as funding becomes harder to find and more winner-take-all. If the neoliberals have their way, being a professor will just be another next-quarter focused corporate job within ten years.
Not sure where that came from. The NSF provides a lot of 3 or 5 year grants. The science org I work for is mostly funded by 3-5 year grants.
I feel the same way. I've never been in academia, but everything he said matches the negative incentive structures I've always seen in the corporate world. For the most part, the incentive structures are designed to catch cheaters (or "slackers"), but since actually discovering and creating something is virtually indistinguishable from "slacking", it just doesn't get done.
And this is a very real issue, for instance in physics. It means that if you go for research there, it is likely to be meh.
I did my PhD in physics, I was thankfully at a moment where simulations and ML were just starting so I could have fun. There was exactly zero interesting discoveries in nuclear/atomic/particle during my time in academia. And the 40 years before.
Compare this with 1090-1950. These years were simply incredible (especially the ones around 1900, to ~1920). You had almost every day discoveries that were shaking the world, and everyone was aware of that.
This is not the fault of the scientists, this is just that the world is like this. It is indeed like complaining that there are no more continents, but then the reasonable thing to do it to invest more in teaching, or applied science.
1900 yeah?
To be honest, I really think that we have wrong metric. Most importantly, if you try to ignore them even if you get some local support within your group. Good luck getting any fund because funding agencies evaluation have the same problems.
Academic researchers spend the majority of their career in tenure or in PhD, with untenured/tenure-track status for only 7-9 years out of a career of, say, 40 years. So one would expect that 75% of academic research time would be spent on "breakthrough topics".
anyone of any era could claim this. and many did. yet here we are, still discovering things, and not discovering new continents.
it is folly to assume that we have discovered everything, or even a small fraction of everything.
I think you will find that it is difficult to find someone who has been vetted by a system who would be open to admitting that said system is fundamentally flawed, especially when successful passage through that system grants things that those who have been through it want to have, such as academia does.
We definitely haven't discovered everything, and we're actually making way more discoveries much faster than people back then. But the things which are easy or even "not super hard" to discover have already been discovered. Most of the discoveries require background knowledge or are more "niche" things, because discoveries which are really big and affect everyone are easy to find and going to have everyone searching for them.
In fact we could maybe discover a new continent. But it would have to be tiny or underwater or camouflaged or otherwise have some reason that despite having a map of the whole Earth and satellites everywhere, we haven't discovered it yet. When Columbus "discovered" America there weren't nearly as many ships floating around as there are boats/planes/satellites today.
Meanwhile in Earth... oh, man, this tiny rock in Pluto has exactly the shame shape of Mickey Mouse's dog, call the press!
They said that the big obvious discoveries have been discovered. There's very little in physics now that doesn't require thousands of person hours, millions in equipment and unbelievable amounts of computing resource.
The stuff that could be discovered by someone working alone has pretty much all been done, because we've already thrown millions of person hours at it.
That's not to say there aren't major advances to be made, it's just REALLY HARD.
They also did not say there are major advances but they are “really hard” (your words), they said there are no major advances left (by comparing them to continents, which we’ve already all discovered). There is no other way to interpret their words, unless you believe there are additional continents on earth to discover. Maybe atlantis?
The OP is not commenting on individual contributors vs large groups but I will comment on that as I disagree. You are describing the current state of physics (and many other stem fields) and the popular consensus about future advancements. While certainly having hundreds of researchers is necessary on many projects, you have no basis to claim an individual will not make a major breakthrough in the distant future. And frankly, as much as they are necessary, giant labs are strongly encouraged by our current culture. Large profits from monetizing research into products, corporate involvement and funding, staggering bloat in universities, fame-chasing, printing off papers like buzzfeed articles, unprecedented levels of organized fraud in academia… very little of the current culture is conducive to modest, brilliant individual contributors in many fields.
You have no idea how the culture will change in science, how much more intelligent and capable people will be than we are in the distant future with natural selection and gene modification, and whether there will be any more breakthroughs by individuals or small groups that will seem “obvious” in hundreds of years. To say otherwise is frankly dogmatic caveman thinking.
Poor choice of words aside, the point stands.
And as for humans being far more intelligent in the future, I don’t buy it. Maybe in millions of years with evolution, but I’d bet on humanity either killing itself or replacing itself before that happens.
As for the other point, it is indeed standing, on air.
Assuming that this is generally the case (which I do), and given that longer planning horizons have become rare in academia over the past decades, the logical implication does seem to be that productivity in science decreases.
How to measure and how to value academic productivity is another interesting question.
(1) Making 30 researchers spend time on grants when only 1 wins is a waste of researchers time, regardless of productivity.
It's a system built for bureaucrats who don't know to evaluate work-product from those who do by comparison if not mutual critique. It's a wealth transfer from researchers to taxpayers.
(2) Grants always are on topic, but the topic is mostly selected to avoid political embarrassment. In the 1990's, it was HIV. In 2010 it was Alzheimer's Amyloid Beta. Then computer security. Unlike the internet or jet engines, these have not panned out as foundational research (except perhaps for some of HIV). Again, topics are selected not to deliver value to citizens, but to politicians.
Yes, this is inefficient. Yes, academic bureaucracy sucks and the publication game is easily corrupted and early advances are more relatable than later ones.
But the real problem with science in the U.S. is that there are not enough jobs for scientists, because companies are happy to get research and development done more cheaply elsewhere. Companies have only disincentives to develop local labor pools.
That's the negative incentive that dwarfs everything in academia and publicly-funded research.
In what word is computer security not a foundational topic? There's lots of reasons to critique the way NSF/NIH/DOD/etc allocate funding, but this is definitely not one of them.
Are there other places where the science-funding mechanisms work well? Otherwise I do not see why your argument about companies being cheap makes sense.
How would you promote teaching and knowledge transfer while not needing to do this? On the job learning?
Something I think would help the incentive situation is providing more incentives for teaching and service. My grandfather was a professor emeritus and I can't find any record of him authoring a research paper (or a PhD thesis for that matter) but even 30+ years since he last taught I still run into people who rave about the formative experiences they had in his classes. He's a minor celebrity among local engineers of a certain age.
Such a person could still provide a lot of value, but it's harder to measure.
If we are trying to improve research productivity, then removing or dramatically reducing teaching responsibilities for researchers sounds a lot more sensible. Besides, modern pedagogy is getting increasingly complicated and having teaching specialists can be much more beneficial to your average 1-2 year undergrad.
Not to diminish his teaching career, it sounds very valuable, but not in a way that's directly relevant to the OP.
If, instead, we made it easier to advance your career as exactly the kind of pedagogy specialist you described (as well as through leadership in professional organizations, work in the community, and all the other kinds of service), I believe you would get fewer applications to those research grants. It might result in better post-secondary education as well.
FWIW, also decided not to go into academia because of how much smarter I realized I needed to be to be a top tier academic!
Somehow.
https://fastgrants.org/ did some good work for covid. Could maybe be a starting point for a new system.
Academic research is not a place to go for extrinsic motivation. If you're not primarily in it for love, you're going to have a bad time.
An example: Who has heard of the Wolf Prize in physics? It is in the set of prizes known as "second prize to the Nobel Prize", yet essentially nobody outside of the field will react with, "Whoa, she won a Wolf Prize? She must be really good."
Even in the well-resourced research groups, researchers will be find much better extrinsic reward outside of academia. Advancing the boundaries of our understanding is priceless intrinsic motivation for those who can find a way to stay, but doing so requires substantial sacrifice -- more than many of those in academia realize.
When you go to top universities, you'll find plenty of academics who are extrinsically motivated:
* Number of research papers
* Cumulative value of grants
* Awards
* Titles. This was insane. Apparently being a Fellow of the IEEE wasn't enough, so they actively sought becoming Fellows in adjacent societies.
Yes, I've also understood it this way.
Turing Award winner Michael Stonebraker talked about this at length here:
https://www.youtube.com/watch?v=DJFKl_5JTnA
Some points:
- Original (risky) research is detrimental and likely fatal to a career in science
- Actually creating systems doubly so
I'm a former professor who has written many proposals and won some of them. I've reviewed many more.
I did a lot of thinking and research when writing proposals. In a proposal, I tried to clearly explain the state of the art and how it could be extended, what resources I would need to do that, and how that work could be structured to make success more likely. I might even do a pilot study to show that the key idea is promising. Note that 'how to extend the state of the art' can be a seriously creative inventive step, or it could be the obvious next thing. But creativity does happen at proposal stage, and it certainly factors in my reviews of other's proposals.
My point is that writing proposals is not entirely a waste of time. It's a bit like writing a movie script. It's not a movie, many scripts never become movies, but for some movies some of the best ideas were in the script. Same for proposals.
This isn't worded properly. It's clearly rational for the scientists to spend time on the pursuit of grants. Otherwise they wouldn't be doing it.
$100 in grant money is not the same as $100 in university income. In fact, the whole point of the grant is to cover the cost of doing research, and in principle should leave the university's net revenue unchanged. (In practice, overhead changes that, but that's a different topic.)
The problem is that the university's objective is to spend as much on research as possible. Grant money is simply the fuel for that process. All else equal, more expensive research is very strongly preferred, because that increases research spending, and that improves the standing of the university.
That is a seriously distorted set of incentives. All perfectly rational. Just insanely stupid.
Is that worded properly? Are you really saying that nobody would ever do something irrational?
If that's not what you mean, then we have to look at what "rational" means here. I think parent didn't mean "rational" in the sense of "maximising personal gain", but "maximising output of quality research".
This is one of the reasons why I left academia. The insane run for grants was just starting and i said that I do not have any intent do do that.
I was a scientist. Doing science. Not filling in papers and going to belly dancing shows.
When I was told that this is compulsory and nobody else is going to do it, I sad good bye and left for the industry. I was earning 10x more, had no worries about funding or copier paper. Plenty of my friends did it and then academia was crying because of the brains drain and how unfair it was. The salme who said that grats are compulsory (they were sitting in the various jurys to accept them, instead of doing science)
This statement may not be entirely accurate. Philosophers, scientists and mathematicians in the early 20th century and before that were independently wealthy or had some sort of “royal” patronage that allowed them to pursue their interests for “public good”. Things changed in the later part of the 20th century but a scientist was a rarified breed until the 1970s. I don’t think most people had the means to pursue a scientific career. Grants and funding exploded at the end of the 20th century and foreign students (especially from China and India) provided the “cheap labor” that’s needed for a lot of the STEM fields PhD/post-doc armies. This has essentially led to a proliferation of scientists in academia. Abundance of school loans means universities can expand teaching faculty and then add research responsibility. It’s a vicious cycle.
At least this is my hypothesis.
We live in an age of institutional decline and it is severe. You see this in (trade) publishing. Your publisher no longer builds your reputation; the publisher has pushed that responsibility unto the author. The ones who already have the personal resources necessary to market their books get further validation and credibility; the ones who don't will go unheard. Academia's the same way: universities no longer provide funding for people of excellence; rather, they have put the onus of funding on the professors themselves--you'll get more funding (and published in better places) on account of using their name, and for that they take a cut. The relationship has inverted; rather than nurturing emerging talent, these institutions are nurtured by emerging talent, and this vampirism is sustainable because those talented people have no other choice insofar as all the other institutions are failing at approximately the same right.
Consequently, we have widespread duplicated effort, channel-flooding due to metrics-gaming (gotta get that h-index into the three digits before tenure time) and a corporatized, mediocre culture in which agreeability (negatively correlated with excellence and conscientiousness) matters far too much and salesmen run the day.
Historically, there were nations of priests and nations of warriors and nations of farmers. We've become a nation of sellers; but we no longer have much to sell but our own talk.
I don't know, for sure, how to solve this. Anyone who pays attention can see that capitalism (which invariably becomes corporate capitalism) is a dead end at a 21st-century technology level... but of course the eradication of capital is merely a necessary, not a sufficient, condition for scientific excellence. Going socialist is mandatory if we want to fix this, but alone does not guarantee much--there are a lot of cultural changes that probably need to happen before we can build healthy institutions again.
Of course, the lede is buried exactly where expected. How novel an idea it is that the the system is separable from its components and that utopia is indeed achievable if only the rest of the world would wake up and learn to serve your idea of a higher purpose. The inevitable counterpoint is that the system is responding predictably and efficaciously to a change to its environment: more undergraduate funding leads to more students pursuing graduate degrees and a greater deal of competition in the academic space. The "problem" is self-correcting. It turns out science doesn't need happy scientists.
The reason why you don't attach your name to comments like this is because your possession of a name worth attaching would preclude this comment from existing.
Socialists are actually oppressed in many countries
Yes that's primarily due to capitalism, specifically neoliberalism, crony capitalism and late-stage capitalism.
Unfortunately socialism isn't enough to save us, because even democratic socialism demands full employment, which is increasingly at odds with automation and UBI.
I don't believe that we have come up with a system yet that incentivizes the kind of one-off revolutionary invention that most of us got into programming for in the first place. The kind of economy where a single invention frees millions of people from forced labor.
Sergey and Larry come up with PageRank and suddenly everyone wants their search engine and they're set for life. Google wins the internet lottery and keeps all of the trillions of dollars. What about the rest of us? We invent something and our business fails due to long-tail effects and we die broke. 90% of businesses fail in their first year. So it's becoming increasingly competitive as tech improves - the ultimate tragic result. Which self-evidently trends toward a crisis of techno-capitalism in the future: The Singularity. Probably the last failure in human history.
We're looking at a situation by the end of the decade where one guy generates an entire movie from notes scribbled on a napkin with some future variation of DALL-E 2 and makes a million dollars. Meanwhile another guy who doesn't have the right connections spends his life working in the service industry as a wage slave.
Meaning that the most likely indicator of someone's success is a coin flip as we enter this era of neo-fuedalism.
Meaning that civil unrest and violence are all but inevitable now (just turn on the TV to see it everywhere).
Now, I think a lot of people on HN and the world at large have not tried working for themselves, so have no idea how hard money is to come by. Sure, we can make a quarter million dollars every year working 40+ hour weeks at a FAANG company. But make that from a personal project? Highly unlikely.
Writing this now, I have given up on a solution to this coming from the top. People close to me, even the majority of people on HN, don't seem to get it. They'll never get it. Something in the idea of UBI irks them deep down inside, just like with student loan forgiveness. So it's over.
On a personal level, I'm looking towards solarpunk and local cooperatives that create resource streams for people outside of the financial incentive. I think it's possible to invent robot kits that provide things like hydroponic produce for very low cost. Once people have their basic needs met, it liberates them from having to beg for grants or even a job. Which is why the entire status quo is geared against this. Rents and basic expenses will squeeze us even harder before the end. Expect crushing regulations against off-grid living, just like how rain barrels have been criminalized in the southwestern US.
As for successful grants, I have seen that, now, 50% of the amounts immediately goes to "overhead" and the PI needs to pay for stipends and equipment from the remaining 50%, yet there are further "transaction charges" even those activities.
The publishing, grant winning, etc are simply "KPIs" so beloved by the MBA hordes.
What I found incredible is the lack of push-back from academics against the encroaching takeover by the biznoids. Speaking of whom, they don't give a damn about research, academia is merely yet another territory to which they extend their rent seeking (for themselves) activities.
Paradox of tolerance. Academics were too peacefully inclined to realize what is happening to them and their jobs until it's too late.
As the left re-emerges, I hope it will evolve into something with some fight in it. De-corporatization is going to take decades and it's not all going to be pretty.
While I don't have the data to support this my intuition is that there are now many, many more researchers than in earlier years- both senior researchers who already have a career and junior researchers that are trying to get theirs started.
The amount of science funding probably has also increased, but there are just too many people exiting PhD programmers for all of us to find cozy research jobs where we are free to pursue our interllectural interests undisturbed while the public money slowly but steadily drips in.
In other words, a moderate pot of grant money combined with a huge number of new PhDs makes for huge competition.
But aren't there still some scientists who work outside of grant funding and prestige ratrace? People at industry research labs who don't write grants, and aren't trying to impress a tenure committee, for example. Shouldn't we be able to point to them and say, "look how much more productive they are"?
Here's my research on the 'Challenges facing Academic Research' - https://miro.com/app/board/uXjVOkNfljM=/?share_link_id=58427...
Quotes that I found important:
"It discourages rigorous research as it is difficult to obtain enough results for a paper (and hence progress) in two to three years.
The constant stress drives otherwise talented and intelligent people out of science also."
- Anonymous
"End the PhD or drastically change it. there is a high level of depression among phd students. long hours, limited career prospects, and low wages contribute to this emotion."
- Don Gibson, Scientist at BioConsortia
Funding "affects what we study, what we publish, the risks we (frequently don't) take, it nudges us to emphasise safe, predictable (read: fundable) science"
- Gary Bennett - Neuroscientist at Duke University
"We need to recognise academic journals for what they are: shop windows for incomplete descriptions of research, that make semi-arbitrary editorial [judgments] about what to publish and often have harmful policies that restrict access to important post-publication critical appraisal of published research." —Ben Goldacre, The Datalab, epidemiology researcher, physician, and author
"An estimated $200 billion - or the equivalent of 85% of global spending on research - is routinely wasted on poorly designed and redundant studies." - https://www.thelancet.com/journals/lancet/article/PIIS0140-6...
"As much as 30% of the most influential original medical research papers turn out to be wrong or exaggerated." - https://jamanetwork.com/journals/jama/fullarticle/201218#COM...
"The BMJ found that one-third of university press releases contained either exaggerated claims of causation (when the study itself only suggested correlation), unwarranted implications about animal studies for people, or unfounded health advice." - https://www.vox.com/2014/12/10/7372921/health-journalism-sci...
Your money is not going to research. Instead, it is going to bureaucracy and "extrinsic motivations" (researchers status games)