Is novel research worth doing? Evidence from peer review
pnas.org
pnas.org
list of own publications
There is immense pressure for early career researchers to work on low-hanging fruit that will give easy guaranteed publications: make a minor change to previous work and write a paper about it; write a paper about what other people wrote papers about (either a review or a meta-analysis); etc. etc. This is how you rack up publications for your tenure review. It's also how you rack up publications to get funding.
Once you have tenure you have a bit more freedom, but you still need to worry about funding (generally). And by that time you've been in the habit of grabbing the lowest-hanging fruit for 1-1.5 decades, assuming you were ~well trained~ in graduate school. After so long reinforcing that habit it becomes hard and scary to build a ladder and go after the better, higher fruit.
I know many young researchers who've specialized in meta analysis for that reason.
There's also tremendous value in doing the dishes and mopping the floor. Without it we'd get diseases and live in filth. And when we do this, we should be respected and appreciated. But - not as brilliant trail-blazers.
- Not novel: Novelty is ill-defined. Most journals and conferences have guidelines on this that come down to "will someone find it useful" which should make "not novel" a rare review, not the most common. Additionally, what's obvious post hoc is not obvious a priori (I actually saw an AC override reviewers because of this!). If something is well written it also comes across as easy and simple. Lack of novelty could be a true lack of novelty or just an indication of a well written paper.
- Incremental: All research is incremental. Research can be too incremental (not enough work done) but this is never expanded upon in those types of reviews. This kind of statement is meaningless in isolation. Incrementalism will always exist with a publish or perish paradigm. One cannot publish breakthroughs year over year. Those take significant periods of time or a lucky break. Neither is a common occurrence, by definition.
- Not enough experiments: "Money is all you need" has become a popular phrase. There may not be enough experiments and this can be true but perfection is the enemy of good. Especially in research, where the experiment space is non-exhaustive. Can a reviewer be satisfied in a finite amount of time and with finite resources?
I've started to change my mind about where blame falls for all this. It starts with the reviewers, but if this becomes the norm then the system is at fault. It has not removed the rotten apples and thus let the barrel spoil. All while we've had decades of discussion about this happening. I think a lot falls on the Area Chairs and Metareviewers, as they should be preventing these types of reviews to pass. But the whole incentive structure of publishing is wrong. There is high pressure to reject and zero pressure to let papers through. This especially hurts our junior researchers (grad students) as it becomes a lottery process for determining if they can graduate. After all, we can't have any wizards without any noobs.
Honestly, I don't like opinions like this. That's just passing off the work that needs to be done to fix these problems. It wasn't even a hundred years ago when people published more in the open and the publish or perish paradigm didn't exist. The latter is a relatively tool, and collapsed because Goodhart's Law.
We can make the changes, but not with defeatist attitudes.
But even from this experiment we can see that there are people actively trying the fix the situation. Likely naive, but something is better than nothing. I wish there would be more push for ACs and Meta Reviewers having the highest standards, but this is where we are.
I am totally for us abandoning journals and conferences and just publishing to arxiv and open review. This is exactly what I would do if I was independently wealthy and could research without the constraints that I am in. But we also need to recognize the issues with this and that popularity and outreach ability greatly affect perceptions of quality of the paper. That this is not meritocratic.
I've also worked at research organizations specifically touted as being highly independent with freedom to explore novel paths and the story is basically the same. Ultimately, someone somewhere holds the purse strings and conservatism kicks in. You at minimum need to spin your research to fit popular trend keyword language or make it apply to these areas.
Financial structures ultimately dictate the lack of support for novelty. Basically, novel research has freedom only when it's independently funded, which limits the scale of most novel research. No one wants to take risk, they want to market taking high risk while taking low risk and selling mediocrity. I think the fundamental underlying issue we have across multiple societies is that we no longer proportionately reward risk anymore and that's why people seek low risk everywhere. Taking high risk often has limited opportunity for reward, even if you are successful, so why bother? You'll net more success taking low risks and only fools take high risk.
The same can apply to war-like scenarios. A lot of novel biological research got funded during COVID. We should expect to see a burst in advancement in biomedical technology as it bears fruit over the next 5-10 years.
The core problem is probably that conservatism makes the most sense at the individual level for most people. Even if you had a casino whose net payout was positive, it still may not make sense individually to gamble. If individual rewards are sporadic and concentrated on a power law distribution then statistically most players will lose.
An obvious solution is a less power-law type distribution, which tends toward some kind of socialism, but that has its own problems. Nobody's figured out yet how to provide a productive channel in a socialist system for humans' totally normal drives to compete and maximize individual outcome. Trying to suppress it doesn't work; it's like trying to totally suppress sexuality. Extreme socialist societies (e.g. Soviet Communism) often evolve into totalitarian mafia states because if there is no positive outlet for those urges they drive people into crime. Eventually you get a meritocracy concentrated in crime and the mob takes over everything.
Nobody has figured out how to maximize individual outcome, or provide a channel for such pursuit. Period. Socialism has nothing to do with it. People are crap at managing more than a few dozen other people, even for the best managers who have ever lived. And the systems we have in place since agriculture took over are trying to manage many, many more people.
Most people in more developed countries live in partially socialist systems. This doesn't prevent them from getting private sector jobs, creating companies, and otherwise trying to pursue the maximization of their individual outcomes.
The reason is that those with ADHD are often physiologically incapable of not pursuing an interest. Often those interest lie in novel areas. It’s unfortunate for those with the traits, but a net benefit for society.
So there’s a game theory equilibrium wherein successful societies will retain a certain percentage of those genes or become too conservative.
I would assume that the kind of personality your are referring to is adaptive for the individual as well, especially the mild variants.
Not necessarily, examples from several species where males get eaten after mating would seem to counter that. In some the males try to escape if possible to mate again.
> the kind of group selection this process would require is implausible at best.
There can certainly be group theory optimums that are less beneficial for individuals but beneficial enough to the whole group that traits bad for an individual will persist. I’d say bees dying after stinging an intruder would be an example. Well or even the hive principle where only the queen reproduces.
Still I would agree that the mild variations of adhd traits can be beneficial to some individuals. At least enough to largely eliminate the traits despite being largely maladaptive in non-nomadic societies given that we’re not hive creatures. One example I’ve read was a paper that showed increased offspring in individuals with adhd traits in nomadic tribes in Africa, with decreased reproductive rates in adjacent non-nomadic tribes.
Our modern society is odd in that it’s not like either traditional society. ADHD traits can be very beneficial for individuals in technology careers requiring creativity while simultaneously being painful due to society being largely optimized for non-adhd neurotypes.
I've recently left academia after ~10 years to work on more blue-sky-stuff in industry, and this rings so true. It took me a few months to unlearn academia's focus on the minimal project we can do to get the next paper out (still unlearning!).
It's hard to get in, it's harder to get out.
Ingrained work habits are hard to break.
"A group of people who have never used a computer are recruited for an experiment, using variations of Unix commands to manipulate files from a command line interface. The users have never heard of computer files before and they don’t in any sense have the utility of files internalized.
The experiment is to determine whether something the experimenters have called utility versus expressiveness is more highly valued. One set of users is taught commands like “rm,” “mkdir,” and “cat,” etc., while another is taught “delete,” “new_directory,” and “type_out,” etc. They are asked to do some housekeeping tasks on a file system. Their speed and accuracy is measured, and they fill in a questionnaire about how hard and how intuitive the task was. The paper itself about these experiments touts “statistically significant results,” with lots of p-values, etc.
No HCI academics bother to invent windows, icons, a mouse, and so on. Such work is viewed as too informal and unscientific."
I just finished my PhD and these lines were in my head since they were published. Between an advisor who would only greenlight plans if they immediately led to a paper and an embarrassing amount of impractical "safe" research that only makes for good tweets or headlines, I've given up hope on anything meaningful coming out of the Ivory Towers. The robot revolution will not be peer reviewed.
I wonder if this could be alleviated by incentivizing the publishing of papers on work with no statistically significant results, as has been discussed here and elsewhere. That might increase quantity of output, while building a useful database of what not to spend time on.
Even if it takes decades.
Which is how long research labs take anyway.
Maybe this means something. Talk about HCI. Coming from a guy back in 1982 who thought pointing & clicking was pretty cool. Almost a decade before the mouse was widely recognized, the trackball/firing-button was in use industrially to select graphic items. At one of the control rooms of the most automated marine chemical installations at the time, they had consoles that looked like NASA Mission Control during the moon mission. Tied into their mainframes. And wired into their tank farms, pipelines, and ship docks. Well a somewhat humble oil company can have more money to spend on this kind of thing than NASA sometimes.
The oversize CRTs would have the sharp monochrome graphics showing little pictures of the available tanks on the left and available docks on the right. In between, connecting them, was the pipeline maze. Scattered across the pipelines were the little markers representing the control valves, which physically controlled where the product was going to go. They were like a little blank spot in the pipeline, just big enough so a cursor could go there and select it with a click so it would no longer be blank. Using a big console-mounted trackball like would appear years later on the Atari Millipede arcade game. And that would end up opening the valve so the product would flow through that segment. Until they had an unbroken line from tank to dock.
When your ship was coming in, you would co-ordinate the exact dock and tanks that were going to be involved, and after review the graphic layout would be executed physically in the field before measurement & testing could be directed for that vessel.
So I went home to my Atari computer, unplugged the Centipede cartridge and popped in the BASIC. Kept the home trackball with the regulation cueball plugged into a joystick port. Mocked up the console GUI in color since this is what really set new things like home Ataris apart from earlier plain-text machines, so besides basic graphics I got accustomed to pointing & clicking at home using my own code. No Mouse. Statically like I had seen. Really just one more step away from people intuitively pointing & clicking for targeting in Missile Command, which was the other main Atari game using the trackball.
I read a number of Atari books and experimented quite a bit and I could see that the truly fast games were really a product of deeper experience in things that I did not possess; immersion in the emerging microprocessor ecosystem, and very long hours of progress, which I could not compare to as a hobbyist. I wanted to make an accurate navigational Google-map type framework where the graphics would be first-person 3-D as you drove along realistic freeways. You would certainly want a racing mode, so that would be a bit of GTA style landmark navigation either way. Obvious idea, but that was really too much for me and my equipment.
Instead I did a much simpler slow-action logic game called Master Mind where you deduce the color pattern that the opposing player (or computer) has secretly encoded. You simply dragged & dropped the colors into position, and the computer answered.
So I was dragging & dropping quite a few years before most of the world had a mouse to point & click with in DOS. Heck, before most of them even had DOS. But it was really just one day's HCI work. What else are you going to do with the equipment? You're going to have to experiment for yourself. It's good not to be hampered by all the traditional academic obstacles.
I think nothing beats returns on inventions which have already been accomplished compared to research yet to be done, but the wisest investments are in the balanced performance of both. They go together so well.
"The enemy of art is the absence of limitations." Seeing research budgets spent on equipment - that either never gets used or is poorly maintained - just to save a spot on a given funding bracket leaves a bad taste in my mouth.
I built my own laboratory from surplus research equipment where I only had certain types of pieces but I ended up with stronger resources than any one original owner.
Was able to use more features over more years doing invoiceable routine work the research way, and between vessels explore applications that were not invoiceable. That's where the real progress is, and it's possible to make it eventually pay off. Got so much more out of the equipment for many more years than the original companies. And I made money making progress on the instruments while it cost them money every inch of the way.
So what if I could have gotten a whole lot more out of millions of dollars worth of new gear, I was just making the most of the limited stuff I could get affordably instead. I was also a lot more of a loose cannon when it didn't really cost me a million dollars either.
The large petrochemical labs, with all their PhDs and virtually unlimited budgets compared to mine, could not do what I could do with their old equipment. Even after they had gotten a head start on newer models they had acquired before declaring their older ones surplus. And it took even more time before I had access to the surplus.
Even though these were industrial labs they were hampered by a carryover from the academic laboratory background.
And I was welcome to the unexplored possibilities overflowing out the basement doors with the truckloads of "obsolete" electronics. So I can't complain.
Ok, so “novel” research is 6.5% more likely to be accepted if submitted, but how likely is it to even be submitted? I would argue that doing “novel” research is much riskier at the earlier stages, and that you need to look at the whole process.
Here’s a thought experiment to perhaps better explain what I mean: Let’s say you’re playing roulette, but with slightly modified rules. If you bet on a color (red or black) you have a chance to double your money as usual. But if you bet on a single number and get it right you only make 6.5% more, or 2.13 your money. Would you bet on a color (i.e. be conservative) or a single number (i.e. try to be “novel”)?
So you still publish, but also get to explore "interesting" stuff (that also needs to be researched). You are covered by your less risky activity.
So, a single number 20%-30% of the time, and a color 70%-80% of the time. And your risky activity could become your comfortable zone if it takes.
Getting published is not the only worthy thing. The content itself and how you enjoy the thing too.
Kind of an odd proxy for novelty of the research, but ok I guess. Maybe they’re going for cross-disciplinary work?
For example, I work in a multidisciplinary field. Once I counted the citations per word of text in papers published in the separate disciplines. I found that biology-oriented research had a cite/word rate that was about 5 times that of physics-oriented research. I don't think that says anything really significant about the two fields, but it is something worth knowing, when judging tenure files.
You can try looking at the past, and derive criteria for it from that, via machine learning for example, but I would be hesitant to leave something like that up to a machine. Also times change, so criteria that worked in the past might not work now. Also, if you learn those criteria once, and then fix them, people will just game them.
You can't, as "truly interesting" is context dependent, changes over time and something that everyone deems as futile may become interesting - that's the point of research. You just increase the bar of entry to get people who work very hard and leave it to them to decide.
You don't want people who work very hard. You want people who will EVENTUALLY put out new, original, and truly interesting research. HOW they do this is not up to you.
And there is a difference between truly interesting research, and just busy work research. It's not that easy to identify truly interesting research without the benefit of hindsight. It is somewhat easier to identify busy work research for an objective subject matter expert (but of course this is not 100% either, and personal preferences can definitely cloud the experts judgement).
It depends on why you are doing it, I suppose. The framing of the piece seems to have a lot of baked in assumptions that likely make sense if your goal is, I guess, successful career in academia.
That's not the driving force behind all research. Research pursued for other reasons may get little to no press but may be successful for the metrics/goals that drove the decision.
I'm in ML and all the academic issues are on full display here (comes with any hyped research area). Reviewer 2 will always knock you on novelty no matter what you do. That's because all research is to some extent derivative. As it should be. We shouldn't be reinventing the wheel. But on the other hand, all research is incremental (especially in a publish or perish system. You can't publish breakthroughs frequently). Leaps and bounds don't happen often and they generally do either by accident (someone doing something dumb) or because someone was spending a lot of time on that area (likely not publishing in main track or at all). But novelty is also hard to measure post hoc. Most things are non-obvious a priori but are obvious post hoc. Reviewers go after these all the time and all it encourages is researchers to make their works convoluted. I've literally been told that papers I've written are too simple and straight forward. I've also seen those papers get better reviews after complexifying the text. I don't feel good about this btw.
But I want to give a perfect example of all this in ML. Diffusion models. The stochastic noise process has been around for decades. People have been applying them to generative models for awhile too. But they were basically unknown until they beat GANs at image generation and within a few short months they are a household name (even more than GANs). This looks like a bigger breakthrough than it is because they were flying under the radar for so long and really just one team spent a lot of time with them AND had access to A LOT of compute resources. But still, I think this should make you think about how novelty works and if you're reviewing, maybe don't squash papers because you think they are dead ends. Diffusion was a "dead end" until it wasn't.