Sorry for the length.
Traditional academics have the "publish or perish" incentive. In practice this means that they prioritize "quick wins" over "slow wins", e.g., given a choice between publishing 1 paper after 1 year (quick win) or publishing 5 papers after 5 years (with no publications before then) (slow win), they'll choose 1 paper after 1 year. If an academic goes too long without a publication, that will be counted against them. The low hanging fruit for quick wins has been taken due to this incentive, but I see no shortage of slow wins. (The scenario I describe is an extreme case, but the same incentive still exists in less extreme cases.)
There's also the problem that what can get funded is not necessarily what's most important. Norbert Wiener discusses this at length in his 1950s book "Invention". Wiener notes that despite the obvious political differences between the USSR and US, research funding is allocated similarly: by people too far removed from the actual research, who are often not in a good position to evaluate its merit. It doesn't matter if these people are managers, bureaucrats, or fellow scientists. There's generally an asymmetry in information between the scientists requesting funding and those able to provide. (Having more time to learn about each proposal could help, but the trend I imagine is that time available to review each proposal has decreased over the years.) This ignores the lottery like nature of the entire funding process.
To get more specific, both of these problems are would disincentivize a traditional academic from publishing this paper I recently submitted to a conference:
https://engrxiv.org/35u7g
In principle, a traditional academic could have written this paper. It's possible, but I think less likely because of "publish or perish" incentives. (To be clear, I am a PhD student right now, and most of the time I was doing the research in the paper I was a TA. I don't have the "publish or perish" incentives that make this research less likely. If I stayed in academia longer I would.)
My advisor and I tried to get funding for this project, but our grant proposal (I wrote the vast majority of it) was rejected for reasons beyond our control (which I have no problem with). We received positive comments on the proposal, and it served as a draft my PhD proposal.
Without going into detail, the paper develops a simple mathematical model of a certain physical process. The theory and its validation would not have been possible unless I did two things that traditional academics seem to think are a waste of time:
1. Very comprehensive literature review.
2. Very comprehensive data compilation.
Now, I think most people would believe these two are just what academics do. But apparently not. Traditional academics are incentivized to do the bare minimum to get another publication. There's an epidemic of copying of citations and merely paraphrasing review sections of papers without reading the original papers, and I think this is caused partly because of these incentives.
The literature review I did (not all of which made it into the short conference paper) was considerably more comprehensive than any I've seen published in the field before, and I was able to synthesize past theories and improve upon them by recognizing some of their flaws.
How do I know I was more comprehensive? One way is by the excellent papers I found which few seem to be aware of. In the paper I mention, papers 3 through 8 have very few citations. Some of them have not been cited at all in the past 40 years to the best of my knowledge. Someone could say that these papers are just unimportant, but they're not. In my view they're "sleeping beauties":
https://www.nature.com/news/sleeping-beauty-papers-slumber-f...
Further, I spent a year or two alone digging deeper and deeper into the literature in this problem. There were several times when I thought I probably had at least touched everything, but a few weeks later I found yet another area that I had missed. Being comprehensive is difficult and time consuming. If you just want the minimum to publish, you won't bother.
I also benefited from certain heuristics which allowed me to identify important neglected research. For example, I spent a lot of time tracking down foreign language papers and books because I recognized that this research was avoided because it was written in a different language, not because it was bad. The entry costs to foreign language literature have dropped greatly over the past decade with options like Google Translate. I've translated around 10 full papers into English right now, and produced many more partial translations. These papers have provided critical insights that were necessary towards writing this paper. At this point some people I know use the fact that I like reading foreign language papers as a joke. Traditional academics think this is absurd, but I see that there's value, just that it takes time to be realized.
It was through my comprehensive literature review that I got the idea behind my data compilation. By taking advantage of the properties of a special case, I was able to get information that most researchers in this field seem to believe is extremely difficult and expensive to obtain. I did not come up with the idea myself. I was translating a 1960s Russian language paper into English when I realized based on what was written in one paragraph that I could use the properties of a special case to get some hard to obtain information. The author was actually leading into this. The next paragraph explicitly said the author was taking advantage of the properties of a special case. So it wasn't very original on my part. The 1960s Russian researcher didn't have a lot of data to use, but there's a lot now 50 years later.
So I started compiling data. I get the impression that few academics would have compiled even half as much as I did, or have been even half as careful as I have about it. I was very careful to select only the least ambiguous data sources. Out of over 100 candidate data sources, there were only around 20 which were acceptable. I then took the time to carefully transcribe all of the relevant data from these sources, and develop a computational framework to handle this data (based on Python and Pandas). It was probably at least 6 months of work, but I can produce several papers based on it, so it's worthwhile in my view. My advisor was not initially enthusiastic about compiling this data, by the way. He's a successful traditional academic, however, and his intuitions are calibrated differently than mine are.