I wouldn't be productive enough for today's academic system (2013)
theguardian.com
theguardian.com
total number of publications, citations, grants obtained and their monetary values, number of students, impact factor of the journals in which the research was published, h-index, gender, sexual orientation, ethnicity, and public visibility.
Did I miss anything?
For once, neither politicians nor administrators were responsible for a system of "perverse incentives," since these gray figures are not involved in either hiring or tenure decisions.
Case in point: one of the leading scientists in a field in which I dabble, and former chief scientist of The Nature Conservancy, routinely had his name on dozens of articles each year. Given the volume of his other academic commitments, at most he would be able to take a look at those papers, certainly he was not able to contribute meaningfully to (any of) them. So why does a top scientist, at the end of his career, whose place in the pecking order is established, still feel this need to have his name on papers to which he has contributed nothing, instead of taking a different path, in which he demonstrates, with facts, that unattainable "productivity" is not what scientists should aspire to?
Someone might say, "Well, maybe it's not him, but his students or collaborators who want his name, so they can say they published a paper with an important scientist." And why would they want to say that? To impress other scientists, the ones who will give them professorships, appointments, fellowships, etc.
A world I am lucky to have left behind.
Yes academics (well typically more the lab leaders etc), we're involved in some of the system that we see now, but it was in reaction to the pressure from politics and society. What do you expect people to do when the places that fund them start to demand measurable outcomes for something that is inherently difficult to measure. You come up with some (often arbitrary) metrics. That whole dynamic then just became an anweful feedback process.
The public (let's take the 75% of the educational attainment level folks, excluding PhDs, so non-academic people who are educated enough) knows nothing about research, how research is conducted, what a paper is, how a paper is published. Zero clues. You can imagine what the bottom 75% think. Researchers are considered by the public to be little to no more than glorified teachers. It is like saying that bigger and better weapons are invented and built to justify the Defense budget to the public, where for the public a bomb is a bomb is a bomb.
How do I know?
(1) I was an academic and I often had low-level conversations about my research, in my home country, in the U.S., in Europe, South America, everywhere. Not a single interlocutor outside of academia knew what is the typical process to follow to have some research results published in a scientific journal, or whether a publication in a certain journal can be indicative of more or less cited/quality research.
(2) I read comments here and there on newspapers, forums, social media, etc. Letters to the editor of your local newspaper are considered to be equivalent to a publication in Cell.
(3) Ask, for fun, like I used to do, your regular guy if (a) an electron is bigger than an atom, (b) how many people live on earth.
Re: research is useless. Most academic science is useless, and the uselessness of 95 percent of research is somehow considered inevitable by the "scientific community." But in that 95 percent there is a portion of research that is speculative and ultimately turns out to be useless, which is a good thing to have, and a portion of research (the majority) that serves only to pay the salaries of the researchers involved, is clear to everyone and their cousin that it will go nowhere, and is published in journals that no one, and rightly so, will ever read.
So if I understood you correctly, as well educated member of the public, you found if "fun" to point out the inadequacies of the scientic knowledge of members of the non acedemic elite? Perhaps your time in higher education could have been better directed.
In the sentence above, "for fun" means "for no other reason to find out something you did not know before", not "to make fun of others".
Last time I asked question (b), I got the answer from a regular joe/jane of 50 billion. Which is neither laughable nor an isolated case (you can try it yourself), but puts into perspective the demands of the public regarding academic decisions that my comment was responding to. There is no need for the regular bricklayer, of office worker, or truck driver, or lawyer to know hoe many people--more or less--are alive on this planet, but, taking it as an illustrative example, it makes far-reaching the hypothesis that the demands on scientists to increase productivity came from the "public".
I recommend being charitable in the interpretation of other people's thinking.
This is incorrect. The government and politicians are definitely some of the biggest factors.
The collapse of the longer timescale corporate R&D labs meant that research funding transferred to the government. Before that, if universities got too demanding, the academics would leave to the corporate labs.
These factors combined with Bayh-Dole and the lottery nature of "intellectual property" to cause universities to start chasing funding of short-term gains rather than long-term research.
(Famous example: Katalin Kariko, the woman behind much of the mRNA groundwork, was denied tenure and had a very difficult time getting funding).
When and where were politicians responsible for the collapse of the "the longer timescale corporate R&D labs"? Are you thinking of ATT/Bell Labs? Are you thinking of just the U.S. and just the East Coast of the U.S., ignoring a much larger world?
"Before that, if universities got too demanding, the academics would leave to the corporate labs." Academics who? Let's say biologists, who are a massive presence in academia, exactly where in the private sector were they working before the aforementioned collapse of "the longer timescale corporate R&D labs"? Pharma is strong, maybe stronger than ever, but most of the top pharma-adjacent biologists prefer to stay in academia and collaborate with Pharma, more than the opposite.
You are just thinking about a bunch of labs (Bell labs?) able to move maybe 1% of the academics at most in their heyday. And it seems to me, like I explained, that the "productivity pressure" is coming from academics themselves, not from outside academia. Before the 70's, it was fairly easy for a moderately successful Ph.D. to get a position in academia, but the number of positions available did not keep up with the increase of Ph.D.'s who wanted to do academic research. And the "productivity" measurements started.
I did not say this and am quite mystified how you managed to construe my words to imply that.
The fact that those corporate labs had a longer term focus and then disappeared is simply a fact.
> Academics who? Let's say biologists
Before PCR (1984ish?), biology had such a vanishingly small presence relative to research funding that it almost doesn't count.
Every big company up through about 1970-1980 had very large R&D departments that had been in existence for anywhere between half to a whole century.
Sure: IBM, Bell, and Xerox are remembered strongly. However, Kodak and Polaroid had big chemical engineering departments. RCA and Motorola were household names. Every big name in steel (US Steel, Bethlehem, etc.) all had lots of research in metallurgy. Corning was a huge driving force behind ceramic chemistry. Dupont similarly for plastics.
When those big companies went down, so, too, did their research departments and the funding driving them. This was a problem because companies funded a LOT of the local universities--Pittsburgh universities, for example, were still using equipment even in the 1990s that had been funded decades earlier by the big steel mills.
The collapse of the corporate giants meant that the percentage of academic research funding from government suddenly went way up.
Once you have too much scale for a few people to truly know what's going on, all that's left to keep people working on the right incentives is trust, and inevitably you'll have some bad actors at some point, and after that you put in some metrics and then it gets worse cause even good actors pursuing those metrics may have gone off the rails.
The only places I've seen avoid it moderately successfully have had exceptional people managers, like 1-in-100/1000 level, to minimize the appeal of playing the game instead of doing the work.
Even in a perfect system, at some scale people need these elementary quantities to be able to model and predict exogenous factors or justify certain other behaviors to operate.
Goodhart's law does tend to come into effect - but Goodhart's law may come to dominate these systems through far more than malfeasance. Just phone game-esque decay of information from one actor to the next can cause it, moreover this occurs across generations which I would posit accelerates the decay. One "generation" may have firsthand experience of the why and how, and upon relaying it to the second a loss occurs, and the third may not ever receive it or they're given some highly heuristicised version which artlessly excises nuance. Very quickly the system is reliant on decayed information which informs garbage processes, but this is ossified as the standard operating procedure.
I would posit that, especially in the temporal domain, this decay should prompt us to reconsider our tolerance for staple institutions and our willingness to facilitate their development, sustainment, and immortalization and thus limit scale while maintaining the integrity of vision and information while jointly allowing voids to open and be closed by legitimate and ostensibly better suited successors. Right now we're making models of cancer, many of which are metastatic.
Nice handle, by the way.
Somewhat related is Dunbar's number
Who's assessing the number of papers published by candidates, the number of citations those papers received, the impact factors of the journals in which the papers were published? Who's writing the recommendation/reference letters for the candidates?
The answer is other scientists, and greater weight is given to the opinion, assessment, letters of "leading scientists".
The "system" was set up over time by scientists, for scientists.
Blaming scientists for this is like blaming employees for poor working conditions because they signed a contract. It ignores the fact that they had no choice but to sign the contract, if they want a job.
In contrast, scientists talk about impact factors, citations, and journals all the time. A few years ago, my home country started this enabling system for tenure candidates. Who do you think proposed as criteria, for tenure, the year-weighted citations, the impact factor of the journals in which the research was published as a discriminating factor? Faculties. Who now say that the system is, for all intents and purposes, unchangeable.
This is a strong claim, which I think needs some references to support it.
Watching my dad's generation in academia from the 1950s forward and my own from the 1980s forward, my impression has been that this change in campus culture has been entirely driven by the administrative level.
EDIT: I guess you can omit "physical science", I suppose we are talking about research across all disciplines. My point remains.
Business people have no say in any hiring decisions made at universities. The hiring committee is made up of faculties at the department hiring the new assistant(often)/associate(rarely)/full(very rare) professor. Tenure is not decided by any business or administrative person working at universities. Same for grant funding in the sciences.
So, a lot of the benefit, but barely any of the accountability
We like metrics because they are not subjective. We don’t like metrics because they tend to be insufficiently specified optimization problems (I.e. we are looking at 3 variables when we really need to look at 1000+). We dance around this with “Goodhart’s law” type statements, but this is blatantly insufficient.
My hypothesis is the brain is obviously working with raw data - photons and sound pressure changes fundamentally and is actually solving the broader optimization problem- albeit subconsciously in the immensely powerful associative cortex. If we were to present the same problem to AI, it would end up being similarly subjective - influenced by its training and structure.
In summary, human intuition is currently the most powerful computational device. It should be objectively trusted over and above any gameable metric. Thus only non-gameable metrics should be accepted and intense scrutiny should be place on all metrics used.
One of my favorite (1993):
Wiles, with his from-childhood fascination with Fermat's Last Theorem, decided to undertake the challenge of proving the conjecture, at least to the extent needed for Frey's curve.[18]: 226 He dedicated all of his research time to this problem for over six years in near-total secrecy, covering up his efforts by releasing prior work in small segments as separate papers and confiding only in his wife.[18]: 229–230
This is all completely normal and reasonable. Except for the part where you are expected to estimate results of scientific research.
I think of it like speculative execution and pipelining.
The company is now winding down the project overall, but decided to keep and adapt this Rust client to its own backends. It is the only part which survived and it was the only thing I wasn’t given orders to build.
We need more trust and fewer metrics / games. The thing I really don’t want to pay for (as a tax payer) is the bureaucracy in the middle of all of this.
> We need more trust and fewer metrics / games.
Except, at scale when dealing with a lot of people, there are plenty of people who are willing to abuse that trust.
I made this mistake early on in my career as a manager. I gave an important piece of work to one of the very senior engineers who I managed. Over the course of a month, he repeatedly said "trust me" regarding the status of his project. When 6 weeks went by and he finally showed it, it was in such a state that it was essentially unusable. We ended up throwing the whole thing out. Furthermore, he left for another job about two months after that, leaving his work in a shitty state.
I made plenty of mistakes there, but I also learned a valuable lesson, "trust but verify". Until someone shows me concrete evidence of what they've done, I really have nothing to go by.
To be clear, I'd love to be able to "have more trust and fewer metrics". But at the end of the day, it's really hard to give people the freedom to, say, hole up for months at a time while they say "trust me".
That's why I get annoyed when people just lament about the problems with things like OKRs and KPIs, while totally disregarding the problems these things were designed to solve and pretending that if managers just had a more "trusting" mindset that a million Higgs and Wiles would be free to blossom.
If you can’t do this yourself, at some point you need to be able to trust (yes trust) someone who can. You could not formulate OKRs or KPIs that would have solved this problem.
The tough part is coming up with a business idea that would allow this type of lifestyle; that’s where I’m currently at. If I could make $130,000 per year, heck, even $65,000, and live in an inexpensive part of America then I’d be fine, but the trick is figuring out how to make that much money in only 20 hours instead of 40.
https://jameswphillips.substack.com/p/s-and-t-a-conversation...
1. Academia has fallen into a self consuming game where fierce individual competition to appear productive means that researchers don't have the space to actually think creatively and come up with incredible new insights.
2. Scientists like Higgs had the privilege of lower hanging fruit to harvest in their day, so a career of lesser intensity was sufficient for people in his cohort to produce incredible contributions. Modern scientists legitimately have to work harder to produce less profound results.
Maybe there's something to both lines of thinking.
I'm also reminded of the classic story of the pottery teacher who graded one class by final project quality and another by total weight produced over the class period, and found that the latter group ended up producing higher quality work. Higgs in this article seems to be of the persuasion that in his profession, the former style produces better results. I think many of us here would find that intuitively wrong in general.
Personally, I’d have loved to be a part of the Academia of the mid 20th century. But my exposure to research 10 years ago (thankfully, before committing to a PhD) quickly taught me that it’d be extremely unfulfilling, difficult, and poorly compensated compared to private industry. It would be cool to be a tenured professor I guess, but the coolness and prestige of that title is riding the coattails of the past when academia was less of a slog.
I get to work on novel CS problems (distributed systems) in my FAANG job without having to publish X papers per year while I progress on the actual meaty problems, for a lot more pay. It’s just strictly better IMO. I’d rather take 3 years from beginning to end to finish something new I’m actually proud of that’ll be used by hundred of millions of people, than be forced to publish papers very few people will read at a regular cadence or lose my job.
Additionally the “publish or perish” regime encourages small incremental research projects, rather than large expensive risky ones that might not produce a paper for many years.
All this against the backdrop that it is getting increasingly harder to find new discoveries since the low-hanging fruit is mostly harvested.
I've seen it happen a lot in both experimental and theoretical fields. Rather than publish one big paper that explains a new idea clearly at a high level + show its numerous applications, they publish many (6-10) small results based on the the "machinery" behind this idea. It can be frustrating as a beginner in such fields, like learning from a textbook that shows you just solutions to problems and doesn't explain the general theory.
If people get rated on that metric, they'll maximize it at the expense of the things that can't be measured.
But, there are tons of subtle ways to undermine metrics. Want to be the top salesperson each month? Give negative feedback to candidates for open positions that seem too strong.
Consumer facing web products value depends clearly on number of users.
If a metric cannot be tied to something material, don’t start.
The customers end up being short term relationships, the sales provide negative value to the company due to reputation loss (not realized for perhaps years), but the salesperson collects the commission and is probably working somewhere else when the smoke clears.
And I'm not sure if you're claiming "number of users" is ungameable. If so, I'd invite you to reconsider that - we've just lived through an entire era where companies gamed user metrics to harvest sweet, sweet VC dollars. Founders game the metrics for VCs, vice presidents game the metrics for founders, product managers game the metrics for VPs.
You could also reformulate the objective function to NPV.
So you modify your metric to try to eliminate the harmful kind. But now you're in the same race as other metrics.
You are stating that cash / sales do not have exactly 1.0 but can be something less. I can accept that. However, the correlation is very high and higher correlation factors are harder to game. Metrics with high correlation factors also do not need to be hidden.
Let's contrast this with another metric: number of emails sent per month. Low correlation factor, easy to game, likely to actually cause harm to a business. This is the kind of metric that you would certainly need to invoke Goodhart's law on (i.e. hide) which is, in turn, a sign that a metric should not be used.
My main thesis is metrics must have high NPV correlation factors or should not used at all. High levels of scrutiny should be placed on those introducing such metrics. Finally, no metric should ever be hidden as it is a clear sign that NPV correlation is low by definition.
If you make the metric "cash over the past 20 years" that would be less able to be gamed, but do you really want to wait 2 decades between each performance review for your sales team?
There's also the problem of one person gaming the cash metric to the detriment of others. Like a used car salesman who lets other salesmen "warm up" the customer but then swoops in to be the one to actually make the sale. Is he really 10x better then the other sales staff or is he jut a leech claiming other people's sales effort as his own?
And you don't want 50 different sales people all trying to sell to the same client, undermining each other's efforts. That means you need some scheme to assign people to opportunities, such as sales territories. They will learn to game that system.
When doing any form of art, the most important thing is to practice. If you make 20 vases, the 20th is most likely far better than the 1st. Same with painting, drawing, other forms of sculpture... nothing teaches like practice. There's such thing as more complicated pieces that require more work, but the majority of actual skill at pottery is practiced the same way whether you are aiming for a relatively complex piece, or one that is just hard enough it doesn't bore you.
On the other hand, in science, there's no such thing as random practice making you better. The more papers you publish, the higher percentage of the time is dedicated to the writing bits, and getting through the approvals, and doing a million submissions, instead of coming up with good hypothesis, and finding something to do. Aiming at a hard problem is going to lead you to publishing less, not more, so the things you'll work on are going to be minor improvements, nothing difficult. Therefore, the chances that you'll discover anything major definitely decrease the more often you publish.
So if in your intuition you'd be better off focusing on making a lot of unimportant papers... why would anyone aim for any fundamental problem, which has a higher chance of failure? You think the efforts to actually publish and submit, all away from the lab aren't mostly wasteful? Because it's really hard for me to imagine how publishing more doesn't mean far less time doing research.
Many interesting papers nowadays come from commercial R&D departments, like MS Research, because unimportant papers probably don't move your main KPIs (which would be "a lot of patents" AFAIU), and only interesting results improve the image of the company and serve as good marketing material.
Wait, how is pottery equivalent to significant expansion of the bounds of scientific knowledge? Humans mastered the contours of pottery thousands of years before farming was a thing, we have no such bearing wrt scientific knowledge.
Not even a hundred dollars in reagents. Four pages. A hand drawn plot and a linear regression. Six references.
I like that guy.
Well, there you have it.
https://www.scientificamerican.com/article/how-the-higgs-bos...
Peter Higgs: I wouldn't be productive enough for today's academic system (2013) - https://news.ycombinator.com/item?id=35303887 - March 2023 (1 comment)
Peter Higgs: I wouldn't be productive enough for today's academic system (2013) - https://news.ycombinator.com/item?id=12717553 - Oct 2016 (141 comments)
Peter Higgs: I wouldn't be productive enough for today's academic system - https://news.ycombinator.com/item?id=6864539 - Dec 2013 (194 comments)
He retired in 1996.
Pursuing in academia is not smart at the individual lebel, that indeed expels a lot of talented individuals.
I had doubts that my field was actually ever going to solve the problem it was supposed to, and brought up the issue with one of my supervisors.
He took my analysis one step further and explained how the (very very slow) progress we appeared to be making was basically just sampling bias, but it didn't matter because the point of being in the a 3 year PhD program wasn't too discover anything but to go through the motions.
Nope!
He also talks about "the lunatic right of the Conservative party" trying to withdraw from Europe with reference to a referendum on Scottish independence. Would be curious his views today...
To play devils advocate, it’s hard to see a way of keeping the system the same when your intake and thus the intake for postgrad courses is so much higher.
Guess it's good he got the Nobel then. ;-)
In the CS world I find it interesting that innovative, high-benefit, systems development usually isn't enough to get a Ph.D. or tenure, but might be good enough to get a Turing award. I also think of Tim Berners-Lee, who as I understand it completed an undergraduate degree in physics.
A smart person (whether in academia or industry or in society) is one that makes the system work for oneself (= hack it).
However silly academic systems will be designed in the future, thankfully there will always people like Wiles or Feynman that go for the hard problems no matter what, whilst managing to be at least tolerated by "the system".
So, you are taking the set of a dozen or half of people good and enough on their research line and privileged enough to possibly make an important impact, and filtering it by requiring that they have that completely unrelated competency of hacking a social system so it turns from forbidding their work into allowing it.
How selective is that new filter? Because if it's anything like 10%, you'll probably end-up with no one at all.
I suspect todays short-term, glamor-based metrics improve the median quality of work, but decrease the amount of top quality work being produced.
The absent minded professor stereotype is dying, but those are the exact types that revolutionize fields.
This is absolutely not what is happening, but perhaps this was irony? We are granting money to very competent people. They are true experts in their fields and have studied for years. And all this knowledge allows them to expertly optimize their trajectory in today's academic system much better than if they were simply innovative scientists. People who revolutionize science are not only 'competent' or the 'best experts'. First and foremost, they're inventive and quite often very lucky. Qualities we don't know how to select for.
Peer replication is useful.