Hacker News Reader

Comment by ProjectArcturis | Hacker News Reader

Just want to add, it's generally not a conscious choice. Everyone wants to do something that is both a genius experiment and gets published in Nature. The Science Game gets propagated in how scientists conceive and plan their work. Everything must be done in the paradigm of publications and grants. You can't do anything if it doesn't have at least interim results within a year or two. It's an unfathomable career risk to try something absolutely brand new. There's a good chance it fails, at which point you have a 2 year blank spot in your CV and you can never get back in the top 10% of scientists where you need to be to get R01 funding and run your own lab.

View on HN

One problem is the incentives are biased toward “ok” papers.

h-index should be replaced with a more competitive index (e.g., Max over citations). The publication hyper-competitiveness is due to publication spam, which boosts citation and publication count. However, twisting a knob and getting slightly different results should be viewed as a failed project.

> The publication hyper-competitiveness is due to publication spam, which boosts citation and publication count.

The publication hyper-competitiveness is a direct result of grant hyper-competitiveness. Funding rates at top US agencies are around 10-20%. Grants are how researchers fund their research, pay their students, pay themselves, and earn tenure, so there's a lot riding on getting them.

I see a lot of talk about the problems with publications on HN and possible fixes to the publication process (like requiring code for example, or as you suggest changing the important metrics), but publications really are just a proxy (and prerequisite) for the grant application process. You can change around the publication process all you want, but nothing will improve unless you address the issue of grants, because that's where the money is.

But the question is, can you make grants more dependent on actual scientific breakthroughs rather than publication spam?

Right. Listing 10 papers with “ok” impact should be viewed as 10 “bad” papers and not 10 “good” papers.

It’s extremely unlikely that a 50 page CV is demonstrating real value throughout. It’s the academic equivalent of putting 20 Javascript frameworks on your Resume because you used them one time.

I agree that h-index isn't very indicative, but max citations seems like a poor alternative.

There is a relation between citations and certain aspects of research quality, but citation count can be easily gamed, and is affected by many factors other than the quality of the work itself.

Trying to use single measures to quantiy something as complex seems flawed to begin with.

Top N papers is already a metric used for professorship slots, so Max or other functions are me simplifying the situation.

I agree though that citations are proxies for goodness and are ultimately poor indications of work. For example, a paper closing the field would have no follow-up work, but a provocative paper would have plenty (regardless of quality).

I think a quantitative metric is necessary because many decisions eventually hit a bean-counter (e.g., admin type), who is not able to assess research quality in the time they have. But even with experts reviewing a set of candidates on research qualifications, the panel would be unable to objectively provide a full ranking of candidates.

A perhaps better way to do it would be to perform proper accounting on research output which stabilizes across a 10-year timeline or so. Each researcher is a “stock” and the stock-price reflects current and projected “revenue”. If someone publishes only papers that go nowhere, their stock tanks. At top institutions, tenure kind of does this (only one time).

Exactly.

> When a measure becomes a target, it ceases to be a good measure. - Goodhart's Law

> The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor. - Campbell's Law

You want to use metrics as a very broad breadth first search to help cull the search space and use trust systems as a depth first search. But once you have found signal through trust, you should completely ignore that first search and even look more into things that you had previously excluded.

Find researchers that you agree with, find the papers that they recommend, read those papers and check that you agree with them. If they recommend something from a no-name university, look at those first.

Unfortunately if this is not your field and you aren't able to determine quality, this becomes impossible. If it's important to you, you need to learn it. This is why I don't like non-technical managers. If the people who approve the grants do not understand the result, this is inevitable. It might work early on when trust still lingers, but as metrics take over the social systems always fall apart.

Reply on news.ycombinator.com