Hacker News Reader

Comment by j7ake | Hacker News Reader

As long as there is some quantitative criteria on which jobs and promotions depend, there will be people gaming the system.

One solution is to couple this quantitative criteria with independent committees that assess people beyond the metrics, but that requires a lot of human effort and not scalable.

Assessing people in ways that don’t scale seem to be the way to avoid this gaming trap in academia.

View on HN

I'd argue that it's not just that the metrics don't scale but the problem is that we are trying to find quantitative metrics for something that can't be easily quantified. The worst outcome is not even the forgeries and fakes as in this article, but more that even the vast majority of ethical academics are being pushed into a direction that is detrimental to longterm scientific progress, in particular short term outcomes instead of longterm progress.

even the vast majority of ethical academics are being pushed into a direction that is detrimental to longterm scientific progress, in particular short term outcomes instead of longterm progress

I agree. The egregious fraud is just the high-sigma wing. It's a symptom, but the real problem is how it affects the majority.

And non-quantitive evaluations are prone to favoritism and prejudice. AKA people gaming the system.

I doubt there's an easy answer.

Trying to better align the short term objectives with longer term ones could help, but that just makes it harder to game, doesn't eliminate it.

It’s why one needs both. You need both undeniable productivity by quantitative metrics, as well as glowing reviews from independent panels that are not influenced by favoritism (almost like an audit).

Interesting point; it is much like the problems of trying to assess programmer productivity.

I was thinking it's much like Google trying to deal with SEO. Most people optimize for high google ranking, not quality content.

Google periodically changes the evaluation, in theory to reward good content and penalize bad, but people still try to game the system, rather than improving content.

Reply on news.ycombinator.com