Comment by timkam | Hacker News Reader

Comment by timkam | Hacker News Reader

Parent Full thread

I largely agree, but we do not write all these papers "so tax payers can sleep". We write these papers because the administration incentivizes us to get the highest profile publications with the lowest effort possible. In relatively creative fields (i.e., much of computer science), this can even mean to make friends in the community and co-author papers with them without being the one who does any of the hard work. (In other sciences, it means being the one who pulls in the money, as far as I understand.)

A related problem I see is that we treat all papers the same: position (opinion) papers, reviews & surveys, and "hard" results (relevant proofs, strong empirical results, software artifacts that are deployed at scale in practice etc.). As the blog post somewhat suggests, the administration should primarily ask us to summarize these results when assessing whether we are worthy of funding and faculty positions; sure, it takes more effort, but I think it is possible.

Exactly, the big issue is that paper counts have become the dominant metric. And like with all structures dominated by a single metric, it's easy to game the system.

Most professors I have worked with in experimental sciences are completely disconnected from actual research. They just optimize for hiring hordes of people to churn out tons of mediocre papers, which they don't care about. I came from theoretical CS & math, so this was really shocking.

It's really depressing and it needs to change. Academia is now in a phase similar to that of an innovative startup which has been filled in with many middle managers, and has turned into a corporate monstrosity.

"Paper count" is about as reliable a metric as "lines of code". Publish or die is a toxic mentality in academia and you are certainly not the first to have observed this, sadly.

It's pretty outdated. In research schools you need to win funding or die nowadays.

And while publications do help you, say in getting hired and promotions, your publications are dug into pretty deeply. They don't want to be tricked into hiring someone useless who can crank out fluff but can't win a grant.

This is completely true -- something that many outside of academics don't understand -- but I'm not sure it really makes the situation any better. In a lot of ways it just kicks the problems down the road.

By whatever metric you use, it's really about the checklist rather than substance. And the checklists are based on stereotypes, and get gamed.

> This is completely true -- something that many outside of academics don't understand -- but I'm not sure it really makes the situation any better.

Indeed in some ways it's worse. When the money is insufficient, the established researchers get it first. Then everyone else has to increasingly work for established researchers as postdocs and similar positions... helping to yet further boost the advantage of the established researchers. It went from senior researchers versus junior researchers, to senior researchers with a herd of postdocs working for them, versus junior researchers.

At least if it was based on publications, everyone gets co-authorship and in fact the first author counts most. So the postdoc and the PI gain the same new items on their CV's for their collaboration. For grants, credit goes entirely to the PI.

Maybe, software solutions can be applied to academia.

In software, we have "source code distributions", such as Debian, Fedora, Gentoo, Arch, Red Hat, SUSE, etc. They are collections of mostly a same free software, with minor differences, but with different policies and purpose. Some are conservative collections of proven software, others are bleeding edge, etc.

IMHO, someone should start to collect papers, which are worth reading, and keep this collection up to date, with fixes to papers (AKA as patches in software), with one paper per topic, with ready to use math models and simulations, with in software tests for models, etc.

It's a nice idea but there are practical problems. The first is that it supposes the scientific publication is a rational and concerted process. It is not, here is an example:

I am interested in Amyotrophic Lateral Disease and I blog about research done on ALS [0].

However there are 15,000 scientific papers published each year on this topic, on average only 20 to 30 articles per year are worth spending the time to read them (one every 500) and a milestone appears only every 3/4 years (TDP-43 in 2006, C9orf72 in 2011, etc).

After 100 years there is still no clue about ALS etiology. Scientists have strong but widely different opinions such as for some it starts in the brain motor area, for other it starts at the peripheral nerve junction with muscles, for other (new trend) it starts in the gut.

And scientists happily pursue hypothesis that their colleagues are supposed to have disproved. Sometimes the same lab (I have one in mind which is perhaps the best known in ALS research) publishes articles contradicting its previous research without discussing the issue.

How to rationally blog about this domain, how to make collections that are worth reading?

[0] https://padiracinnovation.org/News/tag/als

How is this different from compsci? :)

Every year there are 15 gazillion modules published to NPM, on average only 20-30 are worth spending the time to read, and only a handful milestones worth including in said distributions.

After 100 years there is still no clue weather microservices or monoliths are the correct hypothesis and some times the same lab publish articles contradicting previous research.

Thanks for the good laugh!

In free software distribution, we have "leaders" and "maintainers". Maintainers are volunteers which are keeping up to date just few software packages. Leaders are choosing a general direction of a distro.

IMHO, something similar can be applied to science. Leaders can chose a general direction of a collection of papers, while maintainers will work on small areas, keeping them consistent and up to date with new research.

For example:

- LIGO/VIRGO discovery of gravitational waves means that the speed of light in vacuum is not constant. It's constant in "steady" vacuum only.

- Discovery of Higgs boson means that vacuum is not an empty space, because Higgs field is present everywhere.

These recent discoveries significantly changes the field of science, so a bunch of classic paper must be updated or discarded in favor of new ones.

A huge thanks, indeed this makes sense.

That sounds like Wikipedia!

Wikiscience, maybe?

I'm not sure if those two fields are comparable. In software engineering, at least there's usually a dollar value impact for the code you write. In research, the impact isn't as tangible and probably won't be known for years.

> In research, the impact isn't as tangible and probably won't be known for years.

This still describes a lot of software engineering.

Its TPS reports. That what it is. And until academia gets their own Bobs to ask academia's Lumbergh, "Yeah, Academia, lemme ask yah... real quick question here... how much time wouldja say yah spend each week dealing with these papers?"

Maybe you didn't mean it, but 'gaming the system' sounds bad. While I agree the outcome is bad, I think when your boss tells you to publish 10 paper per year and you do that, that this cannot be called 'gaming the system'

> It's really depressing and it needs to change.

It has been like that since ten years and nothing has changed since then. People find out just about the same as you did but nothing changes. It such a convenient metric it will stay for much longer.

Reviews are especially bad. You can tell which people work at schools where citations count in their pay (and reviews are allowed in citation counts) by the ratio of reviews to original research papers they write. Some schools/govts really need to wise up.

I think of it as a kind of predatory or scavenger behavior. They basically steal the citations from the papers they are reviewing in return for a small effort in helping us with information retrieval. We might as well just cite google and elsevier. Ideally, citations should pass through them somehow and get applied to the original sources who did the hard work.

It's true that some review papers are shallow but it's pretty cynical to paint all contributions of this type with the same brush. In my experience, good review papers are more than summaries. They can connect separate lines of research for example, compare and contrast different ideas and they can re-contextaulise a body of research to give a broader picture and reveal new and interesting directions. These papers are hard to write well and take a long time. Even more so in cases where empirical work has been undertaken so as to make direct comparisons and provide reference implementations.

Even the summary papers can be useful. On particularly active problems there can be dozens of papers per year. Sorting out wheat from chaff by highlighting notable works, and pointing out trends in published research, is again a genuine contribution, and helpful for the scientific community.

Of course they are useful, as is google scholar and the search field in various journals. Or some blog that tells you whats was new in the recent conference. That doesn't mean they deserve to be cited over the real sources. They are cited as a crutch: "go here for a more detailed list of sources". The little meta-analysis they do doesn't explain most of the citations they get.

Hard disagree. We need to incentivized more curation, distillation, summarization, comparing/contrasting, systematizing, categorizing approaches along various axes, meta analyses etc., not less.

These are often very valuable, more so than yet another 1% improvement paper that never gets reproduced.

Review papers don't steal citations. These are cited from more distant literature so readers can familiarize themselves with the topic. Those wouldn't cite all the papers mentioned in the review individually.

But as a broader point, yeah, maybe something like PageRank could be used to "pass on" the citation Ina sense.

> These are often very valuable, more so than yet another 1% improvement paper that never gets reproduced.

It's not an either-or.

And one should always read and accurately describe the contents of what you cite, meaning of course you wouldn't cite everything in a review paper directly. Only that which is relevant.

Suppose we count both separately, research citations and review citations, the latter for your great review papers or even if people just want to reference the lit review section of your research paper. What would happen to the incentive? I suggest that reviews would be less incentivized as people would primarily be concerned with rating researchers according to their research citations. Just like no one cares about your textbook sales, popular though it may be for introductory uses. This would imply reviews were not "honest" citations but are done to hack the metric.

People should care about textbooks. I think it's misguided to think that real scientific value lies only in original research.

Well you guys are pulling me to an extreme position here. Of course I love good textbooks too. And review journals like signal processing magazine are my favorites. I'm sure many people do them for great reasons too. Let's not forget that academics teach too, and there are academics and colleges that are entirely devoted to teaching, not research. People are free to prefer them more if they like. However, when it comes to research metrics, the person who did the research deserves the credit for it. Even if we subtract the citations for obvious review papers (which is commonly done) there is still the lost citations by the original researchers.

Reviews also cause other devious problems, like journals hacking their own impact factors. And don't forget the Matthew principle (aka rich get richer) whereby only big-shot researchers can get invited to have their reviews published in high-impact journals, warping the network effects to their favor even more.

Regarding your second paragraph, I don't think this is the case in the UK at least, where academics are judged according to the REF criteria.

I’ve heard the term “minimum knowledge increment” used as a measuring unit for papers. Dice whatever research you have into papers that contribute the minimum necessary for publication. Dicing may only produce 1 paper, good enough yet meager.

Reply on news.ycombinator.com