Hacker News Reader

Comment by bee_rider | Hacker News Reader

The scientific record is not a load-bearing structure. It survived because it was something that only academics cared about. Now that publishing is being used as a source to pull metrics from and The Science* is being used a source of political arguments, we’re in a really unstable and dangerous position.

* I want to be clear that I’m very much in favor of science-based policy, but giving The Science credit and blame is not a tenable path forward, because science is hard and there will be mistakes. The way to do it is to have public figures analyze where the field is and stake their professional reputations on interpreting it correctly. Individual professional reputation is the load bearing structure of a serious society.

View on HN

I don't think that's the problem. The problem is "publish or perish" and the fact that survival and job advancement depend on quantity over quality of publications. That drives a huge demand to be published, and the purpose of getting published is not to advance science but to get more bullet points on an a C.V.

The fact that we pull metrics (for hiring) from the scientific record causes the publish-or-perish problem, so I’m not sure I see a disagreement here.

You are reversing cause and effect. You "perish" by being unable to procure grants. Grants require a publishing record. Why? Because estimating success is harder than simply looking up a list. The entire system works that way. The hiring procedures are simply following this system.

I agree. One thing that is pretty depressing to anyone that spends time in grad school or in academia is that universities and colleges put barely any importance on a professor/scholar's teaching ability. Education is simply not the main focus at all. Instead, they view scholars as research machines. Rather than giving research and scholarly inquiry the time and patience it requires (a lot), they would have published research occur like clockwork and in many cases regardless of the quality of that published research. This looks better for the school; but leads to middling research at best and often shady stuff by harried researchers, and a subpar teaching experience for most students who are always second fiddle to the endless demands to publish. The school benefits (per se, they don't like getting caught in scandals of their own making, of course), while researchers and students get subpar experiences all around.

Exactly why I left academia with my Master's and didn't pursue a PhD. A few years into my professional life I met a PhD who had actually gotten tenure and gave it up to come work in industry. When I asked him whether he missed academia he said, "I miss the idea of it, but the idea isn't what it is."

>"publish or perish" and the fact that survival and job advancement depend on quantity over quality of publications.

I mean, kind of but not really.

Any academic of even marginal skill can smell a padded CV from a mile away. The type that do this are not serious people and will waste your time and resources. Knowing how to identify an imposter is a survival trait.

Usually the type to pad a CV with garbage are eyeing admin or corporate gigs, not academic success. A search committee or tenure committee wouldn't look too kindly at a bunch of wacky papers in the east austrian journal of ass-scratch.

Yes there is huge demand to be published, but quality matters a lot if you intend to seriously participate in the scientific community.

> Any academic of even marginal skill can smell a padded CV from a mile away

A couple drops of "Eau de Népotisme" and the smell goes away miraculously. Torturing your job requisites so only the desired local candidate can apply is a way to "solve" it.

In my experience this happens very rarely. In most places that I'm aware of, there is a very strong aversion to hiring local candidates for tenured (or tenure track) positions.

For example, in Germany it is extremely rare to become a professor at the. University where you did your phd/postdoc. It's almost always seen a something not quite right.

Now there is still lots of politics involved in the hiring process. Tenured positions are rare and often the other academics on the committee want to candidates which strengthen their own research area (although interestingly if there is a colleague down the corridor or someone at a university half way around the world who can do some technique you require, collaboration is much more likely to happen with the external person)

Is inversely proportional to the scarcity of the resources available

Let’s take the perfect paper as an example. This “impossible” paper is the product of Science Done Right. The authors previously registered their hypotheses and methods. The data are collected in the most rigorous fashion. The analyses and the data are all available to the public, and review shows that everything is in fact perfect. The conclusions of the paper are supremely consequential.

Does rational policy immediately change based on the conclusions of this paper? I argue no! Maybe discussions start immediately, confirmation is sought, probably many additional studies with the express intent of confirming the result of the first paper are launched. However, there is no such thing as one paper that needs zero additional support from the community.

I'm not sure what your point is, but I think it's "I don't like Real Science, because it doesn't immediately lead to the outcomes I like."

As for "additional support from the community" that's not the solution -- it's the problem. Almost by definition, "the community" isn't qualified to support the paper; only to advocate for their preferred actions.

You’re off base here. No single study should ever be used as a basis for policy. Real science is a slow deliberative process that incrementally arrives at the truth. Input from the broader community in the form of confirmatory studies and stringent fact checking is very much part of the process, especially in complex fields like biology and psychology.

Biology and psychology should not be in the same sentence. Only one is a real science.

If you meant "scientific community" then I have no issues. I was reacting to all the "The Science is settled!" articles during the pandemic by people from outside it.

>Only one is a real science.

Human psychology is a messy business, but when research is conducted earnestly and rigorously according to the scientific method then it is science by definition.

You can find a discipline problematic and unworthy of your interest without actively gatekeeping it, by the way.

> You can find a discipline problematic and unworthy of your interest without actively gatekeeping it, by the way

Such patronizing. Unworthy of a real scientist.

When their "experiments" generally don't replicate, then it's not a science. Maybe similar to Colbert's word "truthy," what psychologists do is sciencey.

Ha. You have far too high opinion of the field of biology. Go look at the current Dana Farber mess.

I think you misunderstood the parent comment. The first part reads as if you're replying to a different comment, I don't see how you could come to that conclusion based on what was said.

For the second part, they were talking about the scientific community, supporting the paper by reproducing results. Not regular Joes saying "I support this paper" or whatever, if that's what you were thinking.

OK. The word "community" is ambiguous here. You read it as "scientific community" which perhaps is what was intended.

"However, there is no such thing as one paper that needs zero additional support from the community."

The fact your argument is even controversial is indicative of how science has become politicized.

I agree. Good public policy can never be fully rational or science based. There is always a necessary element of emotion, intuition, and forecasting that goes beyond science.

You’ve almost nailed it!

“It survived because it was something that only academics cared about”

The critical change has been in the number of academics, not in their level of care. It was always a small exclusive group. It can only work if it’s a small exclusive group. We can’t make everyone an academic and expect quality output anymore than we can make everyone a coder and expect great software. Academia (higher ed) simply cannot scale and still be what it’s been. Great wine must be produced in small batches, perhaps the elite should be too.

You're implying that bad politics is the product of bad minds but that's not the case at all.

Yes, this sounds accurate to me, because it follows the general pattern that "enforcement by reputation doesn't scale". It works in small groups, because the feedback loop is relatively quick and relatively difficult to evade. The larger the group, the slower the consequences and the easier it is to find people in the group who don't know your reputation yet.

The "solution" which occurs almost immediately to any developer is to automate the reputation, but I think we've seen enough SEO and similar algorithm-hacking to understand how that fails. In fact, "rack up a lot of publications" is itself a method of hacking an attempt at automating reputation.

This is a fair point and it's a valid criticism of scientism which blurs the line between a sort of platonic idea of "science" as an unencumbered pursuit of ultimate truth and "science" as a human endeavour which is subject to current zeitgeist, social norms, economics (funding) and everything else.

The Science has been used for political arguments at least since the founding of the EPA, which occurred over 50 years ago. The Science was used to ban CFCs 30 years ago.

So why now?

Having spent some time in academia I can tell you the answer - it's the cheating. The rise of science in developing and recently-developed countries where corruption is still endemic has led to there being a large cohort of students and "scientists' who think it's ok to cheat. This is a cultural thing - they're not bad people, and often not even bad scientists - they're just used to a society that operates like this. Don't take my word for it - look at the article and we see the following:

> Jack Ben, of a firm whose Chinese name translates to Olive Academic

> researcher and assistant professor in Saudi Arabia, Malaysia, and Jordan.

> the company acts as a broker, sharing payments from the paper mills with multiple editors—including Omar Cheikhrouhou of Taif University in Saudi Arabia and the University of Sfax in Tunisia.

> A Ukrainian paper mill dubbed Tanu.pro

> While he was visiting his parents in India last summer, a Dr. Sarath of iTrilon reached out to him on WhatsApp, offering authorship of “readymade papers” with “100% Acceptance Guarantee.”

> in Russia and several ex-Soviet countries, for example, policies focused on publication metrics, coupled with a culture of corruption and the transition to market economy

This culture is slowly corrupting scientific journaling itself as an undertaking, firstly because established journals will end up employing some of these people (as in the article), but more importantly because as mentioned, it's a culture problem and who doesn't want easy money and guaranteed quid-pro-quo publication when all your friends are doing it? After all, you're not a bad person and I'm sure your science is fine.

There's a subtle circularity problem with your proposal in that "professional reputations" are largely based on those same scientific metrics, which we know are gamed. The ability of even other scientists outside a field to properly evaluate those within a field is very, very low. And then when you get to the general public, it's just going to be a popularity contest fueled by implicit biases.

Reputation is always circular. The ones who are supposed to be reputable evaluate whether someone else is reputable as well.

But it's not based on metrics. Those are only used as an initial filter. When it comes to important decisions, the reputable parts of the academia rely on peer review. Shortlisted candidates – if not everyone - are evaluated by people who are expected to understand the research itself. Final evaluations are based on subjective judgment rather than objective metrics. It's not a perfect system, but it works reasonably well as long as most of the reputable experts deserve their reputation.

What you're saying makes the issue sound even worse.

> It's not a perfect system, but it works reasonably well as long as most of the reputable experts deserve their reputation.

I don't understand why you'd think this is a reasonable expectation. There's plenty of public examples where assumed competency turns out to be a costly, or even fatal mistake. That's just what's documented. Underneath, especially in academia, there's many examples of grifters in high positions. Every grad student (that is 100%) I've talked to has a story about this.

That's just how human institutions are. There are always bad actors, but as long as there are not too many of them, the honest majority keeps the system working. Meanwhile the bad actors have unfair advantages.

There is a good side to the academia, but you are less likely to experience it if you get into politics. Caring about frauds, grifters, and plagiarism is politics. You should make a conscious choice: do you care more about science as an ideal or about your own research. If you focus on your own research, you can seek like-minded people and ignore those who pursue status.

Of course, the academia is a risky career choice, and success is far from guaranteed. That's why it's important to know when to let go. You should learn to estimate what kind of career success you can expect, without wasting too many years in the process. Once you know that, you can determine if you would be content with that for the rest of your career, and make the choices accordingly.

Philosophy aside, the point is there's no mechanism to distinguish grifters from non-grifters in academia. The proposal "staking professional reputation" requires reputation to be an accurate measurement of competency.

I think my point was that individual performance does not matter. It doesn't matter if someone is competent or a grifter. It doesn't matter if someone achieves undeserved success. The academia is not here to produce short-term value. You don't need measurements that supposedly correlate with it. Measurements encourage gaming them, and gameable systems attract more grifters.

What matters is a culture of honesty. If a part of the academia has it, it probably produces long-term value. Unfortunately a culture is a nebulous idea. You can't measure it, and you can't really evaluate it from the outside. You may see it after the fact, if some people produced good science, because they lived in a culture that encouraged it.

Professional reputation is a signal for insiders. It can be used as a proxy for determining if someone is worth working with and if they are a good cultural fit. It's less useful for administrators, but they are not the ones doing science.

No one who uses The Science in political arguments cares to refer to specific published papers.

Reply on news.ycombinator.com