You Want to See My Data? I Thought We Were Friends

That's part of the problem. Publications are seen as achievements. If you got accepted to a prestigious journal or conference, you can list this on your CV as an impressive "award"-like thing. A publication list is not just a list of "Look, this is the kind of stuff I've been working on, have a great read at it", but "Look, my research is so great it got accepted to all these fancy places!".

Publications are therefore unfortunately not merely about sharing new info with the research community but an award show. Ideally a publication would be the start of the conversation: "this is what we found, this is the method we propose, what do you think of it, community? Will you pick it up?" The test is then whether the ideas get adopted. But that's harder to measure. Citations try to approximate it, but it's a very crude approximation. A citation, as such, may mean tons of different things: e.g. a) a deep critique (Negative impact) b) being cited as part of a long block of "these other works exist, too" (Low impact) c) another work substantially based on the deep ideas of the original paper (High impact), d) being listed in a table for comparison, ala "we beat this other method", but no other discussion of the original paper (Low impact), etc.

If a publication was nothing more than a "hey, look, this is interesting", then I'd say, publishing mostly novel sexy results would be fine! After all, the surprising cases are those that teach us the most. However, as I said earlier, a paper is not only about "hey, this is interesting", but a "hey I want to advance my career", too. And in a twisted way of logic, I can agree that therefore we could put a band-aid over some of the problem by publishing (rewarding) systematic work with negative or boring results. But ultimately, this goes against the original purpose of papers, that is alerting the scientific community to potentially new information that we haven't known about before.

----

Ideally, to assess someone's scientific career, there would be at least one smart, attentive, impartial expert taking their time reading through the publications, taking notes, pondering, digesting it all, consulting other experts etc. However, this is too subjective.

Quantitative metrics seem superficially more objective and therefore egalitarian. The original idea is probably that if we just based everything on subjective judgement of scientific importance instead of publication count, there would be even more networking and friendship-based quid-pro-quo back scratching.

But everyone is overworked, and those who aren't, want to keep it that way. So nobody wants to put in the effort to actually interact with the deep content of research. It's too complicated and too opaque.

----

The problem is, flashy results are by their nature more attention-grabbing on all levels. It's not just some small perverse incentive. This is how all of us work, this is how history works, how everything works. The winner takes all, the rich get richer etc. We remember the Einsteins of history, those who just worked systematically and didn't find much aren't heroes. And if that's our bar, then people will do everything to look like they clear it. In any system, scientists would have to hype up their impact, it doesn't matter who makes the decisions.

Currently, universities want to employ researchers who will make a visible impact. Because that means attracting funding, but also attracting bright people from all around. Career-conscious researchers want to go to universities that help them market themselves well (good PR departments etc). PR is not only for laypeople as the audience, there is such a flood of research nowadays that even the experts of a small niche cannot keep up with everything happening.

----

The root of it is human nature, competition, deception, cliques, hierarchies. But the new about it is the scale of it, and the accompanying mechanization of it all. The idea that you can mass-manufacture innovation. That you can expect thousands upon thousands of researchers to make regular breakthroughs and, to say my field as an example, publish tens of thousands of novel AI-related ideas every year. It's related to credential inflation, and fake signaling: people with good academic track records got the good jobs and the respect, so people try to emulate that. Everyone tries to become the 1%, the rock star. And everyone wants to hire the 1%. So just like an evolutionary pressure, people try to appear like the successful. Soon enough the old signal doesn't work anymore. It used to be a high mark of educational level to have passed high school. Today that's a bare minimum. College used to be a meaningful differentiator. Now more than half of young people are "college educated" in developed countries. The next step is about becoming "researchers". Nowadays, having some publications is not a big differentiator. We see this also in title inflation like monkeying around in Excel is "data science" and "AI".

It's not just a monkey's paw. There is no central figure orchestrating it, asking the monkey's paw for more papers. It's a distributed system of agents acting in their self-interest. Nobody wants papers for papers' sake, they want to make defensible, justifiable decisions that will not get them fired and will pass satisfaction up opposite the path where the money is flowing, all the way to the CEOs, politicians and taxpayers.

During a conversation with an academic researcher (non-Computer Science) friend, when I brought up the topic of data sharing, especially in context of the infamous "replication crisis", they have their reasons not sharing. I'm loosely paraphrasing here, while trying hard not to misrepresent/misremember their exact views:

"I want to protect my data; I don't have enough time to present my data in a presentable form; and more importantly, they'll just steal my idea and go present it as theirs—and I might lose funding" ... and so on.

I can empathize with the academic pressure of "publish or perish". And not least of all, "need some food on the table, and roof over my head".

But I still wonder, there must be other effective ways to gently persuade a said researcher (especially in the 'soft sciences'—I'm not using the term derogatorily) on the importance of sharing data that allows reproducibility of a given experiment?

The idea of evaluating experiment design and committing to publish before seeing any data/results also strikes me as deeply flawed. Graduate students can sit around think up many deep questions and draw up tons of beautiful experimental designs. That is the easier part. The harder part is actually running the experiments properly and interpreting the findings correctly. If we are publishing experiment ideas, every graduate student would be submitting 100s of ideas to top tier journals. However, most of those experiments would end up producing inconclusive or utterly garbage results, which would provide little insight if published.

The other issue I see is that many interesting findings come out of serendipitous results that are not related to the initial experiment or hypothesis. You start out trying to answer one question and then stumble upon some else that is very interesting (so you then explore that). If you have already submitted one experimental design and are supposed to publish on that, what do you do if you stumble upon something else that is actually much more interesting? In my experience with research you don't often go directly from one question/hypothesis/method to an answer and then publish. It's much less linear than that with many dead ends along the way. It's just not something you can pre-publish.

All of that said, one area where I think we would agree is on publishing the results of well performed experiments with null results. I think it would be useful to the community if people published reports saying "we had this interesting question, did a proper experiment, and it turns out the hypothesis was null and there was nothing interesting there." In the current system, that will likely not get published. But, other researchers would benefit from seeing that result because they may have the same hypothesis and may waste time trying to explore the same dead end. A Journal of Null Results could be useful here. However, you do run into the issue of not knowing if an experiment failed because of a researcher mistake or error...

The reality is a lot of time spent by scientists is actually defining the problem properly (even after you think you’ve defined the problem properly), and preregistration studies would hinder the flexibility to adapt and change the direction given new understanding.

Problem is I think that the competent are few, and when the cultural norm is that they must be humble then they stand no chance against the many incompetent.

IM(H)O: Science shouldn’t be humble in the face of non-science. As long as it is it will lose. The idea of conflict free great science is a pipe dream. We need a culture that accepts (intellectual) conflict.

I kind of agree, but I will state it somewhat differently. Note my experience is in physics and healthcare, so may not apply for all fields.

In my experience, the desired skill set shifts to more management/admin/bureaucracy/money-chasing once your in a professor or professor-like position, as opposed to nitty-gritty researcher in the grad school phase. The incentives for the grad school phase is good science. The incentives for the professor-like phase is grants/papers/awards.

Indeed, but:

1. We are all - well, almost all of us - incompetent in many aspects of our lives, and competent only in some.

2. The incompetent are often not willing to simply cede their place and let the competent do what (arguably) needs to be done.

3. Proving and verifying competence is quite difficult unless you are yourself competent and close to their field competence in which is in question...

Bit pedantic but you should use they instead of he/she. People can have other pronouns.

> how do differences of culture — however defined — interact with traditional economic mechanisms involving prices, incomes, and simple comparative statics? Are those competing explanations, namely cultural vs. economic?

Richie's answers are mostly focused on changing the culture of science, and while there are lots of ways we could change the incentives, none of them would be pretty.

Example: let's say we want to more closely align research and actionable results, e.g., a product a company can use (Brian Armstrong argues for something like this [1]).

Solution: radically reduce public funding for scientific research and for university education as a whole (in line with Bryan Caplan's arguments in "The Case Against Education" [2]). Academics, who would be many fewer in number, would then have to get more of their funding from companies, who (presumably) would:

A) guide them towards asking market-relevant questions, and

B) have a clear incentive to check the data, re-run the code, etc. -- so that the product they built based on that research didn't flop.

I think most people would recoil at this proposal. But that's what comes to mind when I think about fixing the incentives rather than the culture.

P.S. Small nitpick: Richie gives an _example_ of a perverse incentive in lieu of a definition.

[0] https://marginalrevolution.com/marginalrevolution/2017/01/im...

[1] https://medium.com/@barmstrong/ideas-on-how-to-improve-scien...

[2] https://en.wikipedia.org/wiki/The_Case_Against_Education

Unfortunately, there is no funding for such research. Which is find really sad. Private grants might need to show "impact" but State run grants don't have that many constraints, and they could conceivably offer such grants.

I don't think this will work. All it will do is devalue the value of replication studies because only PHD students do replication studies. It's also not in their best interest especially if they dispute findings of established researchers.

Also, we have to get away from the idea that the scientist's job is to think and write, and literally all of the other work can be shuffled off onto low wage (or no wage), low status workers. This is one of the biggest reasons that science is going through such a crisis. If you want enough papers to consistently get grants you probably need at least 4/5 PHD students every few years. This causes a massive glut in the job market. It also dissociates scientists from their work. I've met esteemed computational biologists who could barely work a computer. All of their code was written, run, and analyzed by graduate students or post docs. They were competent enough at statistics, but that level of abstraction from the actual work is troubling.

Each of the suggested actions, taken together, would seem to have a very positive improvement on the status quo. Would you care to explain why the suggested actions would be "somewhere between ridiculous and useless?"

Nutrition science is real science, but unfortunately any actual nutrition science you might accidentally hear about is overwhelmed by people trying to sell a lifestyle, a book about healthy eating, or your eyeballs (to advertisers).

Ironically keto promoters are a big offender here themselves.

I think that a requirement to publish results either way on any study that's been pre-registered is important.

Well put.

In theory, people like the idea of making science better.

In practice, people don't like the idea of fewer papers, boring papers, ambiguity in hiring and tenure, uncertainty on financial return-on-investment, and less institutional/national pride talking points.

The bad incentives and metrics we have haven't happened by chance - they have emerged from our collective desire for science to be useful, sexy, and a reliable function of money/effort spent.

that's a bit unfair. yes, it does require that in some semblance, but that's not necessarily the one and only lever we have. most of us realistically expect, i hope, that we'd need a variety of political maneuvers to move our democracy toward a more representative and less insular direction.

science is no different--it took many small steps to get into this situation, and will take many small steps to get out, any one of which will seem wholly incapable on its own.

At this point in time, many of the people in power (full professors, administration, etc) see papers as essentially useless, in part because of all the problems being discussed here. p-hacking, guest authorship, etc and so forth and so on. They're seen as a dime a dozen and somewhat ignored.

This might sound good until you understand that this means that they're dismissing the entirety of the scientific dialogue essentially, and that their alternative is grant money. My institution had training workshops for grad students where faculty would tell students that research is essentially worthless unless grant money is attached to it. The idea is that papers are a dime a dozen, and that if something is worthwhile, the feds will put their money where their mouth is so to speak.

The problem with this, of course, is that the grant system is horribly nepotistic and distorted. The biggest predictor of successful grant application last time I looked (based on empirical research in peer-reviewed journals) is co-authoring papers with someone on the review panel. Grant receipt is only weakly related to citation metrics, and from personal experience I can say that institutions are constantly encouraging researchers to inflate grant costs to bring in more indirect costs.

You could fix everything about journals and the publication process and it would do nothing about the shadow scientific world that exists in parallel that drives all the rest of the problems.

Eliminate indirect funds, require tenure of all researchers, set aside funding mechanisms for researchers that aren't tied to specific grant applications, randomize grant rewards, depriortize journals, ... there's a lot of things that need to happen.

This doesn't mean that a vanishingly small percentage of papers are wrong, only that it is very hard to identify errors because papers usually don't contain enough information to fully reconstruct the results. There are a lot of assumptions of good will in the system.

This would form the basis of the Crank Index of a paper, which can be simplified into a stoplight system: Good research with good sources is GREEN. Getting featured in the Journal of BS earns a paper the esteemed distinction of a blaring scarlet RED, citing a RED paper will mark a paper ORANGE, citing ORANGE research leaves you YELLOW... Throwing together lots of ORANGE and YELLOW citations will nudge your paper up the spectrum towards RED.

This would incentivize researchers to not only care about the quantity of the citations they share with each other, but to be extremely vigilant of the quality of those citations as well.

Am I missing something? Is there great pay before you've taken a lot of time to move up the ladder, assuming you are fortunate enough?

Help me understand your reasoning rather than just down voting me.