Hacker News Reader

Comment by karaterobot | Hacker News Reader

Not a researcher, but I built a data governance system for sharing medical research data. When interviewing people, one thing that surprised me initially (but which makes total sense in retrospect) is that most researchers do not want to share their data. They want to keep it proprietary until they have published as much as possible on it. In many cases, their funders require the data to be shared, so they go through the motions, but if you give them a chance they will almost always try to deny access to anybody else. They're very afraid of someone coming along and publishing on their data before they do.

It's very much not an open source mentality. Everybody wants other scientists to share their data, but nobody wants to share their own.

View on HN

It's about the incentives, not the mentality. Collecting the data is often the bulk of the work, but the rewards mostly come from publishing papers that make discoveries from the data. If you want to base science on experiments that are properly planned and executed, the scientists doing that work must get the recognition they need to advance their careers.

Cannot agree more. My experience: 50 to 60% of experiments are collecting data, 20 % doing the actual experiment (data processing) and 20% writing the paper (and going through peer review).

Now think about what "collecting the data" means in the medical field:

- Preparing protocols and getting them through ethical review committees

- Finding suitable patients and getting them on board

- 1 on 1 patient consultations, oftentimes multiple of those for each patient and spread through a few years.

I'd say the balance can easily turn to 90% data collection, 10% everything else in many cases...

I don't see how there's any way 90% of the recognition and reward of published research can go to the data collectors over everybody else combined, even in the cases where data collection is really that onerous.

Which I guess is why all efforts to try to split up the work in a more efficient manner have failed and the current model persists. This means that open academic research effectively remains a cottage industry compared to industrial, military, and other research types that can be kept secret.

This is a great suggestion. Data collection is itself a (sometimes big) contribution to science and humanity's state of knowledge, and we should reward it accordingly.

Data collection is more expensive, more time consuming, and usually more tedious than data analysis and paper writing. The people good at collection are not necessarily good at analysis and vice versa. More people enjoy analysis than collection. The results of the analysis are what is useful to the rest of society. Also, the people doing the analysis need to have very detailed input to its collection.

I was hoping stating it all together like that would make a solution more obvious, but I don't think it did. Having researchers commission data collection and every user having to pay would create barriers to entry we don't want.

You can't exactly decide what to reward. A lot of the important aspects of "reward" are implicit, non-legible, unstructured, informal. It's about how the community perceives you. About how many people walk up to you to chat with you at a conference. About how many requests you get to visit another lab. It's about intangible "status" and respect within the community. And it so happens that this informal respect is mostly given to people who make exciting discoveries. You can't top-down force the criteria for this. Data collection doesn't make you an exciting personality to invite to give a keynote speech or something. And those opportunities lead to more visibility, more collaborations, more papers, more grants, more recognition, more fame etc.

An arrangement that would make sense would be to separate data collection from publication and make the academic success of data collectors predicated on how many publications use their data. That would incentivize data collectors to make their data as widely useful and available as possible, which is very counter to the current situation. I imagine data collection groups talking to a number of data analysis groups and figuring out what to collect that would be useful for the largest number of interesting experiments. This might also make data collected intentionally or accidentally to give a specific outcome less likely.

Unfortunately, it's absurdly hard to change what counts as credit towards a successful academic career, so this seems like a pipe dream. Academia is still waging a very uphill battle to get Research Engineer to be a long-term viable career path, despite the absolutely critical need for software engineering in all areas of modern science.

As a former (PhD) researcher, this is spot on. Here are the factors I've observed:

- fear of getting scooped: there's a general fear that sharing data makes it easier for competing labs to publish on the same topics, leading to the publishing lab "missing out" on publications

- fear of being wrong: sharing the data leads to more scrutiny, leading to a higher chance of a fatal flaw being found

- extra work: publishing data often involves jumping through extra hoops with little to no personal gain

Not a researcher, but have worked closely with them on data sharing.

I think there's a variant of getting scooped worth pointing out, which is that in some fields people are scared of being scooped by a low quality lab that fakes the results.

I had a very senior materials science researcher describe what he thought was a promising line of study get shut down because a low quality lab in china scooped him with results good enough for publication but not good enough that people would want to replicate. He was 100% confident their data was faked but getting grant money once that experiment was done was like pulling teeth and so he abandoned the route for other ideas.

This is interesting. Not something that I ever came across but can very much believe this happens.

Most problems in academia come down to bad incentives. Researchers are incentivised to publish for prestige and citations, which are poor proxies for improving our understanding of the world around us.

All problems, and really all actions, can be explained by incentives. Once you understand that people will never under any circumstances do something that both takes effort and is opposite to their incentives, you understand why nearly every problem occurs and why every action is taken.

This statement is either so broad that it is obviously false, or has so many implied additional conditions that its application is incredibly narrow.

* The time horizon used to judge incentives vary person to person.

* Willing martyrs exist for various causes. Therefore, it is not just the incentives that matter, but the belief about those incentives.

* Self-sacrifice for others exists (e.g. a mother rushing into a burning building for the chance of saving her children). Therefore, it is not just the incentives that matter, but also the subjective weighting of conflicting incentives.

* A person may spend a great deal of time researching betting patterns for playing craps, none of which can overcome the house's advantage. Therefore, it is not just the incentives that matter, but the knowledge of the outcomes.

* Somebody may work in an oppressive environment for their entire life, rather than changing careers. Therefore, it is not just the effort required that matters, but also the relative amount of effort on different time horizons.

While I agree that incentives are the best way to move population-level behavior, and is usually a good starting point for understanding individual behavior, your categorical statement doesn't have those caveats.

Billion dollar question is what a better proxy would be.

Possibly a reformed patent system, e.g. one in which time limits are variable or in which you don't get a full monopoly (forced licensing terms). At one point I was researching an alternative patent system in which you don't get a monopoly on the idea, but rather for every next licensee they have to pay in a share of the total value of the patent such that e.g. the first licensee pays half which goes direct to the original filer, the next licensee pays a third and that money is split between the other two licensees, the next pays a quarter etc. So the patent gets cheaper over time to license, but you lose the first mover advantages.

The problem with that system is that the "value" would have to be determined in some independent manner, e.g. an audited cost+ basis, which wouldn't be reflective of the true value of the idea. Granting time-limited monopoly rights allows the idea to be priced via normal market mechanisms, at the cost of high transaction costs.

At any rate, you get the idea. The justification for academic research is that the private sector, supposedly, won't do long term basic science and that grant recipients will. That's a very dubious set of assumptions. We see basic science being done by private labs all over the place, most obviously in AI but also in many other areas of computer science, we see it in the biotech sector, we see it in semiconductor physics etc. The areas where there's no private investment tend also to be the most controversial areas of academia where many people have concluded that the results are systematically useless e.g. social sciences. And meanwhile what we see amongst grant recipients is systematically unscientific behaviour like knowingly publishing claims that aren't actually true, refusing to share data even after promising to do so, accepting unvalidated modelling as 'science' and a million other things.

This doesn't make sense. Being scooped means other people published the same idea before you.

If he truly believes the released data were fake, he can for sure publish his results. It’s not unusual for scientists to have different views on certain subjects. Some journals even have specific discussion sections for contradictory results. Finding the truth and correcting scientific records is also an achievement.

And at least, in the filed of biology/medicine, there are journals accepting “second first” papers, for example https://elifesciences.org/articles/30076 .

And quit a lot of publications are not grand breaking ideas. Incremental improvements can also be published.

I don’t understand your friend’s rationale to abandon the whole project he already started and didn’t want to find the truth and prove his theory (maybe it’s a different field thing).

> If he truly believes the released data were fake, he can for sure publish his results.

You need the funding to get to that point though, and since it wasn't "new" they weren't able to get funded to research it. That's the meaning I got from the comment.

> I don’t understand your friend’s rationale to abandon the whole project he already started and didn’t want to find the truth and prove his theory (maybe it’s a different field thing).

He has lots of ideas he wants to try, and he ranks them. The incentives for following this particular idea were reduced by the actions of the other lab, so it dropped in the rankings and he pursued one of the more highly ranked ones.

Am a researcher, and also work in data governance, data sharing, etc. for medical research data.

It's not about an "open source mentality" - almost all of my analytical code is online, and when I work with simulated data sets (the bulk of my work) it's freely available.

It's about incentives, and effort. A couple examples:

1) Primary data collection is hugely time and resource intensive. It's hard to be okay with all of that, getting a single paper out, and then immediately having that be expected to be freely open and available.

The argument that it's necessary for reproducibility is a compelling one, but also one that's much rare than what most people are looking for, which is to do new and novel research using your primary data.

2) Research has a potentially long time from data collection to publication. It's entirely possibly the reason isn't "I may want to work on that someday" but "A graduate student is working on that right now". I have been in the position of potentially having a graduate student be scooped on something, and it is terrible (it ended up being a false alarm). They're potentially also the person who put in a lot of the labor on publishing the paper in the first place.

3) Sharing comes with an unknown expectation of support, documentation, etc. wherein there's absolutely no incentive structure in place to support the work that takes. I write in time and budget to do so in my grants, but that also mean my grants have less science in them than other people's, and I have to rely on that mattering to someone enough to make up the difference.

4) Genuine data ownership issues. Health data is especially complex in this space. Do you want a simulated data set that will give you the same broad patterns? That's easy enough. But if you want the real data - that's potentially a very complex issue wherein even if I say "Sure, let's get that process started..." you're potentially looking at a year or more before you get the data.

This x100.

I'd also add that primary data collection is risky. You can spend months to years working to get data, only to end up with nothing---not even a null result---if the experiment itself goes south. I lost over a year of my PhD when an animal I had spent ages training fell while playing. A classmate had a virus wipe out her colony of genetically-engineered mice. Using someone else's data obviates these sorts of risks--while doing a better job preparing you for non-academic careers too!

As a result, I think it's not totally nuts for the original data collectors to maybe get some kind of "risk premium." I can imagine a lot of ways to do it, but early/exclusive access to data has been one of them.

A colleague of mine during graduate school who by consensus did everything right, and then just...no one in her cohort had the exposure she was interested in. To this day, no one is sure why.

The answer for a lot of this is to put in the hard work of building relationships with people who collect a lot of primary data, and make sure your work amplifies their work in a way that benefits you both. But that's a lot of effort, hard, and uncertain in its own ways.

I don't have any direct research but am tangentially aware of some of this. I guess my question comes down to why getting scooped matters?

I definitely understand the worry when it comes to high profile discoveries, but surely the work a graduate student is working on, isn't usually that high profile?

And to be clear, I'm not downplaying the importance of anyones contributions, but rather why the focus isn't shifted onto where data is sourced from? If someone used your data, shouldn't it be immediately obvious if they also have data publishing requirements? And if thats true, why isn't it a bigger faux pas to use another labs data before that lab has even published their paper on it?

In some ways the graduate student is working on the most important work of their career. A PhD requires novel publications and if you get scooped on the main result of your thesis so that you don’t end up with the published paper on that topic very few organizations will grant the PhD. So this can result in losing out on getting a degree you put five years of your life into.

Being first is almost everything -- the product in academic research is new knowledge.

How much press was given to the second time the LHC discovered the Higgs, with higher precision?

Robert Dicke, in the running for "greatest experimental physicist in history", was working rapidly toward discovering the cosmic microwave background, with a keen physical insights driving the research. His team was beaten by a team at Bell Labs, who, when they discovered an irreducible isotropic source of noise emanating from the Universe, literally called Dicke [1] to help them understand the signal.

Who got the Nobel Prize? The team from Bell Labs. Never mind that Dicke literally wrote the companion paper explaining why the Bell Labs measurement was so important. Why? Because they were first.

Regarding graduate students: It is extremely common in physics to have the lead author (and coordinator) on important/cutting-edge research be a graduate student. It is only the largest collaborations where this isn't always the case.

Graduate students tend to follow a single thread of research from beginning to end, learning the ropes on the way, while faculty generally manage and mentor multiple threads (each with a student) simultaneously. When it comes time to publish, frequently it is the student who knows all of the details of the measurement the best.

For your final question, the answer goes back to "because being first is almost everything", combined with "we have to give the credit to somebody, right?". If someone notices that the eighth word on 50% of Guinness World Record citations is "person", should the credit for the discovery go to the person who noticed that interesting fact, or should it primarily go to Guinness World Records, who assembled and produced the dataset over decades of work?

Or, more-concretely, when someone applies for and receives fifteen minutes of time on the Hubble Telescope and, using a clever time-domain analysis, happens to find a repeating optical signal that blinks 13 times, then 53 times, then 13 times, then 53 times (as good a SETI candidate as you'll find), who gets the credit? Is it the researcher with the keen insight and good luck, or is it the huge amalgamation of humanity, spanning a half-century of science and engineering, that made such a measurement possible?

[1] https://theconversation.com/the-cmb-how-an-accidental-discov....

Graduate students have like four good years of research time. If your work is scooped it will almost guarantee low citation counts but very likely it means that the work cannot be published at all. A year down the drain.

If you want a faculty position after graduation you've got one shot to create a strong research profile. Getting scooped once can close career doors forever.

People also often don't understand what grad students do. They do all of the research. 100% of it. PIs are funding the lab and providing mentorship. The first authors on major groundbreaking papers are usually grad students.

So, the caveat to all of this is I'm actually in a field (epidemiology) where getting scooped matters less in many cases than it does for some others (math).

Getting scooped doesn't just matter for high profile discoveries. The example I was thinking of is the graduate student's research asking "Does this thing the field is doing make sense?"

That's a Yes or No answer. There's not a lot of interest once the question has been answered. Even if it can be published, it'll be a harder road, potentially finding itself in a less prestigious journal, etc. At worst, it's a write-off. Most graduate students are coming out with 2 or 3 paper dissertations, so the loss of one, or taking a hit on another one, can set someone behind, dramatically change the tenor of their job search, etc.

"And to be clear, I'm not downplaying the importance of anyones contributions, but rather why the focus isn't shifted onto where data is sourced from? If someone used your data, shouldn't it be immediately obvious if they also have data publishing requirements? And if thats true, why isn't it a bigger faux pas to use another labs data before that lab has even published their paper on it?"

I mean, it might not be the only thing that's been published on that data - that's the point. Once it's publicly available, it's publicly available, even if people are still working on it.

Indeed, one of the "Data is available upon reasonable request" things is "We have a student working on this, could you not?" (another being "You're a crank, and we're not sending you this to misinterpret/misuse it"), but a lot of open data folks actively dislike that for obvious reasons.

I can't speak for all research datasets but I can speak for sequencing based datasets such as whole genome sequencing, RNA sequencing etc. These projects can cost hundreds of thousands to millions of dollars. Academic and industrial research projects tend to do everything they can to avoid sharing this data for as long as possible.

The academic researchers that generate these datasets and their funding sources such as granting agencies and donors want to see the money being used to generate something important. In this case it certainly in the researchers interest to avoid sharing the data for as long as possible to get as many publications as possible. This makes it easier to get more funding.

There is a push by journals now for researchers to include the data in publicly accessible repositories such as the NCBI SRA or European Genome-phenome Archive. These sites archive the data and make it available for researchers. However, in most cases they require a comprehensive data sharing agreement between your institution and theirs. Most of these agreements have extremely demanding requirements that make it very difficult if not impossible for a requesting institutes legal team to agree to. For example, the institute that owns the data may make demands such as "the right to approve any publications or research work created with this data", which entails sending any publication to them prior to submission and giving them veto power. I understand the reasoning for the agreements their intent was to protect private information from being shared publicly while allowing researchers to use the data. But I think the system is being abused now.

On the other hand, it can be very difficult to get past institutes to make data public. For example, sequencing data contains personally identifiable information and getting approval to share the raw data can be difficult. For prospective studies, Human patients need to consent to their data being shared and may not consent to it being shared publicly.

I can't comment on industry as much. However, I have colleagues who work at companies and publish numerous papers on the same proprietary datasets but refuse to share it with anyone. This is particularly challenging in fields such as cancer research where they may publish some fancy/superior new model for disease risk stratification on their own data but without sharing it's impossible for researchers to independently validate it.

I'm an RSE building such a tool right now for a clinical trial, and came in knowing this is the norm.

We're experimenting with building a data collection system that uses adversarial/multiplayer game mechanics incentivizes researchers to share (gain co-authorship) and excludes them from the benefits (e.g. co-authorship) if the minimum submitted data threshold isn't met.

Still early days, but we figured this is the only way to get people to share data, even within other people within the same project / lab. Similarly, I'm getting a lot of ideas from HN on event-driven data collection and immutable data storage to help make the data more tamper proof.

This is really interesting! I'm interested in knowing more and sharing with colleagues in academic library/publishing tech, if you want to drop a link! (and are interested in sharing your work :) )

I would love to talk to you about this. I am currently doing something similar but accentuating the dataset by adding info on publications etc. thereby making them more "legit" (for the lack of a better word)

Couldn't this be fixed by having the creator of the data as a co-author on any paper that uses the data?

The basic conflict is that it's hard/expensive/time-consuming to make data, and easier to write papers on existing data sets. So just split the work, and make it a legitimate occupation to create good data sets. As of now, generating the data by itself has no benefits for scientists, other than the papers they can subsequently write using it. Feels a bit like having to make a screwdriver every time you want to work on your car.

A better solution is to properly credit researchers for releasing data that then gets used by caring about citations to data.

That needs to be tracked better, things are improving but until there are metrics and tracking as a common part of evaluation I don't think it will change.

As a sibling says, it is probably better to improve citation and culture around the importance of datasets.

As an experimentalist, I really want people to make use of my measurements, but I really don't want to be a co-author on a paper that makes wild and incorrect conclusions using my data.

I've heard one even worse.

I know people who were told by their PI to tell other researchers at conferences that they were working on the stuff that they already tried that didn't work. This would encourage competing labs to try out these techniques and waste their time.

Being scooped is a huge blow to a career, for faculty but more intensely for graduate students. With arXiv, you can be scooped at literally any moment rather than just on the known conference schedule. Being scooped was personally traumatic for me and I still have nightmares about the experience.

This leads to a massively toxic ecosystem of over-competition where individual labs win at the detriment of scientific progress.

Interestingly, one of the best ways to rack up citations in ML is to publish a benchmark dataset.

Perhaps the incentives need to be re-aligned to separate the collection of data and the publishing of studies based on that data.

It's not just publishing it before them, it's opening themselves up to criticism. If someone wants to look for mistakes or flaws in your data, it's easier with access.

Of course, arguing about the right answer is supposed to be what science is about. That doesn't make it any more pleasant.

Sounds like it's time to split the industry. One side collects the data and puts it into common repositories. Another group does research using the primary data.

That sounds very inefficient though.

Oh I would love to talk to you about this. I am working on building a similar system and encountered this so much and other issues.

Is there a way I could contact you?

Most researchers are not super smart, and so they cannot get out of the data more than is relatively obvious. If they work hard to obtain the data, but then share it freely, someone else will just steal their lunch.

Reply on news.ycombinator.com