Many researchers were not compliant with their published data sharing statement
jclinepi.com
jclinepi.com
It's very much not an open source mentality. Everybody wants other scientists to share their data, but nobody wants to share their own.
We're experimenting with building a data collection system that uses adversarial/multiplayer game mechanics incentivizes researchers to share (gain co-authorship) and excludes them from the benefits (e.g. co-authorship) if the minimum submitted data threshold isn't met.
Still early days, but we figured this is the only way to get people to share data, even within other people within the same project / lab. Similarly, I'm getting a lot of ideas from HN on event-driven data collection and immutable data storage to help make the data more tamper proof.
- fear of getting scooped: there's a general fear that sharing data makes it easier for competing labs to publish on the same topics, leading to the publishing lab "missing out" on publications
- fear of being wrong: sharing the data leads to more scrutiny, leading to a higher chance of a fatal flaw being found
- extra work: publishing data often involves jumping through extra hoops with little to no personal gain
I think there's a variant of getting scooped worth pointing out, which is that in some fields people are scared of being scooped by a low quality lab that fakes the results.
I had a very senior materials science researcher describe what he thought was a promising line of study get shut down because a low quality lab in china scooped him with results good enough for publication but not good enough that people would want to replicate. He was 100% confident their data was faked but getting grant money once that experiment was done was like pulling teeth and so he abandoned the route for other ideas.
Most problems in academia come down to bad incentives. Researchers are incentivised to publish for prestige and citations, which are poor proxies for improving our understanding of the world around us.
* The time horizon used to judge incentives vary person to person.
* Willing martyrs exist for various causes. Therefore, it is not just the incentives that matter, but the belief about those incentives.
* Self-sacrifice for others exists (e.g. a mother rushing into a burning building for the chance of saving her children). Therefore, it is not just the incentives that matter, but also the subjective weighting of conflicting incentives.
* A person may spend a great deal of time researching betting patterns for playing craps, none of which can overcome the house's advantage. Therefore, it is not just the incentives that matter, but the knowledge of the outcomes.
* Somebody may work in an oppressive environment for their entire life, rather than changing careers. Therefore, it is not just the effort required that matters, but also the relative amount of effort on different time horizons.
While I agree that incentives are the best way to move population-level behavior, and is usually a good starting point for understanding individual behavior, your categorical statement doesn't have those caveats.
The problem with that system is that the "value" would have to be determined in some independent manner, e.g. an audited cost+ basis, which wouldn't be reflective of the true value of the idea. Granting time-limited monopoly rights allows the idea to be priced via normal market mechanisms, at the cost of high transaction costs.
At any rate, you get the idea. The justification for academic research is that the private sector, supposedly, won't do long term basic science and that grant recipients will. That's a very dubious set of assumptions. We see basic science being done by private labs all over the place, most obviously in AI but also in many other areas of computer science, we see it in the biotech sector, we see it in semiconductor physics etc. The areas where there's no private investment tend also to be the most controversial areas of academia where many people have concluded that the results are systematically useless e.g. social sciences. And meanwhile what we see amongst grant recipients is systematically unscientific behaviour like knowingly publishing claims that aren't actually true, refusing to share data even after promising to do so, accepting unvalidated modelling as 'science' and a million other things.
If he truly believes the released data were fake, he can for sure publish his results. It’s not unusual for scientists to have different views on certain subjects. Some journals even have specific discussion sections for contradictory results. Finding the truth and correcting scientific records is also an achievement.
And at least, in the filed of biology/medicine, there are journals accepting “second first” papers, for example https://elifesciences.org/articles/30076 .
And quit a lot of publications are not grand breaking ideas. Incremental improvements can also be published.
I don’t understand your friend’s rationale to abandon the whole project he already started and didn’t want to find the truth and prove his theory (maybe it’s a different field thing).
You need the funding to get to that point though, and since it wasn't "new" they weren't able to get funded to research it. That's the meaning I got from the comment.
He has lots of ideas he wants to try, and he ranks them. The incentives for following this particular idea were reduced by the actions of the other lab, so it dropped in the rankings and he pursued one of the more highly ranked ones.
- Preparing protocols and getting them through ethical review committees
- Finding suitable patients and getting them on board
- 1 on 1 patient consultations, oftentimes multiple of those for each patient and spread through a few years.
I'd say the balance can easily turn to 90% data collection, 10% everything else in many cases...
Which I guess is why all efforts to try to split up the work in a more efficient manner have failed and the current model persists. This means that open academic research effectively remains a cottage industry compared to industrial, military, and other research types that can be kept secret.
I was hoping stating it all together like that would make a solution more obvious, but I don't think it did. Having researchers commission data collection and every user having to pay would create barriers to entry we don't want.
Unfortunately, it's absurdly hard to change what counts as credit towards a successful academic career, so this seems like a pipe dream. Academia is still waging a very uphill battle to get Research Engineer to be a long-term viable career path, despite the absolutely critical need for software engineering in all areas of modern science.
The academic researchers that generate these datasets and their funding sources such as granting agencies and donors want to see the money being used to generate something important. In this case it certainly in the researchers interest to avoid sharing the data for as long as possible to get as many publications as possible. This makes it easier to get more funding.
There is a push by journals now for researchers to include the data in publicly accessible repositories such as the NCBI SRA or European Genome-phenome Archive. These sites archive the data and make it available for researchers. However, in most cases they require a comprehensive data sharing agreement between your institution and theirs. Most of these agreements have extremely demanding requirements that make it very difficult if not impossible for a requesting institutes legal team to agree to. For example, the institute that owns the data may make demands such as "the right to approve any publications or research work created with this data", which entails sending any publication to them prior to submission and giving them veto power. I understand the reasoning for the agreements their intent was to protect private information from being shared publicly while allowing researchers to use the data. But I think the system is being abused now.
On the other hand, it can be very difficult to get past institutes to make data public. For example, sequencing data contains personally identifiable information and getting approval to share the raw data can be difficult. For prospective studies, Human patients need to consent to their data being shared and may not consent to it being shared publicly.
I can't comment on industry as much. However, I have colleagues who work at companies and publish numerous papers on the same proprietary datasets but refuse to share it with anyone. This is particularly challenging in fields such as cancer research where they may publish some fancy/superior new model for disease risk stratification on their own data but without sharing it's impossible for researchers to independently validate it.
Is there a way I could contact you?
It's not about an "open source mentality" - almost all of my analytical code is online, and when I work with simulated data sets (the bulk of my work) it's freely available.
It's about incentives, and effort. A couple examples:
1) Primary data collection is hugely time and resource intensive. It's hard to be okay with all of that, getting a single paper out, and then immediately having that be expected to be freely open and available.
The argument that it's necessary for reproducibility is a compelling one, but also one that's much rare than what most people are looking for, which is to do new and novel research using your primary data.
2) Research has a potentially long time from data collection to publication. It's entirely possibly the reason isn't "I may want to work on that someday" but "A graduate student is working on that right now". I have been in the position of potentially having a graduate student be scooped on something, and it is terrible (it ended up being a false alarm). They're potentially also the person who put in a lot of the labor on publishing the paper in the first place.
3) Sharing comes with an unknown expectation of support, documentation, etc. wherein there's absolutely no incentive structure in place to support the work that takes. I write in time and budget to do so in my grants, but that also mean my grants have less science in them than other people's, and I have to rely on that mattering to someone enough to make up the difference.
4) Genuine data ownership issues. Health data is especially complex in this space. Do you want a simulated data set that will give you the same broad patterns? That's easy enough. But if you want the real data - that's potentially a very complex issue wherein even if I say "Sure, let's get that process started..." you're potentially looking at a year or more before you get the data.
I'd also add that primary data collection is risky. You can spend months to years working to get data, only to end up with nothing---not even a null result---if the experiment itself goes south. I lost over a year of my PhD when an animal I had spent ages training fell while playing. A classmate had a virus wipe out her colony of genetically-engineered mice. Using someone else's data obviates these sorts of risks--while doing a better job preparing you for non-academic careers too!
As a result, I think it's not totally nuts for the original data collectors to maybe get some kind of "risk premium." I can imagine a lot of ways to do it, but early/exclusive access to data has been one of them.
The answer for a lot of this is to put in the hard work of building relationships with people who collect a lot of primary data, and make sure your work amplifies their work in a way that benefits you both. But that's a lot of effort, hard, and uncertain in its own ways.
I definitely understand the worry when it comes to high profile discoveries, but surely the work a graduate student is working on, isn't usually that high profile?
And to be clear, I'm not downplaying the importance of anyones contributions, but rather why the focus isn't shifted onto where data is sourced from? If someone used your data, shouldn't it be immediately obvious if they also have data publishing requirements? And if thats true, why isn't it a bigger faux pas to use another labs data before that lab has even published their paper on it?
How much press was given to the second time the LHC discovered the Higgs, with higher precision?
Robert Dicke, in the running for "greatest experimental physicist in history", was working rapidly toward discovering the cosmic microwave background, with a keen physical insights driving the research. His team was beaten by a team at Bell Labs, who, when they discovered an irreducible isotropic source of noise emanating from the Universe, literally called Dicke [1] to help them understand the signal.
Who got the Nobel Prize? The team from Bell Labs. Never mind that Dicke literally wrote the companion paper explaining why the Bell Labs measurement was so important. Why? Because they were first.
Regarding graduate students: It is extremely common in physics to have the lead author (and coordinator) on important/cutting-edge research be a graduate student. It is only the largest collaborations where this isn't always the case.
Graduate students tend to follow a single thread of research from beginning to end, learning the ropes on the way, while faculty generally manage and mentor multiple threads (each with a student) simultaneously. When it comes time to publish, frequently it is the student who knows all of the details of the measurement the best.
For your final question, the answer goes back to "because being first is almost everything", combined with "we have to give the credit to somebody, right?". If someone notices that the eighth word on 50% of Guinness World Record citations is "person", should the credit for the discovery go to the person who noticed that interesting fact, or should it primarily go to Guinness World Records, who assembled and produced the dataset over decades of work?
Or, more-concretely, when someone applies for and receives fifteen minutes of time on the Hubble Telescope and, using a clever time-domain analysis, happens to find a repeating optical signal that blinks 13 times, then 53 times, then 13 times, then 53 times (as good a SETI candidate as you'll find), who gets the credit? Is it the researcher with the keen insight and good luck, or is it the huge amalgamation of humanity, spanning a half-century of science and engineering, that made such a measurement possible?
[1] https://theconversation.com/the-cmb-how-an-accidental-discov....
If you want a faculty position after graduation you've got one shot to create a strong research profile. Getting scooped once can close career doors forever.
People also often don't understand what grad students do. They do all of the research. 100% of it. PIs are funding the lab and providing mentorship. The first authors on major groundbreaking papers are usually grad students.
Getting scooped doesn't just matter for high profile discoveries. The example I was thinking of is the graduate student's research asking "Does this thing the field is doing make sense?"
That's a Yes or No answer. There's not a lot of interest once the question has been answered. Even if it can be published, it'll be a harder road, potentially finding itself in a less prestigious journal, etc. At worst, it's a write-off. Most graduate students are coming out with 2 or 3 paper dissertations, so the loss of one, or taking a hit on another one, can set someone behind, dramatically change the tenor of their job search, etc.
"And to be clear, I'm not downplaying the importance of anyones contributions, but rather why the focus isn't shifted onto where data is sourced from? If someone used your data, shouldn't it be immediately obvious if they also have data publishing requirements? And if thats true, why isn't it a bigger faux pas to use another labs data before that lab has even published their paper on it?"
I mean, it might not be the only thing that's been published on that data - that's the point. Once it's publicly available, it's publicly available, even if people are still working on it.
Indeed, one of the "Data is available upon reasonable request" things is "We have a student working on this, could you not?" (another being "You're a crank, and we're not sending you this to misinterpret/misuse it"), but a lot of open data folks actively dislike that for obvious reasons.
The basic conflict is that it's hard/expensive/time-consuming to make data, and easier to write papers on existing data sets. So just split the work, and make it a legitimate occupation to create good data sets. As of now, generating the data by itself has no benefits for scientists, other than the papers they can subsequently write using it. Feels a bit like having to make a screwdriver every time you want to work on your car.
That needs to be tracked better, things are improving but until there are metrics and tracking as a common part of evaluation I don't think it will change.
As an experimentalist, I really want people to make use of my measurements, but I really don't want to be a co-author on a paper that makes wild and incorrect conclusions using my data.
I know people who were told by their PI to tell other researchers at conferences that they were working on the stuff that they already tried that didn't work. This would encourage competing labs to try out these techniques and waste their time.
Being scooped is a huge blow to a career, for faculty but more intensely for graduate students. With arXiv, you can be scooped at literally any moment rather than just on the known conference schedule. Being scooped was personally traumatic for me and I still have nightmares about the experience.
This leads to a massively toxic ecosystem of over-competition where individual labs win at the detriment of scientific progress.
Of course, arguing about the right answer is supposed to be what science is about. That doesn't make it any more pleasant.
When I started my lab head got a request for an old paper (before my time) and I had to hunt down the data, put it on the web, find the code, send an email etc - it was a lot of work, almost as much as just sharing it in the first place.
So now all the code goes up on GitHub, all the sequencing reads to public archives, and we fill the supps full of any intermediate CSVs etc we think people might find useful.
I saw it as "people bothering me so I would do admin" but some lab heads see being requested as "strategic networking and being informed about other research in the field" where they can see what others are doing and gain collaborations / be put onto papers etc
I guess it's just expectations: I've met surgeons who have the expectation that because they kept a piece of tumor from a surgery and gave it to a lab, they will be on any paper that ever uses data from that tissue until the end of time (even on labs on the other side of the world re-using the same data).
When I'm collecting public data, I usually don't bother to ask due to the hassle, and move straight on to the actually public stuff.
My bet is that papers with public code/data are used/cited way more often, but the tradeoff is probably their authors get less total papers over all.
Do you want a larger number of people slightly happy with you or a fewer number of "close friends" in your area who may look kindly on you for grants or paper reviews.
In fact, even the very idea of a published paper feels out of date. Why have one model that is set in stone that is published as one snapshot in time?
In the DevOps and Cybersecurity world our alerting mechanisms are essentially simple data models (sometimes not so simple) that are tapped into the raw data feeds, and can be updated and refined on an ongoing basis.
While instrumenting the physical world can have more challenges, I still feel that the sciences have a lot to be gained from some of the more "pedestrian" operational processes that we use in the technology industry.
That's not to say that it's not important, but the labor required is vastly underappreciated. I say this as someone who has completed the task for many papers.
Budgeting for a separate person to do this would eat up a tremendous amount of said budget.
Which is fine, except if we think about this from a productivity standpoint, the person who budgeted for that person is probably short a graduate student compared to the person who didn't, on the outside chance that someone cares about their stuff. For example, I looked at my lab's publicly available repositories linked to papers on Github.
Watches: 15 (probably half of these are people involved in the project) Forks: 3 Stars: 4 Visitors in the Past Month: 4
That's it. For all of them. While I keep doing it based on principle, if I stopped tomorrow, it would impact me not at all.
And undergraduates, candidly, are not usually time savers.
Because the incentive in academia is to take a single idea that kinda works and parcel it out into as many, generally low quality, published papers as you can.
The moment you release the data somebody else can start parceling out those papers instead of you.
I'm not sure how this could work with things like photos etc (though there are plugins for some of this)
- What benefit does this give to the researchers who are publishing the data?
- Who is paying for storing the data - frequently in the TB?
When the answers are "approximately none", and "the researchers" you're not going to get many takers.
And that's assuming it's easy, it reality it often won't be. If there's a lot of data, it's literally just a pain to upload it (network bandwidth). If there's sensitive data (PII), it's a pain to redact it and make sure you aren't leaking any. Data is frequently in strange formats, and it's a pain to translate it to a standard one. Etc.
---
I've worked with 3 university labs as a contract programmer. In all of them I worked with data with one of the issues mentioned above. Health information in one, TBs of photonics data in another (which was being parsed by extremely janky code too), and 4 16-bit channel images in the last (hundreds of GB of them too). Admittedly for the last it would have been easy to upload it as long as you didn't want people to be able to actually view it (on the other hand I wrote some software to let people false color them live in a browser for the lab, so that the researchers could view them).
Which answers the "what benefit does it provide to the researchers publishing the data" question. A quick search answers the funding question as well, it's funded by the NIH, not the individual labs using it.
I think this example supports my point. The NIH came up with a way to give different answers to the two questions I asked, and it gets used. I'm glad the NIH has been making this a thing, it's a great use of public funds.
I'd still caution anyone from trying a "make a data platform and researchers will use it" approach to the problem unless they can answer those questions.
Think about all of the fancy dependencies a piece of code might need to run. Think of all of the processing steps some data needed to go through. Think of the work required to set up some computational cluster without which the work just doesn't happen. Doing it once is hard enough, documenting it so anyone else can do it is maybe an order of magnitude harder.
Maybe the goal should merely be for the author to be able replicate the results from the torrent archive (for a fee of course). That way anyone who needs the conclusion to be correct can buy the validation.
Which is a huge part of why the replication crisis is such a thing (besides the outright fraud and the publication bias). The very fact that the datasets and codebases are so disgusting is precisely why the results coming from that data can't be trusted.
Is it though? From what I’ve seen it is mostly caused by p-hacking/small sample sizes/poor experimental design.
An incomplete list of other causes might contain:
• Wrong data or maths. A remarkably large number of papers (e.g. in psychology) contain statistical aggregates that are mathematically impossible given the study design, like means which can't be calculated from any allowable combination of inputs. Some contain figures that are mathematically possible but in reality totally implausible [1]
• Fraud. I've seen estimates that maybe 20% of all published clinical trials haven't actually been done at all. Anyone who tried to figure out the truth about COVID+ivermectin got a taste of this because a staggering quantity of studies turned out to exhibit disturbing signs of trial fraud. Researchers will happily include obvious Photoshops in their papers and journals will do their best to ignore reports about it [2]
• Bugs. Code doesn't get peer reviewed, sometimes not released either. The famous Report 9 Imperial College London COVID model was in development since 2004 but was riddled with severe bugs like buffer overflows, race conditions, and even a typo in the constants for their hand-rolled PRNG [3]. As a consequence their model produced very different numbers every time you ran it, despite passing fixed PRNG seeds in on the command line. The authors didn't care about this because they'd convinced themselves it wasn't actually a problem (and if you're about to argue with me on this, please don't, if I have to listen to one more academic explaining that scientists don't have to write deterministic "codes" I'll probably puke).
• Pretend data sharing. Twitter bot papers have perfected the art of releasing non-replicable data analysis, because they select a bunch of tweets on topics Twitter is likely to ban, label them "misinformation" and then in their publicly shared data include only the tweet ID, not the content. Anyone attempting to double check their data will discover that almost all the tweets are no longer available, so their classifications can't be disputed. They get to claim their analysis is replicable because they shared their data even though it's not.
• Methods that are described too vaguely to ever replicate.
And so on and so forth. There are an unlimited number of ways to make a paper that looks superficially scientific, but doesn't actually tell us something concrete that can survive being double checked.
[1] e.g. https://hackernoon.com/introducing-sprite-and-the-case-of-th...
[2] https://blog.plan99.net/fake-science-part-i-7e9764571422
[3] https://dailysceptic.org/archive/second-analysis-of-ferguson...
I would wager that the vast majority of analyses / code, in most fields of research, can run on a basic laptop. If I had to put a number on it, I'd say >99.999% of papers.
Not in terms of proportion of data of course, but in terms of proportion of papers.
And I was the most organized programmer among my peers by a long shot. Judging from the messiness of shared code, I can’t imagine how bad other people’s unshared work is.
Some of it is just really clean Python code that generates a bunch of simulated data to train some machine learning code on with human readable configuration files.
But neither one of them involve the words "$COUNTRY Ministry of Health approval...", which kicks things to a whole new level.
I agree generally with data and code sharing policies (all of my recent papers include zenodo deposits of data, and I work on a large open source simulation code). However, there are some serious issues here:
1. Code licensing is not always clear. Sometimes you are working with a code that some PhD student wrote, who left the field, whose advisor shared it with you. Can you share this code? Not necessarily... but this problem will go away with time.
2. Some codes include proprietary components, and chopping them out can make the code inoperable (or useless). One notable instance of this is a very famous hydro code that was used to produce predictions for nukes, that was then repurposed for running simulations of planetary collisions.
3. Sharing data is prohibitive, and if you don't have the data having the code might not really help. Our projects can produce Pbs of data, costing 10s-100s Ms of CPU hours, and as such making the data open source can be really tricky. This then means you can only share a selection of runs. On the other hand, even if you had the code, you can't reproduce a lot of the results, because of the cost of running the simulations... This actually will get _worse_ over time as we run bigger and bigger simulations.
Isn't it the university who owns everything? I don't know about astrophysics, but it's a big problem in engineering. Universities have been known to patent and resell research made by msc/phd student and profit from it without paying any compensation.
Consider this: student at institution A (UK) writes code, postdoc at institution B (USA) modifies it, shared with researcher at institution C (China) who runs it for a paper first-authored by student at institution D (Germany).
Who has the rights to share the code and data? Which jurisdiction would this fall under?
1) Are you sure the data is deidentified? Can your analysis be done on the deidentified data?
2) Do you actually own the data? This is more complicated than "Well you published a paper on it...". The data is likely governed by data use agreements. Those are often extremely complex - especially at the international level. Or if you're working with particular populations - they're not wrong to be wary of who their data is given to, given the history of how science generally has treated them.
3) Is the data in a form that's genuinely suited to being shared? Is it in a flat file, with clear and easy to understand variable names, a good data dictionary, etc.? Is missingness, and the reasons why, clear and evident (the difference between a variable with a lot of missingness and a variable that's a bit garbage is often subtle)?
4) Where do you put it - and importantly, who pays for that, and how long?
The tools for doing this across a whole diverse ecosystem are shall we say not user friendly (if they exist at all), and that is being charitable (and I say this as someone who has written some of those less than user friendly tools). Most of the systems, archives, etc. are not interoperable, they don't share common APIs, much less have some common interface that would allow someone to track all their published data, forget integrate it.
Program officers are going to be desperate for ways to account for this stuff, but I worry that at least initially they are not going to have the leverage needed to get researchers to engage in even the most basic data management practices which would allow them to start solving the problem down the line.
How many files did you upload to archive X? Which project were they for? How many datasets are you expecting to produce for this grant? You have published 10 papers associated with this grant number, where are the datasets that correspond to those papers?
Some investigators can answer these immediately and easily, others will throw a screaming fit that they were even asked for such a thing.
Unfortunately the project seems to have split or evolved into a different project or something. They went through some rebrands, a rewrite, Beaker got abandoned and I lost track of what they're doing now. I think their grant funding ran out. But it was a pretty good attempt at making this stuff easy or easier.
Now this is not entirely uncommon in many research areas; you can get all the current weather and climate datasets with little difficulty as these are shared globally (as is necessary for effective short-term weather forecasting). Anyone publishing DNA/RNA/protein work generally uploads those sequences to GenBank, and that gets updated every two months, so is pretty current relative to the published literature:
https://www.ncbi.nlm.nih.gov/genbank/
However this article seems to focus on "Biomed" and that whole world is where pharmaceutical/corporate influence in academics is off the charts. I'd guess well over half of the researchers (if not over 90%) have some dream of getting their work patented, setting up a biomed startup, and having Pfizer or J&J swoop in and buy them out (they then get a percentage and become wealthy), and their university patent office is likely pushing for that as well (they get another percentage). No way are those people going to share their data, which might very well be slanted or cooked in ways that will make their little entrepreneurial effort more attractive to a big buyer. It's way more like pitching to a VC than doing actual rigorous scientific research in the tradition of open sharing of results and ideas - at least not until the patent is granted. Then you're into whether or not the clinical trial data will be released to the public, and corporations don't like to do that either (as someone might find that their claimed success story is riddled with statistical errors, etc.).
Business and science have orthogonal goals, and mixing the two is almost always a recipe for the rise of fraudulent con artists pushing bogus BS in the name of turning a quick profit.
You'd need to hire the best people with niche expertise. I know from my years in academia, nobody can understand and scrutinize your bullshit unless they work in the same sub-sub-sub-field.
Much of the fraudulent work isn't coming from Chinese researchers, but much of it is.
Fundamentally, 97% of scientists (in this study) refused to share their data because the people paying them don't care. The people paying them also wouldn't care about whatever your billionaire foundation would say, and in fact many would class it as a right wing misinformation operation. Then you'd be blocked from Twitter, downranked on Google, blocked from YouTube and you'd be unable to hire the "best" people because they'd be socially ostracized for joining.
The problems here are really very deep, and they're social in nature. You can't fix it by simply pointing out the problems because everyone who looks already knows what the problems are. They're obvious and numerous. Fixing them requires fixing the incentives but politicians, the media and large chunks of the electorate refuse to acknowledge that research fraud can even exist at all (the "believe science" brigade).
Take the NSF for example. (Pardon a US-centric example.) If you have a record showing that a particular PI has published 10 provably false/fraudulent papers, is the NSF really going to keep funding that person? At some point, the desire to avoid embarrassment and save their own skin may drive the NSF to de-risk by not funding the flagrant frauds.
Imagine if this were a world-class research institute, supported morally and with funding by people like Bill Gates and Elon Musk. Imagine if those people recognize that we can advance science and global progress much farther and faster if we tackle the epidemic of garbage research in academia (which I'd say is over 60%). Imagine running TV ads that say: "No, we are not right-wing nutjobs. There really IS a plague of garbage research."
(In fact, right-wing sponsors should be totally ruled out. As well as clearly left-wing sponsors. We want basically nonpolitical people who are basically trusted, like, again, Bill and/or Melinda Gates, Elon Musk, and so on. So, no Peter Thiel, which is unfortunate, because it would be right up his alley.)
But then you're publishing papers that try to replicate studies and specifically calling out specific frauds and getting their funding taken away, and more broadly arguing for funding agencies and academic departments to stop tolerating the culture of fraud and shoddy work. And you're calling out the worst offenders (in terms of academic departments and funding agencies) and calling on their political sponsors (like Congressmen) to cut their funding if they don't clean it up.
There may be areas of science where the foundation doesn't go, like climate change (maybe race?), because they are too heated. That's OK. It doesn't have to cover everything.
Part of the reason academia is really broken is, people aren't rewarded for proving that results can't be replicated. We need to find a way to incentivize that. The scientific process does not work without it. Also, academia doesn't allow people to just say, "This idea is wrong."
Academia is composed of a million little old-boys-club silos. You don't get into any of these clubs if you are not going to play ball. De facto, you cannot criticize your peers, if you are in academia.
For instance, the NSF has grant-award panels that are composed of researchers in the niche. So if you criticize your peers you literally cannot get NSF funding. Because your funding is literally gated by your "peers" as in "peer review." But if you were gonna be that way, you wouldn't have gotten a tenure-track job in the first place, because tenured professors only want people in the club to become their "peers" and sit on those committees.
I'm concerned I may be kind of overstating my case, but it's hard for me to know exactly how widespread the worst-case version of the problem is. Certainly what I'm describing can and does happen.
You're absolutely on the ball, except that everything is ~100x worse and more hopeless than you're describing :( Far from overstating it, you're understating it.
"If you have a record showing that a particular PI has published 10 provably false/fraudulent papers, is the NSF really going to keep funding that person?"
Absolutely. The problem is: showing fraud according to whom? I can point to dozens of papers by the same authors that I personally consider provably false/fraudulent, funded by the NSF. That isn't going to achieve anything. People can't even reliably get papers retracted, let alone cut bad researcher's funding.
From https://fantasticanachronism.com/2020/09/11/whats-wrong-with...
"If you look at the NSF's 2019 Performance Highlights, you'll find items such as "Foster a culture of inclusion through change management efforts" (Status: "Achieved") and "Inform applicants whether their proposals have been declined or recommended for funding in a timely manner" (Status: "Not Achieved") ... We're talking about an organization with an 8 billion dollar budget that is responsible for a huge part of social science funding, and they can't manage to inform people that their grant was declined! These are the people we must depend on to fix everything."
"with funding by people like Bill Gates"
Bill Gates is one of the primary funding sources for epidemiology, which is more or less dominated by pseudo-science and fake research. The article we're discussing was published in a journal of epidemiology so presumably the researchers polled for data were mostly epidemiologists, and only a few percent of researchers were honest about basic things like whether they'd share their data or not. You can imagine how bad the rest of it gets.
Gates doesn't care. Like all the other sources of funding for public sector research, he wants to be a philanthropist cheerleader for Science™. The act of spending money is the end, not the means. Whether that money is spent well and yields accurate conclusions is neither here nor there to him, as he has too much for that to be relevant.
I mean, you can't get 100x worse than "60% of research is invalid."
> Absolutely
If you get enough pressure on Congress, you can always make changes. I wish you wouldn't be so belligerent.
Society does change, but slowly. You would probably have said slaves will never be free, or women will never vote, or racial integration of schools will never happen.
> Gates doesn't care. Like all the other sources of funding for public sector research, he wants to be a philanthropist cheerleader for Science™.
You don't know that. Unless you've read his secret diary, or something. I suspect he does care. And I think the evidence weighs more heavily on my side of the argument. But you aren't debating the evidence; you're just claiming to know, which you obviously don't.
To make progress here probably need political parties that take up academic reform (or defunding) as a voter wedge issue, and campaign on it for many years. Based on the failure of initiatives that tried to solve these issues so far, I'm very skeptical of any effort that isn't based on some sort of populism. Realistically this means any attempt to dramatically improve standards is going to get tangled up in culture war, as after all, you can't divorce an attempt to raise scientific standards from specific fields and claims (if you do it's admitting that they should be ignored). To get political momentum you need to be able to point to examples of claims that are wrong, and they have to be claims people actually care about.
Re: 60%, that's your personal estimate, right? My impression is that it varies a lot by field. In some fields you don't really get much invalid research at all, it's a curiosity. In other fields 100% of papers are useless because the underlying premises of the field are themselves wrong.
"You don't know that. Unless you've read his secret diary, or something."
I haven't read his secret diary, no. I have talked to someone who worked in epidemiology on malaria research, who described to me how the field is totally distorted by Gates Foundation funding, I've read many of his statements throughout COVID and read a skeptical review of his non-secret book (if you can get past the invective at the start the review is pretty decent):
https://www.eugyppius.com/p/we-must-find-a-way-to-prevent-bi...
... which reinforced the overall impression: Gates is a cheerleader. His approach to find the hierarchically most important people, ask them what they think and then repeat it uncritically, whilst distributing grants to more or less anyone who says they'll make Gates's personal goals come true.
It's also the case that if Gates was reading the outputs of his funded researchers and is as smart as usually claimed, he would have long ago noticed the problems. Yet his book boils down to: what we need next time is way more of all that. He is aware lots of people think Ferguson is a fraud, but thinks that's only because they were misinformed by the press. As far as Gates is concerned Ferguson is great and he repeats Ferguson's defences of his own work verbatim, even though they aren't accurate. As the reviewer points out, it's impossible to believe given what's written in his book that he ever actually read Ferguson's research (I have read it, and his model code, very carefully). Gates' one concession is that the vaccines weren't the silver bullets they were promised to be, but as for everything else - well, he acts as if he's completely unaware that any problems exist.
Now, as you observe, this might be an act. Nobody wants to spend decades lavishly funding people to engage on a noble mission and then one day admit, actually, they were mostly scamming me and we didn't get much out of it. The loss of face would be impossible to handle. Even if Gates did know, we might expect him to act as if he didn't. Still we'd hope he'd find subtle ways to improve things without outright admitting to the problem in plain language. I've never seen any evidence of this.
At least with Gates there's the theoretical possibility he could have a Damascene Conversion and start enforcing rigorous standards. With governments it really does need to become a political issue before anything can happen, as every time people try and improve standards via government, or purely internally inside academia, the new rules seem to be immediately subverted and everyone carries on as before.
I agree that pushing for cultural/political change is a hard way to go given the populism in America. I'm an American. On a personal level, I think the country is so broken that it's not worth fixing, and would like to expatriate.
> Re: 60%, that's your personal estimate, right? My impression is that it varies a lot by field. In some fields you don't really get much invalid research at all, it's a curiosity. In other fields 100% of papers are useless because the underlying premises of the field are themselves wrong.
Yes, it was my estimate. But it's just a number I threw out there. I don't disagree with what you're saying here, but I'm more focused on the structural problems that exist to some (varying) degree in every field. For example, in every field, funding is corrupted. So I wouldn't say there are some fields that don't get much invalid research at all. Outright fraud is not the only kind of "invalid" research. It's still "invalid" if it's honest research but exploring the wrong path because there's some old fart at the funding agency who has a lot of friends who are exploring that particular path, which actually has been played out for 10 years already.
> Blog post about Bill Gates
I appreciate your sharing this. I think it's easy to construct this sort of narrative. That doesn't mean it's true. It might be. I don't know enough to tell.
I really think someone like Bill Gates could see the kind of research validation institute I'm proposing as a "win." I'm not saying all the research he's funding is worthless. I don't believe that, personally. I'm just saying, we can boost the quality and effectiveness of research if we have this kind of institute. Maybe our research process as a civilization gets 2x better. To use military terminology (ugh), it's a "force multiplier." It's not saying all our existing research is garbage. It's picking off the low-hanging fruit, the worst offenders, and thereby improving the signal to noise ratio of research overall. And once you get all the worst offenders, you can look for less low-hanging fruit.
I bet Gates would be happy to admit a lot of the research he funds is "low quality." That isn't a personal indictment against him. He'd probably argue that a lot is "medium quality" and a lot is "high quality." I wouldn't disagree. I'm sure there is some proportion in all three buckets.
As someone who lives outside of America, I really wouldn't be so down on it as a country. I'm lucky to live in a very nice part of the world (Switzerland!) but even so, America is a country and culture I hugely admire. I've spent most of my career working for American firms because that's where the action is, that's where the bravest people tackle the hardest problems. Yes, America is a land of extremes and the lows can be low, but the compensation is that the highs are really high.
"For example, in every field, funding is corrupted. So I wouldn't say there are some fields that don't get much invalid research at all. Outright fraud is not the only kind of "invalid" research."
Ah, I see. Well, I'd say that the funding mechanism makes invalid research possible/easy but doesn't necessarily directly create it. The lights are out but someone still has to misbehave. In some fields there just isn't much incentive to do that, e.g. consider computer graphics or the papers that explore more efficient K/V stores or compilers. You could try and cheat in those papers but why bother? You'd just be undermining any possible future jobs in industry, so I find these papers to be pretty reasonable.
The corruption really seems to kick off in fields that can be twisted into forms of social control in some way, or where there's not much chance of ever getting a good job in the private sector. Social sciences are a great example but public health is the same problem. People see an opportunity to change the world through misrepresenting their science, they see that nobody will stop them, and so they take it. Power is the goal and the apathetic funders are the enabler. You can't change the world or control anyone via compiler research though, so it stays closer to the original ideals.
Now, I agree that if you wider the problem scope to include irrelevant research nobody cares about, then indeed every field has big problems with that. It's probably too much to ask people to care about both invalid and irrelevant research at once though.
"I bet Gates would be happy to admit a lot of the research he funds is "low quality." That isn't a personal indictment against him. He'd probably argue that a lot is "medium quality" and a lot is "high quality." I wouldn't disagree. I'm sure there is some proportion in all three buckets."
Well if you or anyone else can get him to admit that, it'd be a great start.
I was a grad student in CS for 8 years, but left without finishing the PhD. I wasn't in compilers, but that's a reasonable example because I was in some other "niche of a niche." Small community, fairly obscure.
Most of the grad students in my niche really just wanted to be professors. And the way to become a professor was to (a) publish a very high quantity of papers; and (b) make friends with all the senior people in our little niche.
The goal people had wasn't to amass power. It was just to get a tenure-track job. Somebody else in this comment section made a joke about Chinese students. The problem is not at all limited to them. But a tenure-track job at an American university really seems like heaven to someone who's made their way up from the bottom in China, for example. And also to some people from other parts of the world, including America. If publishing tons of low-quality papers is the path to that, and being buddy-buddy with other people, they go for it. It's a "I'll scratch your back, you scratch mine" environment, including regarding "peer" review.
There is a tradeoff between quality and quantity. I was doing empirical research. I couldn't compete with the people doing more pure mathematical/algorithm stuff. They could just spit out papers with some new obscure algorithm (that will never be used anywhere) and a proof of some of its properties. I would have been lucky to have 5 published papers at the end of my grad school career, but a good tenure-track candidate would have like 30.
The senior people, like my thesis adviser, more than enable this kind of behavior. They get grant money basically based on the quantity of papers published. I never understood why my adviser cared so much about grant money. Like, what drove him to put out a super high quantity of crappy papers, to have a ton of students, and get a lot of grant money? What's the point? I never understood it. I mean, he already had tenure. And there were lots of tenured profs I knew who actually just didn't care about grant money and publication count, and didn't do that. Which is great. But you know who all the grant money goes to, and then who has a huge "lab" with lots of students? The ones like my professor who really care about that sort of thing and optimize for it.
I think compilers produces much higher quality than my niche-of-a-niche (which I don't want to name, by the way). But I don't think an area like that is immune to the pressures I'm talking about. It's probably a bigger community (which helps), where the research just has a different dynamic. We could speculate why that might be. But nonetheless I would argue that every area of modern science suffers from the problem I'm describing, to a varying extent between countries and fields and sub-sub fields.
By the way, maybe people go into compilers as PhD students, wanting to go to industry... but the truth is, in computer science, getting a PhD is usually not advancing your career over just putting that many years directly into industry. If your planned route is PhD=>industry, it only makes sense if you want to be in an industry research position and if you care about that more than how much money you make.
My other papers, where I provided no artifacts, generate an average of zero support emails a year but are cited just as much.
The emails are often from first year PhD. students and are vaguely accusatory, along the lines of: "We're having issues replicating experiment X and wanted to give you a chance to correct/review before we publish on it." The cause of the issues is, 100% of the time to date, people not reading documentation or having more than a superficial understanding of the topic. However, I feel compelled to respond so that my old lab doesn't get dragged on twitter or at some conference over something that isn't true.
I barely got paid to do this sort of work when I was in academia and certainly don't now. If I could go back in time, I would have kept my research closed source. The personal benefits are essentially zero and the world would get along just fine if I had never written any papers, so the altruistic reasons aren't terribly compelling either.
The fun could start when you let drafts of that meta-paper start circulating.
When scientists don't publish datasets, it contributes to a culture that allows a small (or not small, depending on who you talk to) portion of scientists can p-hack their way to a publication and ruin the reputation of research for everyone.
At one point, Newton was unhappy with Flamsteed, because Flamsteed wanted to work on his star map instead of Newton's moon observations, but Flamsteed said he needed an accurate star map to produce accurate moon observations.
Data sharing has been an issue in science for hundreds of years.
Suspect they dont really care beyond having it "published".
I think this paper shows that it is just more research theater, meant to distract.
> We did not publish our raw data along with the manuscript because it could be understood that we are publicly shaming authors who did not want to share their data. As for the raw data that were received during this study, we informed our study participants that those raw data will be deleted after being examined and that all data and communication will be treated with strict confidence.
There is absolutely no contradiction here.
But maybe just maybe you shouldn't say that are going to share the data if you are not. How trustworthy is the rest in the article if your Data Availability Statement (DAS) is a lie?
If benefits of non compliance > consequences, don’t comply, else, comply.
That simple, and applies to tech companies as well.
At the end of the day who’s policing this stuff anyway? People only find out when there’s a leak.
For many years, scientific computing facilities were "off limits" to honorable hackers, but I've noticed a persistent air of entitlement from the biological sciences, paired with a deep disrespect for the field of information science.
I have a bachelor's in information science with a concentrations in information security and related areas in psychology and English literature from one of the highest ranked schools of information science in the world[0], followed by a PhD I left "ABD" after numerous peer reviewed publications and scores of conference talks and other informal lectures and volunteer work.
I once saw an entire zoo aquarium's exhibits fog up in my hometown as it switched to brownouts, as a conference's attendees went on to obstruct policy initatives that would prevent the entire state they claim to want to be permanent residents from one day losing power in a similar manner to what happened in Texas.[1]
It's why I started one of my pet phrases when I'd do volunteer work:
Voting is a right, whereas electricity, like driving, is a privilege.
Anyways, sorry for the stern response, but I hate being the guy who just goes "Oh, those results are unsurprising", but some nights I feel like I really need to mansplain what can happen if folks ignore my "crazy" left-libertarian advice.
[1] After a decade of warnings, a weak mandate to "winterize" https://www.npr.org/2021/06/02/1002277720/texas-lawmakers-pa...
I don't understand the connection between scientific computing facilities being unwilling to host "honorable hackers" in "information science" to use their computational power and data being closed.
I don't understand what your "stern response" is to.
I don't understand why you used the word "mansplain" to describe what you're doing when it has nothing to do with identity politics.
I don't know what your "highest ranked school" is. You added it as a footnote but it doesn't seem to appear later in the comment.
I'm not required to explain my art to you.