How do we clean up the scientific record?
academic.oup.com
academic.oup.com
Quantity of papers is #1 and citations is a distant #2. People will outright tell you quality is rarely the goal and that unique results face tough publication odds.
Peter Higgs recently mentioned that his department looked to fire him before his Nobel prize, and his lack of interest in filling the metrics.
This is just an optimization problem that has favored "exploit" for many years, even when "explore" is the direct mission statement of many of these institutions.
The academic research has lagged our commercial developments and continued to publish things that are commercially available in our systems... which use the same code-base and algorithms to demonstrate it. When we founders were at the big academic institute we won many academic awards for this tech, published the algorithms and methods.
One could make many arguments for training people or expertise at said academic institute or whatever but instead these publications are presented as "novel, unpublished etc". Just not true, and the licensure of this technology means that they need to move forward. So far 5 years later they are not.
Their public press releases deliberately do not mention our industry awards because if they did, they would not have much to publish.
It's useless to reinvent the bicycle only if the workings of bicycles were already published. No one cares what inventions are hidden in someone's desk drawers or in black-box products, we care about knowledge that is accessible to the community.
That's if you can get a journal to publish your results, isn't it? I can imagine a reviewer recommending a strong reject on the grounds that a submission is already "well-known" by virtue of having a commercial pendant available. Reverse engineering is impressive, but alas provides no novelty.
From what I can tell, they are struggling to pay people given the 3x base salary overhead at the academic institute.
They haven't been pushing out free tools, haven't been demonstrating in new scenarios, etc. They continue to publish things that are in the open literature which work for corner cases. The company has expanded the generality of solutions, which they are not innovating on or publishing.
I would also say they have more money than they can spend and have delayed projects over 2years because they don't have people to lead them (getting back to the personnel problem).
I'm just disappointed in them as an academic institution, deliberately missing their own mission statement and refusing to cite publications before them, and feel the 3x overhead is a waste... at the very least they should pay their people better.
Such as by assigning a quality score, based on sentiment or some other parameters, to each citation.
Edit to add: some research is so niche that only one or two groups may be able to work on it due to equipment, resources or training/talent—in this case the independent body should be allowed to audit the results of such labs. Maybe scientists may game the index by saying "my competitor's results cannot be reproduced", in which the lab who this is claimed against may file a petition to the independent body to resolve the matter.
Tragedy of the commons is a predictable outcome no matter how smart the agents are, because they're each acting in their own best interests.
able to interrogate the information supply chain . . . . You see it in the media and that comes from, a government study, that's based on academic study, that's based on a data set. . . . have a actual academic supply chain where you can trace the etiology, the origin of a fact or an assertion all the way through the literature. And there's a famous paper that actually did track something like this all the way through the literature and found, it was just something that just got repeated with some like medical nostrum that actually didn't really have that base of evidentiary support where when you track all the back, you couldn't find the brought table.
(for example, when a piece of legislation is widely reported on, I usually can manage to find and actually read said legislation, which can be quite informative as to whether said reporting is accurate)
it would be interesting if the entire lab notebook record was available to such an AI and cryptographically signed and dated. that would potentially prevent people from doctoring images and data, as has been recently alleged on multiple occasions.
it would be interesting if that AI was trained to do causal inference and do-calculus, and every paper published generated an incremental update to an automatically constructed meta-analysis and systematic research review.
keep going and make a large body of real-world e.g. medical data available to the AI, and every interaction could be an addition to the clinical trial data. and then make the AI able to generate new hypotheses and directions for experimentation to test those hypotheses.
https://www.science.org/content/blog-post/faked-crystals-and...
Since many of the journals (much less authors) don't seem to have an incentive to correct the record, perhaps journals need to be banned, if they can't police obvious frauds.
When I working in R&D and a paper was published with an unexpected result, the automatic response was to look at the lab who published it. Reputation matters and some labs were rock solid, and others, shall we say, "played lose and fast with the data".
And even if it was published by a reputable lab, it was assumed it was probably an anamoly, or some measurement error, or some other problem until other reputable labs reproduced it.
So I guess the solution is for scientists to stand fast with their skepticism. My PI told his students "your job is to punch holes in other people's work". That was always encouraged. Question everything - and if the scientist hasn't shown they already checked that - assume the results are bullshit.
Only after rigorous examination and repeated validation should we believe any science. Not because scientists are making things up or lying (though some are), but because even the most honest scientist sometimes misses something or makes mistakes.
> classical model tested over millennia
Peer review as standard practice is less than a century old. Scientific journals are less than 400 years old.
[9] https://link.springer.com/article/10.1007/s10838-022-09607-4
In particular, the first issue pointed out are misaligned incentives for these corrections (think of wikipedia spam; kernel contributions for trivial spelling mistakes or adding <3...). Or the lack of recognition for doing this work. Or author rights, when the "corrections" introduce quite radical changes (e.g. think of policy articles). "Just make it a wiki" will not magically fix these issues.