“Is there a heuristic we might use to identify and flag questionable papers?”
statmodeling.stat.columbia.edu
statmodeling.stat.columbia.edu
Meaning the authors get paid the more they publish?
How? Per paper or career advancements-->> higher salary?
That has been a good incentive to date (I insist, this is maths and there are very few possibilities for cheating in my field, but of course there are instances), as I see it, because this way you incentivate academic professors to really do research (otherwise one can have a truly leisurely life in Spanish public universities once one gets tenure, without any repercussions).
Well, it is a permanent 120Eu per month (usually inflation adjusted, more or less), which considering that the tenured professor salary starts at like 1800Eu per month, is not so bad (at least it is some incentive). You can end up with 5 (or 6, if lucky) of these.
You see: the problem is not the incentive, but the salary itself.
(All the numbers above are more or less with taxes and Social Security already discounted).
Edit: never underestimate the power of saving and compound interest.
Tools like scite (https://scite.ai/) do automatic processing (NLP) of other papers and determine whether a reference relationships supports or rejects the paper.
Also, http://octopus.ac/ did NLP processing on papers to make a breakdown of them and extract pieces to populate the content on their website, but they seem to have removed the content on their Github since then.
For example, I could L1 trust people to evaluate papers and the trustworthiness of other users, and L2 trust others only to evaluate the trustworthiness of papers. If I notice a spurious signal getting in, I can find out who it came from and prune my list of trusted users.
I imagine I'm not the first to think of this. Anyone know of reading material on the topic?
(Journal clubs are groups of people, usually working in the same field, who pick & present a paper to the rest of the group on an rotating basis.)
I haven't seen more attempts at similar things, which doesn't mean they don't exist though. (There were mentions of a similar feature being added when Elon Musk was about to buy Twitter, but just gossips)
An advantage of having open peer review system is also that this kind of attempts could be detected by tools doing automatic processing of scientific knowledge graphs generated by peer review activity.
This, coupled with verification of users, should be able to minimize the risk quite significantly.
“”” The GRIM test is straightforward to perform. For each reported mean in a paper, the sample size (N) is found, and all fractions with denominator N are calculated. The mean is then checked against this list (being aware of the fact that values may be rounded inconsistently: depending on the context, a mean of 1.125 may be reported as 1.12 or 1.13). If the mean is not in this list, it is highlighted as mathematically impossible.[2][3] “””
Source: Wikipedia
Let's say the reported mean is called m and the sample size is N. Multiply m and N together. Ideally mN should be almost an exact integer. Find ceil(mN)/N and floor(mN)/N and see if they're the same as the reported m, within precision limits.
Example: m=6.24, N=13
mN = 81.12 (exact)
81/13 = 6.231
82/13 = 6.308
So there is no integer numerator (for denominator 13) that gives 6.24, to 2 decimal places, so 6.24 is a mistake or a lie.
It's also wrong to call a paper 'questionable' without qualifier. Questionable can mean anything, and some of the most groundbreaking ideas were questionable when they were put in the wild. I suppose the author means technically flawed.
Hence, I'm afraid peer-review is still the only option left.
In my circles (Computational Linguistics), getting your paper approved means that you convinced at least three PhD students with a published paper (or higher, all the way up to Professor) that your paper is good PLUS the area chair(s) considered it worth publishing. All of this without knowing who you are nor what your institution is. And once your paper is accepted it is freely available on the internet forever.
No human activity is free of issues, but to me double-blind peer review is a fairly effective system. And every year the system is tweaked to try and adapt it to newer developments such as the rise of ArXiv. Most criticism I read uses the term "peer review" to refer to whatever Nature, Science, and/or Elsevier are doing at the moment, but just because they suck it doesn't mean that peer-review as a whole does.
This is a recursive definition of scientist "A scientist(t=n) is someone approved by scientists(t=n-1)". If you have a good initial condition maybe it can work for a while, but it seems that time may be coming to an end.
> Pournelle's Iron Law of Bureaucracy states that in any bureaucratic organization there will be two kinds of people: First, there will be those who are devoted to the goals of the organization. Examples are dedicated classroom teachers in an educational bureaucracy, many of the engineers and launch technicians and scientists at NASA, even some agricultural scientists and advisors in the former Soviet Union collective farming administration. Secondly, there will be those dedicated to the organization itself. Examples are many of the administrators in the education system, many professors of education, many teachers union officials, much of the NASA headquarters staff, etc. The Iron Law states that in every case the second group will gain and keep control of the organization. It will write the rules, and control promotions within the organization.
There is no formal definition for a "scientist" and likely will never be. Definitely a scientist is not "one who has a paper accepted".
Here the commenter is just claiming that in order to publish a paper in a specific journal you do need be accepted by a handful of people who have previously been accepted in the same journal. Barring the obvious chicken and egg problem which is easily solved, it does not seem to be like something which "is coming to an end".
You're simply reformulating what peer means in peer-review.
What other way would you suggest? Layman-review?
At the moment people do things "for living", in those usual bureaucratic institutions that all institutions become sooner or later, there are initiatives at power that work against the initial idea behind that institution.
That's why progress in science occurs at best "one funeral at a time", but usually not before a whole generation bites the dust.
This plus bureaucracy seem to almost killed larger parts of science by now.
We have collusion rings milking the system. We have broad pure esoteric nonsense "science" circles. We have more and more utter garbage and scam. And we have holy churches that will send the inquisition on you if you dare to disagree.
I don't know how to fix that. But I guess the initiatives, and especially the cash flows would need some re-engineering. But how concretely, I have no clue. Throwing away everything you have is of course almost never a good idea. But some things need to change. Change in a radical way, I guess.
1. The validity of the results
2. The novelty of the results
3. The importance of the results
Papers are routinely rejected in peer review not because their methods or results are suspect but because their results are not seen as impactful enough to justify space in a conference or journal. This sucks and introduces all sorts of nasty biases into the community. I'd love for these to be disconnected so one could demonstrate that their paper had been read by other experts and found to be within reason without having to worry that one person on the PC who just doesn't think that a certain research direction is valuable will sink my career.
But historically when the world was smaller, people writing papers were more likely to "speak the same language," have experiences in common that fostered clear communication, etc.
It's a big, wide world. The sheer variety of experiences these days introduces complications never before seen.
After this I use the following heuristics:
- => Lower belief
+ => higher belief
effects multiply, of course, but each is few %.
++ => sufficient details present that I think I can attempt to replicate without contacting authors (this is rare)
+ => code used is available in public repository (subset of above)
+ => published in med-to-high ranking journal
--- => published in obscure journal
-- => socially desirable outcome
+++ => socially undesirable outcome
-- => non-US, non-EU university or nat. lab
+ => national lab
I mostly read papers in my field (exp. physics), of course. But I began reading a bit more in medical and psyche a few years ago. My views are influenced by books like "Medical Nihilism", and "Science Fictions". Reading COVID papers was also caused me to revise my prior-probabilities downward.
--- => journal declares its ISSN prominently
---- => journal prints its ISSN on every page
Also could you please expand on the thought process behind this heuristic?
Yes. At least, that was how I interpreted the grandparent post's scale and how I tried to match it.
> Also could you please expand on the thought process behind this heuristic?
Have... have you ever encountered these journals in the wild? They're trash. But they come up in search results because search engines are dumb, and I don't know the "good" journals in every field, so I can't filter them out myself. Until I click. Then I can!
Usually what's in them is papers from researchers based in countries that have a publication count score or bonus. They'll take some overcomplicated homework problem and solve it or set up some overcomplicated scenario and show that it has property X or have a few diagrams or photos of their current overcomplicated-senior-project research and show that it will do Y when they finish it. All worthless. There is no insight whatsoever.
- Only a tiny minority of your beliefs have any impact on the outcome of your life. Perhaps your concept of [insert social or political policy] or [scientific topic] or [cultural idea] might have been gamed by someone at some point, but ultimately it will rarely matter whether you even hear of these concepts to begin with or not.
- Of the beliefs that remain, these often become pretty defensible through practice and results.
Put another way, there are plenty of barely-literate people who still have lead fascinating and worthwhile lives even in the modern era.
The proximal desirability bias is that of publication gatekeepers. Biases of society at large also can play a significant role for certain topics.
Which is going to have an easier time getting to publication?
If publication is in the Nazi Journal of Nazi Naziness, the first one will be easier to get published. If for whatever reason you're reading journals there, the second of those papers is the one you should expect to be unusually strong.
If publication is in Nature, they probably don't have a lot of Nazis on their editorial team. If that paper got published there, then probably something about it is unusually good somehow despite the unpleasant conclusion.
Caveats:
1. With a conclusion as horrifying as the one in your original question, though, I think it's fair to reason as follows: Even if the paper gave tremendously strong evidence for genetic differences between races, there's no way any sane editor would publish a paper that says the "... and should be culled" bit. Therefore, this paper was almost certainly railroaded through to publication by a crazy editor. Therefore, the fact that it made it is not evidence that anything about it is any good.
2. It's OK for your expectations of a paper's merits to be affected by how plausible you find its conclusions before you read it. Many "socially undesirable" ideas are that way at least partly because most people find them highly implausible, and if most people do then there's a good chance you do too, and if you do then it's fine for that to lower your expectations for the paper.
The question is essentially whether it is possible to have a structural procedure to evaluate the semantic validity of a statement.
Is there outright fraud, or are they just dull/pointless papers? if first, yeah that's questionable, but a lot of these example papers look more like they're dull, rather than committing fraud.
We should rather tear down the awful system instead of playing whack-a-mole with academia's newest outcropping.
There isn’t one “system” that can be torn down and replaced. A lot of these papers are coming from groups and institutions in foreign countries that are basically trying to role-play as researchers so they can blend in with actual researchers and try to reap the reputation and career rewards.
It’s not really centralized. This is akin to suggesting that we need to tear down the entire “e-mail system” because spammers are abusing it to send spam appearing as legitimate email.
Why is it a problem? Because these contribute nothing but noise and confusion to the field and make it more difficult for actual researchers to sort through real research. There is no reason to forgive these faux-researchers for trying to extract money and prestige from a system without making actual contributions.
The idea to “tear down the system” doesn’t even make sense unless you propose something different. Are you suggesting we stop rewarding anyone in any way for research? That would remove the incentives, but now we have a problem that research isn’t rewarded at all and real researchers won’t do it because they need to eat.
And to be honest, once it was known to be used in this way, it would be simple to make fake data comply with the law.
Big picture, the goal is not to catch every fake. It's to (1) catch enough so that smarter fakers learn to submit their stuff elsewhere (vs. to the journal, conference, or institution in question, which really cares about fake detection). And (2) dimmer fakers are mostly caught ~promptly.
Statistics applied to not fully observable stable systems in the modern day alchemy.
Simulations have similar flaws - once the model does not match reality perfectly one would get an arbitrary wrong results.
Look at all the COVID related simulations and statistical models - all of them are not even close to reality.
That's the reason we still have these overpriced journals. The "publishing" part is easy now, it is just a matter of putting a pdf on a server, journals are selling reputation.
If we are paying for reputation and the journals reputation is a function of the reviewers skill, maybe there is a way to cut out the middleman and pay for skillful reviewers instead.
H2. If there's no git repo with the complete code to reproduce all experiments, then you can assume it's bullshit.
That seems to be the actual heuristic, but it would not fly politically.
Paradoxically enough, it would be the opposite of science.
Because in science it's totally forbidden to accuse someone of deliberate fraud. You always must find a face saving way for the authors when dismissing their paper, like "bad statistics are being used".
What does a post-writing civilization look like? Is it reasonable to imagine scientific progress without the burden of having to battle the computational capability of sufficiently motivated actors who can easily manipulate authored information spaces?
Writing is essentially a massive attack vector to the collective human consciousness prone to Sybil attacks at an alarming saturation point in a post gpt3 society.
It feels like we need something post-writing.
Computational resources are useless because you cannot fake papers with trusted authors. Trusted authors won't publish a bad paper because they want to keep their reputation.
All an adversary can have is something like a network of fake articles. Unlike web pages, scientific papers are not opinions. The scientific core group can check if papers are relevant and true. The scientists can limit those checks to authors whom they have met in person to minimize their time investment.
The only way reputation networks maintain durability is if they stay physically constrained and limited by physical contact. Any system built externally and allowed to represent facts and truth will be compromised due to computational superiority. It must not be a network hosted by a computer. It must run on culture and physical reality itself.
Additionally agents of deception are regularly embedded in networks to encourage malicious behavior. Spies are pretty common.
See: https://m-cacm.acm.org/magazines/2021/6/252840-collusion-rin...
I think the key bias we're contesting is that I'm arguing computational superiority allows creating networks that are indistinguishable from truthful ones because they have the resources to have the "longest version of history" to act as the authority. If the history is not there, it can be fabricated and embed enough "covert actors" to maintain a narrative.
Then you also generate useful idiots who don't know they are being used to propagate misdirection or what have you. They become unable to detect any "obviously fake content" because it doesn't require fake content to conduct deception.
Very often, it is real information indeed that is most able to misdirect attention.
You don't know. Evolution will decide if your seed group was good enough.
>how do you guarantee it is the person in question?
You can't. You have to role back if an identity is compromised
>It must run on culture and physical reality itself.
Yes. That's the difference between science and writing in general. Science is meant to be checked with reality.
> think the key bias ... the resources to have the "longest version of history"
History is secondary if you verify all statements. History should only grant preferred treatment.
>Then you also generate useful idiots
That's outside of the trust network. You can help them by making education easily accessible.
>Very often, it is real information indeed that is most able to misdirect attention.
That's when the game becomes interesting. You can also have truths that are not verifiable etc. It all depends on what the participatns want to know.
> Yes. That's the difference between science and writing in general. Science is meant to be checked with reality.
When we live in a world that is increasingly using simulations for verification instead of reality itself, it seems as it becomes an ever increasing attack surface for controlling and manipulating what reality and truth actually is. The scary quote "Science Academics" are largely publishing statements based on mathematical models that are probalistic or theoretical with no physical extant items.
When the burden of verification is solely on the individual, yet the individual is increasingly pushed in the direction of having to believe things as facts rather than experiencing them, then it sort of compromises that integrity, no?
It seems the only way to have certainty is to be a witness and have experience over a set of events, that is the only thing that can be understood to be true (and only to you), and that is what contributes to certainty. All else is belief and unknown risk to deception.
tl;dr science is heading towards imaginary non-extant things that happen to predict physical things but they can only be verified non-physically and since certainty require physical experience then the security of the system is at risk of being corrupted.
tl;dr;dr what to do about simulations and models posing for reality?
Also interesting:
> That's when the game becomes interesting. You can also have truths that are not verifiable etc. It all depends on what the participatns want to know.
Can you expand on this? You're saying something incredibly poignant and I'd like to hear more.
It is or will become indistinguishable from thinking. We can ask the same question about our thoughts.
The science trick is that it requires models that can be falsified in reality. If you make predictions about simulations, then you don't make science but math.
The problem about simulations is that they make verification equally or more expensive than the simulation itself. It's the end of 'scientific cooperation'. Why verify a simulation when you can make your own?
>>It all depends on what the participants want to know. >Can you expand on this?
I wouldn't call it poignant, but here you are: As you wrote "Very often, it is real information indeed that is most able to misdirect attention." - there is a process that makes choices. Truth is a tool to reach the goals of that process. We don't care about the scientific method in itself.
Most people don't need truths to reach their goals because there is kind of a market of facts. Some facts are just known, without scientific backing, and people can just choose to use them.
When there is deception there is also cooperation. The misdirecting information is only there because somebody else wants to achieve something. Maybe it's easier to cooperate with them about their goals than handling the misinformation?
The fundamental issue is that there is a cost to deception that I reason is greater than the benefit to the ones gaining from the cooperation and execution of the competing trajectories of narrative control. It essentially breeds psychic cancer. It incentivized all the behavior that leads to distrust then inevitably chaos.
So the only way to avoid that is no competing factions. How do we transition to a non game theoretic civilization? Can we achieve it without assimilating to silicon?
And related to your points on truth which I agree with, is the classic Greek comparison between Techne and episteme. It translates also to understanding and applying facts demonstrating “actually getting it” vs just regurgitating the fact market. The Tao te Ching has some anecdotes on the danger of knowledge as it leads to people thinking they know something but all they really have is the ability to echo or parrot. I guess you need your messages delivered in a network one way or another.
Thanks for the thoughts. Would enjoy hearing more from you if you’re on a discord or similar.
I am not prepared to continue this outside of hn. How could I reach you later?
Regarding the deception: We also decept ourselves. So even if you go silicon, how do you know that that silicon is without flaws?
- precisely re: silicon. One can never be sure about external systems outside the self it seems, and even that is potentially at mercy to whatever can manipulate the senses. However there is an increased risk of deception / corruption by introducing a universal thing that only 0.0001% or so know how it actually works or could build one end to end.
It doesn’t necessarily mean to abandon silicon, but it does mean that it can’t ever be trusted. So designing systems around that and probably minimizing dependence. I can see this happening with “cold computation” that is very specific in purpose and low in cost. Something like the green array chips. But that does have the potential to swarm. All I know is that if silicon replicates it’s going to be in conflict with dna.
It seems prudent for a dna based life form to stick to dna for truth. All else is risk of assimilation or some form of deception at large unable to be detected.
Much more to unpack there though.
If there is a heuristic it's quantity. If a laboratory, person, or country is producing an inordinate amount of papers then perhaps it's a good idea to sample and test some of those.
Yes I realise this is an assignment and not an exam but Scheduled Assignment Optimisation doesn't really fit. Grumble.
The ranking should assign the score of a paper that was debunked to the paper that debunked it - that would provide incentive to replicating famous sketchy-looking papers.
That's questionable.
But I am not allowed to value my privacy and security and ask questions of Columbia.
I was under the impression we knew this method for a pretty long damn time by now.
This is confirmation bias, and has been used for terrible reasons, as well as to great comedic effect (see Sea Matheson fat studies)