The phrase "no evidence" is a red flag for bad science communication (2021)

Having evidence for untrue things also turns up naturally because of statistical mirages.

"No evidence" is a phrase that gets used in untruthful ways. There usually is evidence, of everything. True, false or irrelevant - there will be evidence of it in some form or another. The question is what standard the evidence reaches, not whether it exists.

Sometimes “no evidence” means that we haven't found compelling evidence yet. Some people are desperate and want all the studies to be done immediately and evaluate the risk if the theory is real or not.

But for scientists the desire to reach certain outcome is actually counterproductive as it can introduce bias. Slower and less passionate process can lead to better results.

For example:

1) We had no evidence of covid being airborne and then we found this evidence.

2) We had no evidence that masks help and then we found no evidence.

Two different theories, two different outcomes. Covid is airborne and it changes our understanding (however, we gradually realized that it is impossible to limit the spread and all the measures ultimately were useless). And that masks most likely had very little effect.

The reporters could write better for lay public explaining that “no evidence” means that currently we don't have evidence but it could be found later or that “no evidence” is actually that we have a lot of evidence that is indicating in some other direction and the chance of new evidence that rejects those findings is smaller but still could happen.

When I worked at Apple retail, we were trained that if we were asked a question that we didn’t know the answer to was to say, “I don’t know, let’s find out together” and you either did some research on Apple’s site or asked someone who may know with the customer.

Because nobody wants to say that they don’t know the answer…yet (and that’s key) they take the current working theory and say there’s no evidence to the contrary.

Question: If science communicators use the colloquial meaning instead of the technical one, are they bad science communicators? Or especially good ones?

I think it's the latter. I also think if you're smart enough to know the technical definition, you should be smart enough to make this translation without making a stink about it. And given that half of the given examples are about COVID, there's some chance that the author has an axe to grind.

> If the story is that nobody has ever investigated snake oil, and you have no strong opinion on it, and for some reason that’s newsworthy, use the words “either way”: “No Evidence Either Way About Whether Snake Oil Works”.

You spent how long telling me that "no evidence" is bad writing, and your solution is "no evidence" + weasel words? That's even worse. Just recognize that the thing you have no opinion on is not newsworthy, and just drop it. That's right, less writing is good journalism.

The greater problem here is the fear of authority. If there is a position to present, then present it directly. If the position is that snake oil doesn't work, then just say it. If you aren't a good enough authority, then find someone who is.

This is where scientific communication has ultimately failed. The strongest authorities we have all use the weakest language that they can! By refusing to represent a strong position, we have left a void. That void will inevitably be filled by the least scrupulous authors. Whatever fills the void will be implicitly held in awe by the lack of direct criticism we had constructed around the void in the first place.

Refusing to participate creates an inverse cult. An anti-cult where everyone agrees to never share what they believe. It spreads its anti-message just as quickly, and cements itself just as divisively, as any evangelist organization does. We must recognize that the anti-cult lives on equal footing with the cult. The true power of each exists, not in their core truth claim, but at the boundary of argument. Argument is engagement, and engagement is fuel. To avoid engagement is to starve.

I think "no evidence" is problematic and I think Bayesian reasoning is fundamental to science reporting. Maybe it doesn't even go far enough though, sometimes you need domain expertise.

Consider:

● Pesticide A was evaluated by the FDA and was declared safe in USA.

● Pesticide A was evaluated and banned in Europe.

● Pesticide A's molecule is similar to pesticide B, which is declared unsafe in USA.

● Of the last 10,000 pesticides to be declared safe by the USA, it actually turns out that Y% of them end up later being declared unsafe or only safer at a lesser dose.

● Pesticide A has only been evaluated by giving a 10,000x dose and looking for effects within a 1-week period. It has never been evaluated by giving a natural dose and looking for long term effects, nor effects on pregnant mothers.

● State that uses Pesticide A (but also many other products) has a 10x elevation of very rare cancers

This is a very realistic picture of the type of datapoints you might start with.

---

Or consider a case like this:

● Fighter pilot says he saw an alien spacecraft accelerate 100x the speed of a rocket with no noise, or visible propellant

● 50 other people say they saw strange things in incidents too

● There's good reason to think that faster-than-light-travel violates causality

● Scientists haven't found any radiowaves that indicate intelligent life anywhere they have looked

I think the point is: I don't see how you get good answers without a bunch of really good bayesianists in the FDA or in Science

Science is a process, not an authority. We should not "trust the science". We should analyse and question it for ourselves.

I think the essential problem with the style of journalism from the article is that it reduces the substance of stories into an appeal-to-authority. It is not a trustworthy authority - there is plenty of history for people employed as scientists to make assertions that now look ridiculous.

Journalists should put more focus to the arguments and less to the conclusions.

A lot of the time the effect size is low (and will probably disappear with better controls) so leaving this out and leaning on the term "significant" is good for clickbait.

I get similarly annoyed by the use of the term "significant" in statistics, which is a technical term in the field, not an expression of the importance or magnitude of the finding.

It's very easy to satisfy the human mind. It's very easy to stay satisfied. The human mind wants to be satisfied, it does not want the truth. Consult any list of fallacies to see the difference between facts and satisfaction. The true nature of the universe is very complex and hard to understand.

Thus we have science. Science, as a practiced art, is a way to delay satisfaction of the mind. My mind will not be satisfied about question X until certain questions are answered.

(Bear with me) It can be very clearly observed that the Sun goes around the Earth. Every day I can observe this. I am satisfied. It is ONLY when other questions come up, that I must become unsatisfied and make more observations, and come up with a new theory.

I may observe the phases of the moon, and see that the Sun shines on the lunar surface from a certain angle. This observation is not consistent with the Sun orbiting the Earth, as I know that the Moon is closer to Earth than the Sun; thus the daily Lunar cycle should match Earth's cycle. My observations are not consistent. (There's also a lot of other evidence, but I like this one).

In fact, it is just an illusion caused by the Earth spinning round.

====

Science communicators need to talk about tests a lot more - what mental model is an expert using, and how do they test that model? How is evidence incorporated into that model to update it?

We have a lot of similar shorthands in journalism, the one that comes to my mind immediately is the phrase "the markets reacted", even though there is no living, conscious thing which is "the market" and can have a reaction. Similar to "no evidence", this phrase has a meaning which varies in detail but which is superficially the same and it's up to the reader to understand it.

If some random person in 1970 jokingly were to have said "Yeah, I killed JFK." That wouldn't have been considered "evidence" of that person's culpability in JFK's death. But if there were other evidence that established the plausibility of that person having possibly killed JFK (worked at the Book Depository, hated JFK, owned a rifle, expert markmsman, etc...) then the statement becomes a kind of evidence-- a confession.

So the status of something as "evidence" and not merely data can change based on the context.

Here's another example. There was water all over the floor of my kitchen. Initially, we suspected that I may have spilled some water while filling a humidifier. There was evidence for that in the form of my own memory that I had spilled some, a few hours earlier, and there were water streaks down the side of the humidifier. This seemed to be supprting evidence. But then my wife remembered that she had filled a pan to check for leaks in it and then forgot all about it. That pan was now empty. In light of that evidence, which was very compelling (we even verified that there was indeed a leak in that pan) the earlier "evidence" STOPPED being evidence at all.

And this ambiguity, imho, is exploited politically all the time.

I think Scott means to wrap that up in the second bullet point ('we have evidence X is false'), but of course, there are important distinctions there. Of which journalists may not be aware, or may not spend much time on.

Words and phrases can have different meanings in different contexts, and that's fine. It sort of reminds me of debates over the term "proof". It's perfectly legitimate to say that something has been "scientifically proven", as long as we understand that we're not talking about proof in the mathematical or logical sense, but in a probabilistic sense.

But with science, we don't actually _know_ that to be true unless we look.

This is an important distinction because in popular culture, 'no evidence' seems to get a lot more headline space than 'we have an untested hypothesis'.

Over the past decade, it feels like the aspect of curiosity and hypothesis have disappeared from science, which has become much more institutionalized, compartmentalized, and formalized. It's 'peer review or gtfo'.

It is not literally correct to say there's 'no evidence'. We don't know! There could be! 'No evidence _yet_' at the absolute minimum.

Coming from academic physics background, the state of medical research boggles me. Let's start from so called "meta studies". Why are they needed? You'd think that if the individual studies were done carefully enough, a single study would be enough to establish a fact. Instead, you see meta studies with sentences like "we examined 142 studies on X for Y, out of which 76% showed statistically significant improvement". If we assume that the majority "vote" here is the correct result, what the hell did the 24% of the studies do wrong? Why aren't the people who made them expunged from the community for doing questionable research?

The meeting concluded ;)

https://allthingslinguistic.com/post/45448571632/the-opposit...

These signal words act as a silent, passive test of competence in that once you hear them, you can be assured of the speaker's ignorance and discount their opinions accordingly.

Instead I keep a corral of trusted communicators on X. Since I’ve vetted them I can trust their takes a bit more.

Edit: since people seem to be interested. As a practical example, when that latest high temp super conductor news came out I immediately had two or three different deep dives on my feed with balanced takes and solid explanations.

Of course there is abundant evidence that herbal/natural remedies work: thousands and thousands of years' worth, but said evidence has systematically been supressed, destroyed, and denied by that which came after it (many times, on behalf of the Catholic Church which sought to erase paganism in all its forms, which included herbalism and natural healing.) For example, the Aztecs used the poinsetta plant as an antipyretic, which may explain its quite widespread cultivation by the time that the Conquistadores arrived, who subsequently used it as a symbol of Christmas.

But it's hilarious to read the medical "experts" of Wikipedia hold their breath and stamp their widdle feet as they insist that stuff doesn't work, because nobody's dissected and tested it in a sterile lab over the last 50 years.

I hate that phrase. It implies to many people that any level whatsoever of X is a danger to you.

Some examples: "There is no safe level of alcohol consumption" and "There is no safe level of lead".

These both may be technically true. That even one beer a year is worse for your body than complete abstinence. Or even a single fishing weight making incidental contact with your hand is worse then never having touched one.

But c'mon. The amount of risk posed by very low levels of consumption or exposure is so minuscule that we can safely (hah!) declare it okay.

I've seen people calling for lead to be banned from roofing flashing applications, and total bans on alcohol, citing these statements that "there is no safe level...".

You give me a reasonable definition of "safe", and I bet I can determine the safe levels of each of those things. The safe level of alcohol consumption is not 0.0000 units. It's somewhere above that. Same with lead exposure. You can have minimal exposure that results in no discernible difference in your quality of life. Where's that line?

Yeah, it's going to be impossible to "communicate" with someone who filled their ears with concrete and is shouting as loud as possible "I think you are the devil (literally)"

This has nothing to do with science communication. Entire swaths of the American public believe "My ignorance is as good as your evidence", or worse, that if they don't personally understand something, then it can't be true.

Quick example, we had creationism before 2010, after 2010 something even worse(terraplanism) appeared.

First of all, the authors of the declarations in the example headlines are not "scientists and journalists", they are authorities and journalists. People like CDC officials or the director general of the WHO are of course scientists, but they are not speaking as scientists in those news pieces. They are speaking as politicians.

Secondly, I don't think they used the "no evidence" template so much with COVID because they were "fundamentally confused". Most of the headlines in those examples are clearly politically motivated: authorities wanted people not to panic, to keep working, and above all, to keep consuming. Hence the proliferation of "no evidence" headlines for things like the virus being airborne (inconvenient for bars, restaurants, shops, etc. - in fact one headline explicitly mentions bars and gyms); kids spreading the virus (inconvenient for parents taking their kids to school and therefore being able to work as ususal), etc. Each and every one of the headlines was very convenient to keep the wheels of "business as usual" spinning, what a coincidence, right? So no, they were not "fundamentally confused", they just were trying to bias the public towards a certain direction, and a great way to do that is to use the "no evidence" narrative which, as the post author explains, is ambiguous so no one can accuse them of outright lying.

Note that I'm not saying there aren't examples where such headlines do stem from confusion. I just don't think it's the case of the particular examples in this post.