Hacker News Reader

Comment by radford-neal | Hacker News Reader

Here's how they handled the exposure vs. control group:

Mice in the CO2-exposed group were housed within a semi-sealed, custom-made 393-L exposure chamber... the exposure chamber was also fitted with two 80-mm fans that ran continuously... Mice in the control group were housed in the same type of cages in the same room, but not within the exposure chamber.

So the exposure group were in a semi-sealed chamber where other toxins might have accumulated, and were subjected to continuous fan noise. Whereas the control group were not in a chamber and were not subjected to fan noise.

I don't think one can draw any valid conclusions from this experiment. They should have put both groups in the same type of chamber, both with fans, but just not introduced CO2 into the chamber for the control group.

View on HN

I don't really see how this hand-wavy dismissal invalidates anything about the study. Both groups of mice were in the same room, and while they would be exposed to somewhat different levels of noise due to relative distance to the fans, the difference would be negligible. Same for the toxin buildup, the very reason the fans were present was to circulate the air, which was constantly both monitored and regulated. This was a core part of the whole setup and the main focus of the study.

Unless you find it more likely that the small background noise difference had a starker effect on the lungs than the mixture of gases going through the lungs constantly, I don't really see how any of the criticisms hold.

The fans were to circulate air within the chamber (because they worried, I think incorrectly, that the CO2 might sink to the bottom). There was nothing to stop the smell of urine, and other volatiles, such as outgassing from plastics, from accumulating within the chamber.

There's no reason to think the difference in noise was negligible, and noise can have a big effect on things like stress levels, which can have wide-ranging health effects.

Plus, there may well have been other effects of the chamber, such as different light levels, which could also have unknown health effects.

It's absolutely fundamental for an experiment like this to try to match the exposure and control conditions as closely as possible, aside from the one variable you're looking at. They failed to do this, so their results are worthless.

> There was nothing to stop the smell of urine, and other volatiles, such as outgassing from plastics, from accumulating within the chamber.

The paper states that the substrate and changing schedule had been shown to keep ammonia to low levels.

> There's no reason to think the difference in noise was negligible...

Actually, there is. A good 80 mm fan can push more than 10 cubic feet per minute at less than 10 dBA. That's far quieter than a human whisper or rustling leaves. It's comparable to the sound of a pin dropping. If mice are adversely affected by that, I don't see how they could survive the wild.

> Plus, there may well have been other effects of the chamber, such as different light levels, which could also have unknown health effects.

That seems like pure speculation.

What about things other than ammonia?

And did they have a "good" fan, or a bad fan?

The point of making the environment for the exposure and control groups be as near identical as possible is to equalize not only the factors you thought of, but also the factors you didn't think of.

It's the idea that there were no important differences between exposure and control groups, despite their obviously different living environments, that is "pure speculation".

> What about things other than ammonia?

Certainly possible, but it seems more likely that the control group would have been exposed to something harmful from the lab environment air versus the CO2 group that was getting medical air.

> And did they have a "good" fan, or a bad fan?

The cheapest 80mm fan I can find on newegg.com is a $3.49 Zalman[1] that is 20-23 dBA. That's comparable to the sound of rustling leaves and far less than the sound of a quiet whisper (30 dBA) or a babbling brook (40 dBA). Someone would have to go out of their way to find an 80mm fan that is so bad that it would be louder than a natural environment.

> The point of making the environment for the exposure and control groups be as near identical as possible is to equalize not only the factors you thought of, but also the factors you didn't think of.

I completely agree. I don't recall seeing a paper I couldn't poke holes in though. The fact that their methods were imperfect does not make the study worthless. They found a modest effect for the given methods. It suggests there may be merit to follow-on studies, which would likely use different methods. That's a useful result that has added new information.

[1] https://www.newegg.com/zalman-f1-led-case-fan/p/N82E16835118...

It’s too late to edit my previous comments, but I should mention that I actually agree with your statements about not being able to draw valid conclusions and that they should have used the same kind of chamber; however, I believe the methods were reasonable enough to provide some value.

The point of control groups is, in effect, to not have to have the conversation we are having. It is incredibly glarring to have such uncontrolled differences between the two groups. I personally tend to believe the results are "correct" and "logical" and "expected", but this belief of mine is how confirmation bias works, hence I can not relly on this experiment exactly because of the reasons OP mentioned.

What does fan noise have to do with lung growth? What “toxins” are you predicting would accumulate? This comment seems insightful on the surface, but does not make much sense in the context of the experiment. I would say they used a valid control here. There are plenty of better reasons to be cautious on their conclusions.

Personally, upon skimming the paper, I found it curious that male mice never saw significant effects in these experiments whereas females were weakly significant for only a few of these. Sometimes opposite trends were observed between males and females. That could be due to the difference in sample size (30 female vs 23 male). It also might only be affecting female mice, which would be unexpected (for me). But the differences and effects are not strong, and I am not at all convinced.

I also don’t like that they separated males and females here because I would not expect there to be differences in lung development between the sexes (do we have different lungs? I am not a physiologist so I definitely could be wrong on this) and I try and avoid designs that require two-way ANOVA.

Anyway, no need to bash the cage design, which is misleading I think, when the data themselves are as weak as they are.

while generally agreeing the experiment setup is far from perfect, I can imagine a number of physical manifestations of constant stress exposure. Heck, animals and people die from it if strong/persistent enough.

If a person can turn their dark hair grey overnight by exposure to massive stress, other effects can be present too.

Did you notice if they pre-registered the separate analyses of males and females? Flexibility in the analysis of data ("researcher degrees of freedom") is likely one of the major causes of the replication crisis. Gelman's paper is excellent in showing how these issues manifest without any intended malfeasance:

http://www.stat.columbia.edu/~gelman/research/unpublished/p_...

I suggest anyone doubting the importance of eliminating all differences between the experimental cohort and the control group to read that speech from Feynman: https://calteches.library.caltech.edu/51/2/CargoCult.htm or this article that highlights it's most important part: https://www.realclearscience.com/blog/2014/02/the_rat_experi... .

If I had to reduce that to some phrases I would say : the easiest way to do bad science is to lie to yourself, so to do good science you must foremost learn how not to do that. And avoiding avoidable difference in cohorts is one of the first step to stay honest.

Why, time and time again, do we see results of major studies by career scientists which don't pass a moment's scrutiny by the educated layman?

Have career scientists been held accountable for the replication crisis? I know Hanlon's razor says not to, but at this point I can't help but think medical and social science have been more or less taken over to produce propaganda. Not too dissimilar to what the CIA did during the Cold War.

It really doesn't need to be that complicated. From what I can see (in Psychology) a combination of statistical illiteracy and motivated cognition is more than enough.

Let’s just assume for the moment that the experiment is actually invalid (due to poor control group).

Experiments that have good control groups don’t show spectacular results. Only those experiments that have spectacular results get publicized by the popular media. Therefore you are seeing a kind of bias particular more to the popular media than necessarily the scientific field.

Even more troubling, they completely screwed up the analysis, making a mistake that even undergraduates know to avoid.

They measured 12 factors (as shown in table 1) across 4 treatment groups (male/female in two generations separately). That's 48 variables, so with the p<0.05 significance level they used, they would, on average, see 2.4 "statistically significant" results just by random chance.

If I'm counting correctly, they found 2 "statistically significant" results, exactly as expected by random chance.

In other words, their results are statistically indistinguishable from chance, exactly like the famous "jelly beans cause acne" thought experiment.

https://www.explainxkcd.com/wiki/index.php/882:_Significant

Edit: Who's downvoting this without a response? Is there a mistake or is it simply inconvenient for the people who badly want to see an effect?

Yes, it’s not valid to reject a null hypothesis involving multiple tests without using a multiple testing correction. The simplest conservative approach would be to use a Bonferroni correction and check whether any of the M tests has a p-value below alpha/M. Clearly, these results are not that significant. The study suggests a couple potential effects. To credibly evaluate them, the study needs to be repeated with more mice and/or fewer tests along with appropriate statistical analysis.

Sounds like someone could simply recreate that experiment if they felt those variables had any bearing on the result.

Yes and they should do so before anyone draws any conclusions from this.

> So the exposure group were in a semi-sealed chamber where other toxins might have accumulated, and were subjected to continuous fan noise. Whereas the control group were not in a chamber and were not subjected to fan noise.

It's fine to be critical of methodology, but if they are testing for lung function and tissue compliance, then I am struggling to find a reason for fan noise to be a lurking variable in this case.

Depends on the effect of general stress from the environment? Cortisol isn’t exactly a great thing for us to marinate in.

(That is way out of my field of expertise, I don’t know how to properly gauge the relative strength of effect of stress hormones vs. CO2)

Frequently, in experiments done before "double blind" and "control group" became the standard, we had results where something for which we were "struggling to find a reason" to be important, actually turned out to be important. In a properly done experiment we would not need to have a conversation in which lay people guess what else might have affected the results.

Reply on news.ycombinator.com