Comment by ed | Hacker News Reader

Comment by ed | Hacker News Reader

Here’s the paper https://www.aging-us.com/article/203485

> The study population comprised adults (5 males, 1 female) with significant memory decline aged 64 years and older

> The protocol consisted of 60 daily sessions at 5 sessions per week within a three-month period. Each session included breathing 100% oxygen by mask at 2ATA for 90 min with 5 min air breaks every 20 min.

> At baseline, patients attained a mean global cognitive score (102.4±7.3) similar to the average score in the general population normalized for age and education level (100), while memory scores were significantly lower (86.6 ± 9.2). Cognitive assessment following HBOT revealed a significant increase in the global cognitive score (102.4 ± 7.3 to 109.5 ± 5.8, p=0.004), where memory, attention and information processing speed domain scores were the most ameliorated (Figure 8C). Moreover, post-HBOT mean memory scores improved to the mean score (100.9 ± 7.8), normalized per age and education level (100).

I skimmed the article for the pertinent table. Three observations:

1. There is no control group. Since people get better with cognitive tests with practice, we don’t know whether these improvements were due to the treatment or just the normal practice effect. (Though if I were a betting person, my money is on the latter.)

2. Cognitive tests are not noiseless. The same person may vary by a few points between repeated tests. The modest improvements on most tests found here might not be above this noise floor.

3. The most impressive result (memory) is almost certainly regression to the mean. That is, the group did unusually poorly on this task in the first session by chance, then they performed more typically in the second, which looks falsely like a big improvement.

Basically, I think this paper provides no meaningful evidence of cognitive improvement in humans. I can’t speak about the mice stuff, that might be fine.

While a control group is essential to determine if the effect is really due to the treatment or to practice, if they did the mouse study first and expected a positive effect, then they might have felt pressure to only have a human treatment group in what looks like a pilot trial. Maybe their original proposal included a control group, but their institutional review board decided against it?

I'm a little surprised they wouldn't have done a larger secondary trial before publishing, the impact would have been much greater if the effect was still observed in a larger group, even if they still did not have a control group- or perhaps had an alternative treatment schedule (shorter exposure times- is treatment duration correlated with degree of improvement? Then you have less of a moral dilemma of denying participants of a useful treatment while still getting some data that would indicate the treatment itself actually does something).

(working on a PhD in genetics)

I am not a practitioner, so I have no idea if this is actually plausible, but it would seem absurd to me for an experimental, out-of-left-field treatment for Alzheimers, in a <10 person study, could be denied a control group on ethical grounds. There are tens if not hundreds of millions of Alzheimer's patients not receiving this potential treatment already.

This. There is basically zero chance that an ethics review would see this as a case in which a control group is unethical. I have been through plenty of IRB processes.

Resource constraints are a much more plausible explanation - running the study with a control group would cost twice as much (in time as well as money). This is either a pilot to apply for funding to run a better-controlled trial, or it's just not scientifically serious. Both are very plausible, in my experience.

Thanks for the perspective, I have only done data analysis after the data is generated, and never have actually had to deal with an IRB directly for the design of a study.

I don’t want to speculate about why they designed their study this way or that way, because who knows.

I do feel confident in saying that Scientifically, the human data in this study are largely uninformative.

Yes control group is very important. This link might interest you https://www.ncbi.nlm.nih.gov/pmc/articles/PMC300808/?fbclid=...

6!? They used an N=6 to make such a bold claim? If I had ever thought about that as a sample size for any scientific research my professor would have run me off campus.

I also find it a bit hard to believe that going from 102±7 to 109±5 has a p<0.005. If I run a simple t-test, it says p-value (two-tailed) = 0.07.

Unfortunately a lot of medical science involving expensive laboratory resources and trials currently rests on sample sizes around this range (~5-20). Then whenever there's any kind of technical innovations that makes it cheaper to collect the biosignals there is a sudden series of breakthroughs in what is really happening. All science is a work in progress.

It is a paired t-test as the test takers are the same individuals. This is confirmed in the statistical analysis section of the actual paper. Paired t-test have more statistical power because we don't correct for group variation (as they are the same group measured twice).

Edit: One way to think about this is assuming the effects are random and symmetric then the probability of going up is 1/2. If all 6 participants had increased scores, then the p-value is at least 1/2^6 and this is not taking into account the magnitude of the shift.

You're right about the paired test (although I think there's an issue with the group variance). But your final calculation is for a sign test. For a t-test where every point shifts a very little bit (e.g. 0.01 stddev), p is larger, I think.

Yes, my edit was more meant to illustrate how 6 data points can yield a significant result. Paired t-test is all about the differences. So if all test subjects shift up by 0.01 (no variance in the differences) then the statistic will actually be infinite (i.e. very large) and so p is theoretically 0 (very small). I also agree that the paired testing needs more convincing from the researchers.

With 6 subjects, it feels a bit silly to even turn on the math. 5% is about two legs of a participant. I think humans can process those low numbers instantly, intuitively; we can do it by pattern recognition, without the need for abstraction. The statistics of a dinner party. I think even some animals could do some "math" on 6 food.

Should be called "collected case studies" or something.

Here is what WP says about this journal:

https://en.wikipedia.org/wiki/Aging_(journal)

Since it's not immediately obvious what the issue is since it's in the "reception" section

> The peer review process employed by the journal has been criticized by Jeffrey Beall, a university librarian and expert on predatory open access publishing, who also included the journal and its publisher on his list of "potential, possible, or probable predatory scholarly open-access journals" in July 2015.

(102.4 ± 7.3 to 109.5 ± 5.8, p=0.004)

106 is well within a standard deviation of both of those. In fact, 109.5 is only one standard deviation away from the first, and the reverse is almost true. How do they get p = 0.004?

The baseline score was (86.6 ± 9.2). You are seeing the improved score (maybe the 95% confidence interval?).

Yeah good point, it might be the 95% confidence interval and not the standard deviation.

The popular paper completely fails to to state sample size or selection. It seems 10+10 mice also were studied. Maybe a little early to call it a breakthrough...

What do you mean? The sample size was 6, and they were selected on having the same symptoms. There is more information on the statistical analysis in the paper, but the conclusion is that it was significant.

If this is a significant treatment of Alzheimer's then that's a breakthrough to me, what bar would you set?

statistical significance is not the same as clinical significance. statistical significance in a small non-placebo controlled trial is less likely to result in a clinically meaningful result.

I mean that the article on technology.org doesn't state the sample size, but still claims this has proven the effectiveness of the treatment. You would have a hard time getting a treatment cleared for public use with those numbers. If you treat 6 people and they seem to go in some direction, then you have to put that into context by having some sort of control or else it is not possible to evaluate the degree to which the improvement was in patient selection or pure chance. I think that OPs article overstates the significance at this point and should have highlighted that further studies are necessary.

> If this is a significant treatment of Alzheimer's then that's a breakthrough to me, what bar would you set?

Wouldn't having some kind of proof against a placebo or at least a dose-dependent improvement be the least possible requirements to actually believe the paper found any kind of effect at all?

If you randomly give 6 Alzheimer's patients an apple a day for 5 weeks, they could show improvements in cognitive tests by pure accident, especially for a disease that is known to ebb and flow.

Reply on news.ycombinator.com