Comment by arkades | Hacker News Reader

Comment by arkades | Hacker News Reader

Parent Full thread

(1) This isn't Nature, it's SciRep. Don't get distracted by the URL.

(2) Nature is an extremely high impact factor journal. However, it's ... not really a medical journal. Nor a psych journal. Nor a developmental peds journal. It's a basic sciences journal. What I mean to say is, even if this were Nature, it'd be a super high-impact journal that isn't actually read by clinicians who would be best positioned to criticize the study.

(2a) For instance, their bowel transfer prep involved a clean-out with a two week course of vanco. The only bowel bugs it's used for are c. diff colitis and staph aureus enterocolitis. The reason it's really only used for that is because it really doesn't touch any other common bowel inhabitants. It's useful for those because it doesn't absorbed very effectively, so if you've got C Diff sitting in the bowel lumen, great, send a wash of vanco down the pipe. But, uh, for any other bugs in the gut? It's not even second line. So, how weird that it would be used to knock out pre-existing gut flora and "suppress pathogenic bacteria." C Diff and MRSA are not the leading pathogenic gut bacteria. The pre-existing flora are predominantly anaerobes.

(3) An unblinded, uncontrolled trial of a disease whose symptoms are highly subjective in measure? Excellent. I'm sure they took measures to ensure they weren't influenced in their subjective analys-

(3a) Oh. The senior author has a bunch of patents out trying to commercialize probiotics for autism.

(3b) Oh, they used the PGI3, CARS, SRS, and ABC scores. I guess if you fill out enough bubbles you can overcome the fact that those are still subjective enough that they're shit for an open-label trial. Whoops.

I don't mean to be overly dismissive. It's a fine pilot. The results are interesting enough that I'd want to see a real study tackle this. My comments are mostly aimed at the prevailing sentiment here taking this at face value.

From the article:

> Further research with a placebo-control arm is warranted

IIUC they don't have a control group. They don't have a control group!!! Reading again your comment, you wrote this in point (3) and (3b), but it needs a few more exclamation marks.

That sentence doesnt end there. It also includes, "double blinded".

Again, parent poster mentioned this but as you say, more exclamation marks are needed.

> What I mean to say is, even if this were Nature, it'd be a super high-impact journal that isn't actually read by clinicians who would be best positioned to criticize the study.

More importantly, the peer-reviewers are likely scientists, not physicians. This can make a subtle difference in how and what they review.

But you did a great job listing out some of the shortcomings!

Is SciRep the equivalent of "Op-eds"?

As for one of the senior authors filing a patent for commercialize probiotics for autism, shouldn't there be a mandatory disclosure in their papers that have a conflict of interest?!

Publishing companies tend to create many journals for different (sub-)communities and also numerous other reasons. Nature Research (the company) has next to Nature (its most well known journal) many smaller ones, like Nature Biotechnology, Nature Communications or in this case Scientific Reports. Some are well regarded (e.g. Nature Biotechnology, Nature Communications). Scientific Reports has a comparatively low impact factor, which has furthermore dropped every year in the last few years. Impact factor measures the average number of citations a published paper receives. Its impact factor of 4 is not very impressive. The main Nature journal has an IF of 41 (Nat Biotechnol has 35 and Nat Comm has 11).

In this article the patent applications are disclosed in the Competing Interests section. The readers can draw their own conclusions from those. It would be very unusual for a journal to not demand a declaration of conflicting interests.

I was going to ask, this sounds overly dismissive, but then I read your last paragraph.

Also I know SciRep is fairly new but isn't it still under the Nature umbrella? ie. There's still some peer review; the study is likely to be legitimate.

The authors are pretty explicit about the weaknesses in their study and I think that the flaws are reasonable for a pilot study. I'm surprised the authors didn't say anything about their conflict of interest though.

Showing a link between autism and the gut microbiome would be incredible. Obviously more work needs to be done before anything is concluded but this is a really interesting result.

I think he was understating. Dismissing this results is in fact the right approach for a scientist.

Agree, and furthermore, dismissing any single paper result without replication is the right approach for most scientists and (I'd argue) almost all non-scientists.

The publication incentives are extremely messed up and in favor of exciting results that aren't true.

(0) There's no control group. This isn't science.

Perhaps this paper is the 'observation of a phenomenon' which may lead (or has lead) to the formulation of a hypothesis.

Thus, this paper is saying what the scientific method should test next.

That was my impression as well. The tone seemed to be "we turned over a couple rocks and we think we noticed a couple things. We want to share that hike in the Autism woods."

The number of subjects was small. They even added: "We note that due to the open-label nature of this initial trial, all of the assessments are subject to placebo effect..."

The problem is that this kind of studies should be discussed only inside medical centers where the next more reliable studies are designed. The discussion with general public degenerates because most people don't understand the difference between a preliminary study and a good clinical study. You will see in a few days a press article in a major newspaper with the title "Fecal transplant cure definitively autism forever, scientist say" and I will cry.

This has already happened, and quack doctors are already using this to peddle fecal transplant cures on local autism groups.

Scientific method:

Step 1: Observation

Step 2: Hypothesis

Step 3: Test

Do you know/understand how this study deals with regression to the mean? I can't figure it out from reading it.

Also, what exactly is SciRep?

Regression to the mean is briefly touched on (although they don't call it that) near the end of the "Results and Discussion" section.

tl;dr: "In this other observational study, many subjects with autism didn't improve over time on a different metric, so the improvements in our metrics were probably because of the treatment and not spontaneous. Also, we asked the parents if there was any regression to the mean, and they said no."

great summary.

there are remarkable parallels to Andrew Wakefield's paper about autism, vaccination, and his hypothesis about the gut.

Perhaps this should have been published in the Lancet[1]

https://www.thelancet.com/journals/lancet/article/PIIS014067...

Reply on news.ycombinator.com