Why Self-Track? The Possibility of Hard-to-Explain Change

(2) "NO published result related to nutrition" is overbroad. Obesity for example is clearly related to nutrition and there are many published results where many variables which could affect obesity are quite well-isolated -- twin studies to isolate genetics, studies of how obesity rates vary based on physical location in various cities, and so forth.

(3) I aim to "discard the results" only insofar as they claim that they have measured an agentic relationship, which these results have indeed claimed to measure. Seth's official conclusion is that this was "data that suggested butter improved my mental function." If the variable is not isolated, then attributing the agency to butter is worthless. It might have been an otherwise random variation at around the same time as he switched to butter -- the data he's describing has a 40ms variance as well as certain long-term patterns and the effect that he's describing is a 30ms improvement, so it's quite possible that in an autoregressive model you would just see one substantial "step down" of 80ms which doesn't get compensated due to hysteresis. Or perhaps the correlation is indeed correct but wrongly attributed -- perhaps Seth has not noticed but he tends to use butter to fry peppers and pork fat to fry meats, and now that he is using butter his diet contains 2% more vegetables. Perhaps now that he uses more butter, he eats more toast and gets 2% more fiber in his diet, without having written that down in his journal.

The problem is precisely the word "makes" in your sentence "butter makes Seth faster at arithmetic." You have absolutely no evidence that it's the butter which is making Seth faster at it. And that is why the conclusion "butter makes Seth faster at arithmetic" is not a scientific conclusion.

There's theory, and then there's practice. In 3 clinical experiments that I've followed closely, it was officially "double blinded" (and at least one was published as such), but because the placebo was sugar, and the substance or treatment under test was not inert (one caused dry mouth, one caused flushing, and in one laser treatment, people found the real thing painful, unlike the placebo light). So the patients knew about it for sure, and the experimenters had to be idiots to not know.

Do you think the published results eventually reported that? The answer is no. That would have killed FDA approval, and potentially cost hundreds of millions of dollars. Is that scientific fraud? I would say yes, but you'd be surprised at the answers you'd get if you ask the people involved (rather than the theoretical question).

Do you think this situation is uncommon? I don't. I might be biased here by my own experience, but I would assume that when hundreds of millions of $ are on the line, that science takes a back seat. FDA slap-on-wrist fines seem to agree with that.

> Are you saying there is no evidence that a sample size larger than one leads to better results?

No, I was not saying that. I was referring to the "hundreds of years, hundreds of smart people" statement which was vague and did not point to anything in particular.

> But you seem to be suggesting that the solution to this is good intentions and a scrappy can-do attitude. If the well funded experiments with large sample sizes, double blinding and excellent controls still screw it up most of the time because of some bias or error in procedure then why would I expect removing all controls would result in something even worth looking at?

Read what you just wrote. Science's "golden standard" is essentially unattainable: 88% of the most (assumed) useful results about cancer in the last 10 years cannot be replicated (therefore, scientifically wrong , or in your own words, shitty results accepted as valid). Read that again. And again, and let that sink. That's not an accident and that's not just because science is hard. That's a direct result of the way science is practiced these days (The xkcd is the super-short summary of one of the reasons, the Ioanaddis piece is the "scientific" version with many identified reasons).

Guess what - if you do not change how science is practiced, you're guaranteed to get more shitty results. In fact, you're guaranteed to get a higher percentage of them with time (I don't remember if this piece by Ioanaddis touches that, but he did in other places). People in science I've talked to expected a 50% "wrong result" ratio from the golden standard that's supposed to give you good faith. No one expected the 88% failure result, but no one thinks it is exaggerated.

Again, these conclusions (results done the same double blind, N>>1) are going to get shittier and shittier, unless you change something about how you do stuff.

> The way you pointed out that unreplicated papers are probably wrong seemed like you thought you had some kind of gotcha.

Question, if Seth did acceptable double-blind N=30 (assume sufficient) experiments, would have pointed out that "well, that's not scientific because it wasn't replicated?", if so - kudos. But you'd be hard pressed to accept any nutrition or medical result published in the last 30 years, because the vast majority does not comply with that standard. Do you have that response to any scientific news you hear?

If indeed you hadn't replied like that, then you are moving the goalposts. But I'll give you the extremely unlikely benefit that you would have.

> Well the "golden rules of science" are pretty clear on the value to be placed on unreplicated papers in a field overrun by media attention.

That is wishful thinking. For a few years, I read the medical literature and followed how things became standard practice (my SO at the time was a doctor, I'm an engineer). Results such as described by Ioannadis and the Amgen guy become accepted in medicine and nutrition without any attempt at replication. And then pulled back 10-20 years later when people are dying of e.g. Vioxx, or statins (the sh*t has to hit the fan on this one; wait another 10 years), or realize that most of the Prozac data was cooked to look much more significant than it was.

> Which side are you arguing?

I'm arguing medical/nutritional science as practiced today (especially its relationship with "publish or perish" academia and "a bad result will cost us upward of $400M" pharma) has painted itself into a corner. The incentive structure essentially guarantees that the percentage of wrong answers is going up. Replication is part of the "gold standard", but is rarely ever practiced. (The Amgen paper is the only modern systematic attempt at replication that I'm aware of; and of the few casual replication attempts that fail, most don't get published).

It's converging into an insane "what must do this, even though we don't have the facilities, but anything else is rotten, so we'll pretend it's ok and not bother replicate unless someone points out we're wrong".

What I'm arguing is not "science is hard" (it is!) or "science is easy" (it isn't). It is "we need to find a new way to do science". The golden "double blinded, n->inf, multiple replication" is the easiest to be convinced of, if done right, but definitely not the only way to do science, and definitely not often done right (and with the existing incentive structure, will be done less and less so).

I am arguing that we should find other ways to make progress, and I think Roberts is on the right track. Following him for the last 7 years, I know that he produced more useful results (safe, easily testable, showing improvement for a large percentage of people who attempt to follow) than most hundreds-of-million-dollars scientifically-golden experiments. And he did it in his free time, and with no funding.

The way science is done needs a disruption. Double blind n->inf multiple independent confirmation is indeed the easiest way to be convinced of a result, but it is NOT the only way, and it is becoming increasingly rare/impossible to carry out.

What I'm arguing is that the gold standard is actually not achieved; You don't disagree, I think. What we disagree about is how meaningful the results that are coming out are compared to the effort/cost. I believe that the existing tradeoff is not acceptable; you seem to believe there is no tradeoff as there is only one way to do science (it's just that the ideal is not achieved).

What I'm arguing is that Roberts is giving an example about how the way science is done could be disrupted. It's immature, it's not easy to replicate (I'm not as placebo-resistant as Seth is, for example). But it has applicable results, and is cheap and effective.