Hacker News Reader

Comment by IfOnlyYouKnew | Hacker News Reader

It's only a "great way to create statistical fraud" if you assume that the statisticians, and the FDA, are either stupid or corrupt. And contrary to popular belief, they tend to be competent enough to at least cope with the sort of hair-brained schemes HN believes would fool them.[0]

Stopping a trial early and it's statistical implications should, by the way, be somewhat familiar with web developers: it's commonly done with A/B tests, and the related problem is the "One-armed bandit" (https://en.wikipedia.org/wiki/Multi-armed_bandit#Empirical_m...)

[0]: Legend says they are even familiar with correlation != causation, and some have mastered the advanced skill of knowing that trial size matters, which is why they started, a few years back, to test drugs on more than one person.

View on HN

> It's only a "great way to create statistical fraud" if you assume that the statisticians, and the FDA, are either stupid or corrupt.

This is a legitimate issue in the scientific community. The FDA made it a requirement to publish drug trial criteria beforehand to prevent trials from stopping prematurely or adding more participants. This resulted in a massive drop in drugs with positive results: "...before the registry was created, more than half of the published studies of heart disease showed positive results. After the registry was created, only 8 percent had positive results". [0]

The FDA and scientific community isn't as "pure" as your flippant dismissal would suggest. The FDA can only do so much, and trial runners have misaligned incentives.

[0] https://www.npr.org/templates/transcript/transcript.php?stor...

> It's only a "great way to create statistical fraud" if you assume that the statisticians, and the FDA, are either stupid or corrupt.

Considering how much money is in pharmaceuticals, I'd definitely go with corrupt, especially considering this explains how a majority of the world works now and also since there's currently an outbreak and likely want to capitalize on it.

I'm not at all saying this is the case, but that if something like that did come out then I really wouldn't br surprised considering it seems to be a weekly occurrence that you hear about corrupt behavior now.

Are you familiar with what PJ O'Rouke says about politicians?

Have a listen to this https://www.youtube.com/watch?v=JbIqKqojOZU

He is the author of Parliament of Whores

Parliament of Whores is an international best-selling political humor book by P. J. O'Rourke published by Atlantic Monthly Press in 1991. Subtitled "A Lone Humorist Attempts to Explain the Entire US Government"

I feel like Hanlon's razor is applicable here.

The issue with cynicism and corruption conspiracy theories is that they become a self-fulfilling prophecy.

If everyone is corrupt, then it’s OK if you are, etc.

There is no fair application of the rule of law, there’s no reason for rule of law, only the rule of men. Better hope they’re on your side!

My point is that clear corruption as we think of it (fraud, bibery, self dealing) in most Western countries is NOT considered normal. Ie. It happens but we’ve built institutions to counteract it. The FDA is notoriously bureaucratic in order to detect and prevent corruption, and for the few cases to slip through, to correct it.

When you hear about corruption stories, that’s about the system WORKING against those cynics that are testing it for their own benefit.

Cynicism is self destructive. If our institutions are broken, we must work to repair them.

> Stopping a trial early and it's statistical implications should, by the way, be somewhat familiar with web developers: it's commonly done with A/B tests

AFAIK stopping an A/B test early is the way to go if you want to convince yourself (or your customer) that something has an effect, even though it hasn't.

Yes, if you "peek" at the results of an A/B test before it's done in order to decide whether to stop early, the numbers you "peeked" at have much lower statistical power than if you foreswore stopping early. Obviously, failing to take that into account when drawing conclusions about the effectiveness of the treatment is a colossal mistake.

However, the decrease in statistical power is still quantifiable, and with the right math you can still calculate an accurate 95% confidence interval on the effect size (which will be much wider than the wrong math where you naively don't account for the "peeking"). And of course, it's totally possible that the treatment is so effective that even the accurate calculation shows that the lower bound on the effect size is so much higher than the control group that the responsible thing to do is to stop the trial early.

Here's some the math on how much lower the statistical power is if you "peek": https://www.evanmiller.org/bayesian-ab-testing.html

The word is harebrained unless you think hair has a brain.

Thanks - I didn't know this. http://www.word-detective.com/2012/08/harebrained/

I was a contractor working on an IS at the FDA in the early 2000's. Shortly before the VIOXX scandal (https://www.drugwatch.com/vioxx/). The number of cheats they were specifying to go around safeguards was scary. Fortunately, they fired the company I worked for and awarded the contract to the company employing the husband of one of the FDA managers. ISYN. A relief to leave.

rule #1 of hn: anyone on hn is smarter than everyone not on hn no matter the domain. we could solve all the world's problems by just getting everyone into hn!

rule #2 of hn: High signal to noise is preferred over low signal to noise. HN > twitter.

To be fair, and when it comes to health and biology, Twitter >> HN. The comments on most bio-related posts here make me cringe.

This.

I read HN regularly, but for health and biology related information, I'd much rather turn to the various professional communities on Twitter. HN is at best "Lies my Bio 101 Professor Told Me" most of the time, and at worst a worked example of physicist/engineer's syndrome.

We have the best brains!

>>the sort of hair-brained schemes HN believes would fool them

Is there much value in an ad-hominem reply?

Or in generalizing it to an entire community? The irony seems hard to avoid when such a generalization is made without mentioning any statistical foundation, while pontificating about statistics.

> Is there much value in an ad-hominem reply?

> Or in generalizing it to an entire community?

I don't like this style either but sometimes I think we deserve it for being a smug echo chamber.

Basically, much the same thing could have been said in a different way and I would have upvoted it.

> we deserve it for being a smug echo chamber.

This and more. Almost every comment here is made with absolute confidence and authority, much more than is warranted. The tone of the person who said that "it's a great way to commit statistical fraud" without a caveat sounded like a know-it-all who knows better than the FDA and the scientists who made this cure in the first place or that they're outright unethical.

Most times such comments get upvoted. Sometimes they get downvoted when they're called out on their BS. So there's some balance.

Have you read any thread that involves statistics? I havent really detected that low an opinion for the FDA in particular but I recognised the broad outline painted.

I could add quite a few more generalisations myself while I'm at it, HN isn't solely populated by hyper intelligent, well reasoned, logical doctors, who form their ideas completely independently of everyone else. There's going to be group think, there's going to be like minded individuals attracted to one another, that's human nature, deal with it.

I don't even think it's an ad-hominem. HNers are generally argumentative gits (generalisation), who like finding an exception/hack/new way of looking at something (generalisation). I have no problem with some of those schemes being described as 'hairbrained' (sic although actually... [1]), but that's why I come here, because if people didn't think that way they wouldn't be Hackers.

[1] https://www.thefreedictionary.com/harebrained

> sic although actually...

Not sure if that was for comedic effect but I laughed out loud! A brilliant demonstration of the behaviour you describe.

It's a fair question, this shouldn't be downvoted even though I also think the original generalization was fair.

Reply on news.ycombinator.com