apathy

I heard it was "worse" way "back then" but I had no idea how much worse. Without regulations and scrutiny, competitive people will push the envelope on how bad they can abuse others and for how little money. Gotta push back.

Apparently the judge did not see that as a reason to prevent him from finishing his work and delivering evidence one way or the other (maybe he's awesome and his work will show this, maybe he sucks and his work will show this).

"Team science" is actively penalized, just so you know. You're either a rock star PI with 5 R01s and an indentured servant "research professor" to handle actual work, or you're nobody. It is, and always has been, all about the money -- those sweet, sweet indirects from NIH grants, and the endowments from gullible old rich people.

Rubin sounds like an asshole, exactly the sort of PI that is selected for at many top-tier institutions. If he's really clever he'll "help" German GTFO of his lab and then write some coded instructions in recommendation letters ("if you've any questions, please call me at 617-867-5309"). And then things can go back to normal, hopefully with Rubin being more cautious and/or less vindictive the next time around.

This is awesome. The way that (often foreign, often on J1 or H1B or student visa) students are treated in major labs is often horrible. If this destroys Rubin's lab, so much the better. Maybe he can serve as a warning to other PIs.

You are right, I shot from the hip. Sorry about that.

I also have noprocrast set in my profile so I couldn't go back and edit it (something I thought about doing). I probably would have toned it down if I hadn't requested that Hacker News kick me off after 15 minutes.

Your line of discussion is productive. It's just important that people understand the difference between degrees of belief and degrees of evidence from a specific study and never confuse the two. Trouble is, lots of folks confuse them, and lots of other folks prey on that confusion.

Anyways, sorry for being a jerk.

"Wrong" isn't the word we're looking for here, I don't think. But your above example is bullshit -- nobody puts 1000 patients at risk in a phase I (safety) trial, and if the dose isn't reasonably well calibrated by the phase III study you're describing above, someone's going to jail. In Phase II we will often have stopping rules for exactly this reason, just in case the sampling was biased in the small Phase I sample.

Above there are a number of things to notice:

1) The phasing approximates Thompson sampling to a degree, in that large late-phase trials MUST follow smaller early phase trials. Nobody is going to waste patients on SuperMab (look it up).

2) The endpoints are hard, fast, and pre-specified:

IFF we have N adverse events in M patients, we shut down the trial for toxicity.

IFF we have X or more complete responses in Y patients, we shut down the trial because it would be unethical to deprive the control arm.

IFF we have Z or fewer responses in the treatment arm, given our ultimate accrual goal (total sample size), it will be impossible to conclude (using the test we have selected and preregistered) that the new drug isn't WORSE than the standard, so we'll shut it down for futility. Those patients will be better served by another trial.

You are massively oversimplifying a well-understood problem. Decision theory is a thing, and it's been a thing for 100 years. Instead of lighting your strawman on fire, how about reframing it?

Stopping isn't "always" wrong, but stopping because you've managed to hit some extremal value is pretty much always biased. The "Winner's curse", regression to the mean, all of these things happen because people forget about sampling variability. It's also why point estimates (even test statistics) rather than posterior distributions are misleading. If you're going to stop at an uncertain time or for unspecified reasons, you need to include the "slop" in your estimates.

"We estimate that the new page is 2x (95% CI, 1.0001x-10x) more likely to result in a conversion"... hey, you stopped early and at least you're being honest about it... but if we leave out the uncertainty then it's just misleading.

All of the above is taken into account when designing trials because not only do we not like killing people, we don't like going to jail for stupid avoidable mistakes.

It is a problem. You are "spending" some of your error "budget" whenever you peek prior to an endpoint. This is why if you're going to do interim analyses in clinical trials, you have to include it in the design.

People got tired of trialists playing games with statistics and patients dying or ineffective drugs making it to market. Now if you want to run a trial as evidence for approval, you need to specify an endpoint, how it will be tested, when, with what alpha (false positive) threshold, and what's the minimum effect size required for this.

If you are doing interim analysis, futility, or non-inferiority, you have to write that into the design, too.

People can jerk around with subgroup analyses in publications but the FDA won't accept that sort of horse shit for actual approval. And thank heavens for that.

Good lord, even after all this time, ggvis' demo is still rather a disaster on mobile. Oh well, it simply isn't possible for everything to be top priority. Rstudio has done some amazing work, I'm afraid ggvis is not yet up to the standards of their other projects.

This was such a beautiful comment. Elsevirus is a step down from Weekly World News. The graphical abstracts would make Jack Chick proud.

Nice theory but there are a lot of people in Africa without a lot of money who are looking for things to eat. No safaris etc. means no justification for some of these pristine parks, least of all for enforcement.

Witness the scores of elephant poisonings in Zimbabwe for poaching purposes. If you view people as contradictory to Nature, your efforts on the largest of continents will come to naught.

It's so easy to argue for ideological purity when it doesn't directly affect you or your family (abstract you, for all I know you may live down the road from Kruger). Ultimately though there has to be an incentive for the people affected to part with the resources and opportunity offered by the big animals. Safaris are one (not terribly effective) compromise that at least justifies economically why big animals need to be preserved (no animals means no safaris means a lot less rich visitors).

If you read Nature and science papers, plan on reading the supplementary materials (typically 20-200 pages) where we are forced to put all the real science. And the online methods (a half measure spurred by outrage at the preceding).

Wouldn't want to waste words on silly shit like methods, references, discussions, etc in a CNS paper, would we?

LIGO doesn't even bother sending papers to Nature or science any more. The editors at both are unbearably pompous.

This is punishingly stupid. If you are working on a large enough project (many libraries, many developers, many organizations, daily or nightly builds because it takes 4 hours to run all the unit tests even in parallel due to dependencies) you use semantic versioning.

If you're some pissant shop writing CRUD apps then yeah, don't bother. As soon as you outgrow that stage, there won't be any more questions.

Monolithic standalone apps or tiny projects don't need semver. Practically all collaborative software in between benefits massively from it.

Sounds like an excellent argument for not using node

Looks like they did OK...

That's the one I was thinking of. Thank gawd.

shit, that was true as recently as 15 years ago, from personal experience. You can't just drive down 101 from SF and buy shit at Greybar anymore?

Possibly. I was thinking along the lines of "look at idea, see if easily, reliably, and cheaply implemented, assess market & competition, decide accordingly".

Hell, even that could probably be automated. Hmm. ;-)

I don't doubt it, but the same could be said of ML in general. Being in the right place at the right time counts for a lot.

Re: 1: no shit. The turnaround is for rebuttals. I try to do thorough reviews (I have published plenty as well) but realistically I recognize that no one has the sort of free cycles required to catch everything in advance. What's worse is that an awful lot of reviews are pure political bullshit.

In the end it's the people who may or may not try to build on your work (or mine!) that are the best judges of whether it's BS.

Re: 2: probably, but diffusion of ideas benefits massively from readily accessible documentation. One of the things that will cause me to immediately reject a paper is if its method implementation does not work as described. I will always reject in such cases. They are alarmingly common and this is one of the few easy cases in reviewing. Lack of an implementation is also cause to reject.

Re: 3: it has everything to do with it. Unless you're running and publishing trials please don't lecture me on this. See opentrials for a particularly frightening take on why published trials so poorly represent those submitted to the FDA or registered with CTEP. Meanwhile actual RCTs get buried by shitbags demanding more subjects than there are recorded cases in the past 40 years (no joke).

If you're not doing trials, you may have to take my word for it (or get involved in RCTs and find out for yourself). A lot of the enterprise is actively harmful to science by omission.

I would literally pay to subscribe to a comments feed on posts like this. It's so obvious and so powerful, I'm sitting here kicking myself for not thinking of it first.

Nearly every time I've put in a successful patent, that last bit ("god damn it! Why didn't I think of that?") comes up. The other times are derived from clinical trials in rare populations (whereupon other trialists say "god damn it! Why didn't I think of measuring that?").

The idea being, look for simple ideas like this, jump into the market second, and split the market for a while. Then when it gets too crowded, jump to the next-best-idea in the queue. Could be very profitable in this big ML transition.

re: your last line:

That is the classic industrial ML advice, isn't it? Read a bunch of papers, look at the stupid obsolete method they all compare against, implement the latter.

1) Iowa Farm Boys can, you know, farm things

2) We're going to have to figure this out as a society unless we like the idea of Mad Max: Interstate 80. The incoming administration may or may not be ideal for the moment in time which we find ourselves :-/ but eventually this comes to a breaking point.

Doctors and lawyers listen to things for money, too...

CNS (Cell, Nature, Science) == Buzzfeed science

Although that's not giving BuzzFeed enough credit, they tend to check their sources & verify results/reports. Also if a piece of fake news is simply resubmitted, they don't pretend like they never checked on it before.

There are plenty of checks (mostly from NIH, mostly for incremental garbage work on dead models like cell lines) and very few balances against the tyranny of CNS...

The majority of content in most journals is asinine or incorrect. About the only thing that can fix this is quicker turnaround (cf STAP, NgAgo, the godawful GWAS paper in Neuron recently, etc).

It's not at all clear to me that the checks or balances were helping beforehand. Hiring a shit ton of MBA types to administer research probably didn't help much either.

Also, out of curiosity, is this open access / fast review problem what led to machine learning falling into irrelevance?

Because we wouldn't want that to happen to other fields. Can you imagine if raw clinical trial data was out there, for example? (The Horror! Best keep these appallingly expensive results locked up!)

Publishing sometimes does 1), and rarely does 2) (as a statistician you surely know that the relationship between impact factor and retraction is nonlinear and rises in strength as you get into CNS, NEJM, and the like).

I review for others because others have done 1) for me. But I'll never review for Elsevier, and lately I've had the luxury of reviewing for the most cited of open journals (by operating bioRxiv, and accepting direct submissions from it, I claim that Genome Research is "close enough").

It makes me very happy that this is possible (my CV has not suffered for only publishing as first author, and whenever possible as senior or co-senior, in fully open journals). I'm pretty sure this wasn't possible for most people a few short years ago. That engenders optimism about the future of scholarship, for me at least.

Hopefully you as well.

When I worked at the Cornell Theory Center in the late 90s, I didn't know about the arXiv, but I sure as hell knew about screen readers. One of the best FORTRAN programmers (numerical analysis/applied math research faculty I think) was blind as a bat.

I learned a lot about not underestimating people there.

It's a strange university but it always makes me happy to think that it (specifically Ginsparg, and to some degree strogatz and the law scholars) pushed forward what the Web was supposed to be.

Not a place to buy stuff, but a place to learn stuff, without waiting for it to make its way through a hyper politicized review process so that it could be printed in a 17th century fashion and mailed to some corners of the world, eventually perhaps reaching a fraction of the people who could use it. Rather, everyone everywhere with a connection.

Anyone who doesn't deposit preprints (arXiv, biorxiv, or wherever) or who doesn't agitate for their coauthors to do so is not really in it for the science. It's fine to be competitive -- deposit yours first. Make the fucking discovery instead of parasitically piggybacking on those who do the work.

But that last part, that's hard. Very hard. As long as there is enough money left in academia to encourage lazy shits, those of us who care about scholarship will have to push, hard, to remove the last refuge of these scoundrels.

You're either on the side of justice -- open data, open formats, open scholarship -- or you are tacitly endorsing Elsevier & Springer, who haven't the slightest problem using crap like incremental JavaScript & mangled PDFs to deny access to scholarship even to those who have paid.

David (blocking on his last name) at CTC made that choice for me. He set an example that forced me to admit what was right. I hope others will do the same. It's the right thing to do.

This is not a short or simple read. However, if you believe that the American experiment was not a mistake, and the sclerotic dynasties of Europe are a Bad Thing, it is worth the time to read through to the end.

With the incoming administration, I have a very bad feeling that the events in the story -- militant common people fighting for their own demise -- will be replayed on a much larger scale. The more people willing to die for a cause realize that it is a lie, the more warm bodies there will be left to defend the principles that at least some of us still believe in.

Post your galleys to http://biorxiv.org/submit-a-manuscript when you send the MS out for review. Simple as that. Oh, and respond to RFI/RFCs from NIH asking whether interim research products should count in grant reviews (yes).

You will note that there are now clinical trials on biorXiv, and I've been pushing (as a coauthor) for more of them (NEJM & Blood are the only journals worth noting that won't accept submissions that are archived as preprints).

Elsevier is at least as bad as Blood or NEJM, but seemingly lacks the editorial standards of either. They can suck their 35% profit margins out of someone else, TYVM.

More than those, but genetic and environmental variation in their "cycles" (if you will) as well as the stochastic events involved in turning them over seems to produce extremely wide variation in their eventual readings.

I work on senescence in adult stem cells, for whatever that's worth. We have found 72-year-olds whose viable fractions were greater than those of 26-year-olds, but as far as we can tell (with sample sizes in the dozens instead of dozens of thousands, this isn't going to be much), there aren't a ton of good predictors save chemo & radiation.

YMMV

This is fantastic. My only concern is that, after the last election's results, it's not clear to me whether enough members of the electorate are willing to vote in their own best interests (rather than "cheering" for their "team", which in similar fashion to football or baseball teams could not care much less about the "fans").

Without something to motivate that, it'll be a tough slog.

Interested to hear how this is "precision oncology". Do the 5hmC patterns predict treatment response? (Spoiler: no, because this was retrospective; need a trial to validate) Do they provide information above and beyond IDH or co-1/19 status? (Spoiler: also no, partly because N=30 instead of over 1,000 in the TCGA LGG/GBM cohort)

I would expect this from the PR organs of a major university or other donation-pandering body. I am saddened to see it from Collins, who knows better. The treatments that seem to work in glioblastoma center on dendritic cell vaccines or T-cell priming, at least at this point in time. Radiation is a temporary respite and only succeeds for focal (rather than diffuse) lesions in most cases; furthermore, "success" is relative (very few GBM cases are cured, even in younger patients).

So while trials are afoot doing the brutally difficult work of evaluating actual candidates for cures, we're directed towards studies of marks that are almost certainly a proxy for metabolic activity. In bulk cells, natch. Way to go.

:-(