Comment by jknz | Hacker News Reader

Comment by jknz | Hacker News Reader

From the article, it seems that there are 2 opposite "evidence based medicine":

(a) do research, publish papers, test thoroughly new treatments in well defined trials and slowly build up a biological theory as well as official guidelines and treatments. And

(b): measure the outcomes of different hospitals, declare the best performer's methods as the state of the art and expand these methods elsewhere. If the measurement is sound, one can argue that they are both evidence based. The former is slow but may provide a deeper understanding of the inner workings of the deaease. The latter is fast but it's hard to tell exactly why it works so well.

The was a shift in machine learning research in recent years.

(a): write theoretical papers and study using math the generalization performance of algorithms.

(b) release a new challenging dataset every year (except a test set) and organize a prediction competition on this dataset. The winner algorithm is declared the state of the art, and can be applied to other datasets, event though Boone understands why it works so well.

The approach (b) was particularly fruitful and efficient in recent years. Let's hope that applying this approach to medicine Will lead to great outcomes!

You can't rightly say in either medicine or machine learning that the latter method is "better" than the former. You need the former to make progress past whatever local maximum your buckshot approach gets you. If you don't understand how it works, then eventually all the refinements lead to overfitting in machine learning and superstition in medicine.

Concerning your last paragraph, this article is from 2004, so technically we can examine the efficacy of this approach right now if it were utilized.

Intel also used (b), in "Copy Exactly!", including seemingly irrelevant details. They also tried to work out why it worked... but didn't let not knowing stop them from using it. In science, experimental results often precede theory anyway (and indeed incite it).

Reading the article, it struck me that simple geographical factors might also influence success rates: e.g. weather (humidity being a CF treatment), genetic factors in that population, socio-economic factors (esp. time and effort available for the intensive care required).

BTW I like to think that's not a typo for Noone, but a US colloquialism paralleling "God knows why" but instead referencing Daniel Boone.

The obvious difference here is that no one dies if I start training my new weird deep learning architecture on some challenging training set on an AWS gpu instance and said architecture turns out to be crap.

Meanwhile, medicine is a lot less forgiving of wild goose chases, and requiring that the taking of risks is soundly justified seems pretty reasonable, as exciting as the idea of unrelenting research sounds.

except that people are already dying. it is not the case that nobody dies now and only this newfangled method has some chance of people dying. people are dying now, and the faster we can make fewer people die, the better. even the notoriously slow and bureaucratic US drug approval process recognizes that sometimes new drugs are discovered that are so much better than the status quo that is is unethical to check if there are unknown side effects; it doesn't really matter if someone gets kidney disease ten years down the road if they have advanced cancer right now.

edit: of course I agree that the new way is not necessarily better, I'm just saying that the old way is also not necessarily better.

It isn't true that the FDA recognizes that some drugs are so beneficial it is unethical to check for side effects

The FDA approval process is slow partly because it is bureaucratic, but largely for two very important reasons: 1) because it is easier to prove effectiveness than safety (i.e. Easier to prove a positive than a negative), FDA requires rigorous standards to ensure drugs are safe (many FDA rules came into existence because people died from taking approved drugs) and 2) to prevent snake oil sales

It is difficult to imagine how pervasive and socially detrimental "snake oil salesmen" are (i.e. People who sell useless products to vulnerable sick people by taking advantage of their need for care), but without regulations this would be a huge problem

The FDA does have a fast track process, but it's more of a "we'll review your submission in two months instead of a year" kind of deal. You still have to do all the clinical trials, although in a few cases (e.g., the ebola vaccine during the recent epidemic), it is possible to overlap the trials a bit.

According to the people who actually work on these drugs, the FDA is neither excessively or insufficiently bureaucratic. It's worth remembering that only about 10% of the drugs that start the process actually finish it, and half of the submissions turn out to fail at the final, expensive step because they don't work.

Yes fast track and breakthrough destination and priority review allow you to cut review time down and get more FDA input on your dev plan, but none of them allow you to ignore getting safety data as the person I responded to said.

Accelerated approval actually does reduce the number of studies you need to do to get approval. One of the reasons cancer is such a hot space is that FDA has been lettingamy drugs get accelerated approval from phase 2 studies looking at tumor response endpoints or progression free survival, rather than overall survival, which is a more relevant endpoint but one that takes longer to measure

I work in the field and find FDA is generally ok but I've been in situations where FDA used its power to enforce its will based on internal politics rather than scientific evidence. Very discouraging but that isn't the norm

"One day when I was a junior medical student, a very important Boston surgeon visited the school and delivered a great treatise on a large number of patients who had undergone successful operations for vascular reconstruction. At the end of the lecture, a young student at the back of the room timidly asked, “Do you have any controls?” Well, the great surgeon drew himself up to his full height, hit the desk, and said, “Do you mean did I not operate on half of the patients?” The hall grew very quiet then. The voice at the back of the room very hesitantly replied, “Yes, that’s what I had in mind.” Then the visitor’s fist really came down as he thundered, “Of course not. That would have doomed half of them to their death.” God, it was quiet then, and one could scarcely hear the small voice ask, “Which half?”

—Dr. E. E. Peacock, Jr., University of Arizona College of Medicine;

I've seen this come up in EMS. There is the question for certain types of patients if it better for the EMTs to provide care on scene or just get the patient to the hospital soonest. To help determine what's better they pick days of the week where EMTs provide minimum care. It's rough to do this to patients and EMTs, but it's also important to know what truly affects patients' outcomes.

Meh I don't think your example about machine learning is particularly apropos...or even very accurate.

Brought to mind http://slatestarcodex.com/2017/08/29/my-irb-nightmare/ . Maybe our standards for a)-style evidence are too high, and it needs to be easier to do evidence-based work within clinical practice?

Reply on news.ycombinator.com