Hacker News Reader

Comment by austinjp | Hacker News Reader

> My wife ... is an extremely specialized surgeon and she is horrified by the academic publishing in her area of expertise. It is often utterly wrong and conflicts directly with her experience.

Genuine question: how does someone in her position know they're right and the research is wrong, and not the reverse?

I'm certainly not denying that she may be correct, and I'm not unquestioningly defending research evidence either.

We've come a long way from "eminence-based medicine", where the senior clinician was impossible to challenge. Research has helped us move towards a more nuanced approach to care (I would say "holistic" but that word has become tarnished).

EBM "phase one" alienated clinicians despite explicitly acknowledging clinician expertise, by overemphasizing evidence and undervaluing experience. EBM "phase two" aims to redress the balance, but perhaps the mistrust is too significant now. It would be a shame if so.

Some reading:

https://www.bmj.com/content/348/bmj.g3725

https://www.cebm.ox.ac.uk/news/views/it-is-time-for-evidence...

View on HN

> Genuine question: how does someone in her position know they're right and the research is wrong, and not the reverse?

She doesn't know, and that's the problem with medical research: being a specialized surgeon is like a carpenter who's made hundreds of a particular kind of furniture. Then a scientist comes along, saying we should change how we do it. As an experienced carpenter, you know that most of those scientists actually have very little practical experience, because they used that time to publish and are actually very bad, practically speaking. And therefore oftentimes reach nonsensical conclusions. So, what to do? You go ask the most practically experienced guy you can find, hoping he'll be able to guide you. Repeat 100x/year.

I'm not exaggerating: some of our academic practitioners who've published a lot are really among the worst practitioners I know, and do kill people who should have lived on a regular basis! Ironically, some of them are respected professors and patients often seek their care thinking it'll be better. One of them is a surgeon and has received an award for outstanding teaching. He's the absolute worst surgeon I know.

So, no easy conclusion here...

Not the OP, but one of the organisations I work for publishes best practice guidelines for diagnostic tests for various rare genetic disorders in an attempt to avoid this issue. However the process takes between 1 and 2 years and starts from identifying a group of leading clinicians and academics who specialise in that condition who are willing to write the guidelines (on their own time, for free). They then do a literature search to gather any relevant material and at the same time put out a call for submissions to clinicians around the world asking them to contribute. A (usually virtual) conference is then held and the writing begins. Over the next few months a set of draft guidelines are produced and then put out for comment. After a period for feedback a second draft is produced and sent out again. A consensus building exercise is then conducted (aka another mini conference involving the guideline writers and others) and the guidelines finalised according to the consensus. They're then published in a peer reviewed journal.

As you can imagine this takes time and money and all of it is sponsored by a not for profit EQA organisation, with no pharma companies involved (since they're who provide the tests and that's a massive conflict of interest). As a result the org only manages to produce two or three guidelines a year, when really they'd like to be able to produce a dozen or more. And they also need to be reviewed at regular intervals which while slightly less long winded is still long winded. Really this is the sort of thing that is a prime candidate for being sponsored by government, and sometimes it is, but since these are rare conditions the cost / benefit analysis is not always favourable.

I can totally believe that. What you’re describing has an uncanny resemblance to academic software engineering research: many of the published insights scale up all the way to one semester student projects — in other words, are completely useless in the real world.

This reminds me of agriculture under the Soviet Union. Stalin wanted to build a technocracy with the brightest in charge of there specialized fields, so he brought in the most famous Russian agronomist, Lysenko, and gave him complete control over the Soviet agricultural system. Lysenko took control away from those ignorant and backward Russian farmers and revolutionized Soviet farming with all the latest scientific breakthroughs.... It wasn't long before millions of people were starving to death.

Except that this is not the only (or even the main reason) the big starvation happened. It was because Stalin had to pay the US for their technological knowledge and deliveries of factories and as the Ruble wasn't worth much the US demanded to be payed in natural resources, mainly agricultural products. As Stalin saw the writing on the wall in regards to the danger of a middle European nation trying to overtake Russia he made that deal and ferried most of the Russian produce towards the US.

So to be able to (at first) produce tractors and later switch that to tanks he sold his people to the US that knowingly and gladly accepted the cheap food.

So both nation's political and industrial leaders are responsible for the mass starvations. One can't just blame it on Stalin or on Lysenko.

But Lysenko might have made the problem even worse. That much we can agree on.

Source for further reading: https://twitter.com/kamilkazani/status/1505247886908424195

So if a parent spends their whole paycheck on a new shotgun and their child goes hungry, it is equally Remington's fault. Got it

Remington’s raison d’être is producing shotguns.

The US’ raison d’être is (hopefully) not knowingly starving millions of people to death. When you’re making a deal of that size (between countries, not a company and consumer) no deal is purely transactional, there are ramifications.

The point is that the US (government) knew at the time of these ramifications. So in my book they were complicit.

Not that any other country would have acted differently. I am not here to metaphorically shit in the yard of the US. It is just business as usual when it comes to politics. Sadly. I just don't expect people to act with integrity when in positions of power. I still like to be pleasantly surprised if it happens - I just don't expect it.

Edit: Typo

It took me a while to understand in which direction you are arguing.

Of course the US is complicit (just not solely complicit as you imply I claimed - nice rhetorical figure by the way just a bit too obviously applied). They knowingly let a dictator pay them in natural resources while these resources were lacking for the people and this lead to famine and mass starvation.

So to stay in your image: When Remington sells me a shotgun knowing I will go and use it to kill my family (because it was clearly visible) they are complicit.

If the US would not have know the consequences of their deal with Stalin for the ordinary people we would be talking about a different situation here. But the US knew. And they still took the grain and other resources. And that makes them complicit.

In my world, that makes the U.S. very clearly an accomplice to a dictator who is starving his people. And also profiteers from such a system.

But hey as if this were something new or shocking. I don't know about US educational system, but i thought it was public knowledge, that the US acted that way as their standard modus operandi for many decades now.

No matter if with South American dictators, with African dictators or even with dictatorial regimes in the Middle East, as long as they do not become too rebellious and do not challenge the supremacy of the USA the US are willing to trade and not look too closely towards the crimes against humanity these dictators are performing.

Take Saddam Hussein as an example.

Wasn't it the U.S. that continued to financially and militarily support Saddam Hussein in the Iraq-Iran war even after he openly used chemical weapons? Without the U.S., Saddam would never have become head of state in the late 1970s. And the US gladly bought his oil - ignoring what he did.

This is the same as we Germans are currently also responsible (at least in part) for the prolongation of the war in Ukraine, because we still do not want to give up the gas supply from Russia. We are prolonging the war because our money is helping to finance Putin's military advance.

Our convenience and the intransigence of our government makes us complicit in the death of many civilians.

> So to stay in your image: When Remington sells me a shotgun knowing I will go and use it to kill my family (because it was clearly visible) they are complicit.

No, that's a very different metaphor.

The item being bought causes no harm. The problem is reckless spending. That puts the bar of responsibility in a very different place.

If the parent needs a shotgun because of Nazis on their front door and has no real choice, then yeah, pretty much. Not "equally" but not zero.

False, Russia was attempting to rapidly industrialize and needed to purchase foreign technology and needed foreign currency to do so.

So he sold the grain that would feed his people in exchange for USD, Deutschmarks, Pound Sterling that he could use to buy industrial equipment.

I mean, it's not exactly out of place for Stalin to sacrifice his people's lives to realize his goals for Mother Russia. Plus the added benefit that the people dying were troublemakers anyways (Ukrainian nationalists).

Do note that when the famine did strike, those same foreign countries offered famine relief that Stalin promptly turned down.

That thread establishes that Stalin exported agricultural products, it doesn't say that the US businesses were trading directly for grain.

The US was probably a net grain exporter at the time, so I think you are not presenting the situation very well. There is certainly complicity in working with an oppressive government that is starving its people, but saying it was done for "cheap food" when USSR was selling the food elsewhere and paying in dollars isn't the right way to present it.

It's a great way to present it if you were a present-day dictator and wanted to justify invading a neighbouring grain-producing country and blame it on historic wrongs perpetrated by the evil USA.

> ...had to pay the US...

had to or chose to?

It's an interesting thread, thanks for the link, but "knowingly and gladly" is not supported there, I don't see how you can lay any responsibility for Holodomor at Henry Ford's feet.

This is happening in Sri Lanka on a smaller scale right now under the guise of organic farming.

Afrer getting degree and working in the software industry im kinda worried when going to the doc, LOL.

Not to trash the profession, but in reality it's no different than a computer engineer, plumber, accountant or anything else.

I used to think "well med school is really hard, so the bare minimum for a doctor is pretty good". Then I started working in healthcare and realize, no, that's not true at all.

The best example I can think of is when I worked on a project where we wanted to get ready for a much better competitor in a disease area. Not "slightly better", but "mind blowing better and not using it would be malpractice". Most doctors had heard of it, but we want to remind them not to purchase the company's older drug because they'll just return it a year later.

Turns out there were multiple doctors, after the new drug was approved and available, that continued to use the old drug. We literally had to go to them and say "hey stop buying the company's drug please". And I realized, damn, these doctors' patients are getting really shitty treatment.

This is why pharmaceutical sales representatives are so important. Most doctors are far too busy treating patients day to day to stay up on every new drug that comes out. CME can help cover some of this, especially with new procedures or treatments, but nothing can keep up with the 100 new drugs for diabetes. what really helps doctors is having people that can come talk to them, tell them that drug X is better than Y, give them the literature if they want it, tell them what insurance has a chance of covering it. Some doctors will always do what they know, but the majority just need to know there is a better option.

Or just follow the clinical guidelines and associated updates. They're there for this exact reason, to make sure every doctor uses up-to-date methods. No need for sales reps in this case.

How quickly do you think clinical guidelines are updated.

If a new cancer drug was launched that could help would you prefer a doctor that said “sorry, the next guidelines update is in 9 months, so I’m not using that drug”?

How can they trust the literature? The article was just talking about how the evidence can't be trusted.

Doctors aren’t stupid. They can judge literature themselves.

My 60 years old father was at the hospital for a heart attack and the doctor wrote in his form he was a pregnant woman. That day I lost hope for any kind of baseline.

> scientist comes along, saying we should change how we do it

That doesn’t sound like science. An example or pointer would be helpful.

Remember the start of the pandemic? Some prof published that we should withold steroids and intubate ASAP on the basis of preliminary evidence, throwing basically everything we knew about ARDS out of the window. Most reasonable ICU guys were skeptical on the basis of their own practical experience, but had no choice because who would dare contradict science, right?? We very likely killed many people that way.

What do we do now? The exact opposite, and much closer to usual ARDS practices. You can argue that wasn't a mistake because it followed the process of scientific discovery, but it's nevertheless a tragic example of experience trumping the advice of the scientific community.

I feel like we should be critiquing society’s response to science rather than implying that their paper should never have been published.

> implying that their paper should never have been published.

That's really not what I meant. I have no problem with the publication. But put yourself in the shoes of someone such as myself, who's intubated tens of cases on admission and probably killed many doing so. Wouldn't you be questioning the structure of our research system then? Not that I have any better solution at hand, mind you...

> but had no choice because who would dare contradict science, right??

This is what I don't get. Science is contradicted and questioned all the time. It's the main mechanism by which science evolves.

Also a crucial feature of science is that it's literally always wrong. Science is the iterative process of building models of reality, and those models by virtue of being models never 100% represent reality. Scientists understand this and are always working to improve our knowledge by further refining our models.

Why do you feel like science is not to be questioned? Moreover, if you're dissatisfied as a practitioner with scientific review, why don't you get involved more with the review process? In my field practitioners publish and are involved with our research and peer review all the time, is this not true for your field? Can you work to change it?

In my experience if you don't agree with published research, you don't just lay down and accept it, you do your own research to counter it, and publish that with a big fat citation of the old paper saying they're wrong. Where does this idea come from that science is proclaimed from on high? If you don't like it, roll up your sleeves and prove them wrong -- science is done from the bottom-up.

What you're saying is technically accurate to the intended definition and purpose of science, but if you're seriously asking why these people didn't question or contradict, I'm not sure where you've been for the last several years. So many pandemic response items, in particular, have been "Trust the science"/"Follow the science", and any questions or possible alternatives to the prescribed Method Of Treatment are not only dismissed, but actively persecuted.

Thanks for clarifying. I was actually going to respond in a similar manner as ModernMech. Your original comment came off very defeatist towards the scientific method. I see now that it is aimed at our collective response to the scientific method. I also share your disdain towards this response, especially with pandemic response.

At the same time though, I get it and I do this too. This might be too reductionist but I think most of it comes down to trust. It's very expensive, in terms of time, to have a critical understanding of any specialized field. So taking the shortcuts like past experience, intuition, relying on others are more profitable. Especially if the goal is survival and not understanding of a system. This then becomes a game of finding the most "trustable" person but that's the catch 22.

To be clear the persecution has always been a feature of the scientific process, well before Covid. I don't like it or think it's necessary, but look around -- who can argue with the results? It's always been there, and no one has figured out how to do science without people getting really attached to their ideas and then attacking (sometimes very viciously) others who go against them. Science has factions and politics and rivalries, and all the inherent human baggage associated with those words. It's never been a bunch of mentats or vulcans deliberating over the best ideas without emotion; it's been apes warring over ideas, and not even necessarily the "best", or "good"/"correct" ideas. Did we forget the whole concept of tenure is so that your institution can't fire you if you start thinking the wrong things? Talk about dismissal and persecution!

I don't know who came up with this notion that science is high-minded (probably a scientist) but it's really as petty as any human social process. Go back through history and you'll note plenty of scientists persecuting some other group because they're challenging the orthodoxy. If that makes you uncomfortable maybe science isn't for you, but that's the truth of how it's always worked (and always will work unless you can solve the human angle of this).

Most ideas, especially scientific ones, are almost never accepted on their own merits. They have to be forcefully pushed on the world, and that ruffles some feathers, because that means crowding out some other competing idea. Remember we call this a "marketplace" of ideas. Not everyone comes home from the market happy.

I guess my bottom line is: welcome to how the sausage is made. During Covid some people got a look and found out that they don't like it, shocker. The good news is you're allowed (no one is stopping you) to participate in the process with your own ideas. You just have to steel yourself against criticism and inoculate yourself against the inevitable retaliation and pushback. If you believe in your ideas, don't give in!

This is an excellent question (and reading through those articles now, thank you). I know she thinks about this a lot because her patients bring her research which is obviously broken, but why should they trust her? In her specific case she has done thousands and thousands of an extremely specialized operation - one of very few people in the world to have done that many. She keeps an open mind of course, but she also sees a lot of harm done. Medical research is hard, lots of confounding factors, and almost all the time in her particular area researchers have a very small n, so they draw conclusions because of the small n (and because they are often just not good at research, they are doctors!). I'm sure the scientific method will come through in the long run, that's my opinion, but along the way it is doing a lot of harm too.

>I'm sure the scientific method will come through in the long run, that's my opinion, but along the way it is doing a lot of harm too.

And that's the rub. Most people who try to improve the system reinvent it with the same or worse flaws. You'll notice for all the hand wringing and complaining here about how flawed the process is, no one really has put forth a suggestion on how to fix it. A lot of people here are not even part of the process and looking in from the outside, so they don't even understand the incentives involved enough to fix it. Hence the endless complaining...

Not a surgeon, but in my area often it's not even so much that the results conflict with experience in a clinician versus researcher sense (although they often do), but that the research design itself is obviously flawed, in the sense that the assumptions made in the design, or the corners that are cut, or the way the data is massaged, or whatnot, are completely unrealistic, and the conclusions being drawn are completely inappropriate compared to what was done.

It might even be that you don't have to be an experienced clinician to tell something is wrong, but an experienced clinician would pick up on the problems immediately just through familiarity.

Feynman used the term "cargo cult science" and in my experience it applies to EBM more than any other area I've encountered.

As a statistician, I see a constant flood of incorrect statistical conclusions and poorly designed experiments.

The fact of the matter is that the vast majority of people are just barely competent at even bullshit paper-pushing jobs, let alone highly technical work.

The reason the majority of people end up with higher degrees and the jobs that require them has more to do with wealth/class and social privilege than actual competence or intelligence. It's almost to the point where people who are actually good at something end up with an appropriate line of work almost by coincidence.

That's the natural outcome when you organize your society by wealth instead of skill.

"how does someone in her position know they're right and the research is wrong, and not the reverse?"

This is where you pull out the rusty ol' tools of Figuring Things Out From Evidence, which mostly boil down to:

come up with your own theories and listen to theories from others

think really hard about which one explains historical evidence the most consistently.

think about experiments to make sure theories aren't overfit to historical evidence

conduct those experiments, and "penalize" theories which don't explain the experimental results

these tools are strange things because they (noisily, but surprisingly reliably when applied truly honestly and openly) predict which theories are predictive of future experimental results.

and that's all you want out of a theory, really.

and why should anybody else trust her? well, if she just says "actually, I'm right and the science is wrong", nobody should! Anybody can say that, regardless of whether they're right or not. I could say that!

if she says, though, "actually, I'm right and the science is wrong though, and here's why:"

and explains her entire process, what results she's seen hands-on in her own surgeries, and is careful to also include anything she saw that isn't consistent with her theory..., shows her reasoning, and shows her work... that's convincing and persuasive. It's also not easy, and requires lots of work. So does figuring out things in general.

And that's called "an argument".

And if one bad paper causes a thousand people to have to put in all that work to rebut it, we have a big problem with the system.

It’s sometimes called reasoning from first principles too.

Cannot speak for to this situation in particular, but some highly specialized surgeons track results, in- and out-going data etc. very thoroughly and might therefore actually have better visibility than a researcher looking at a small data set.

You need just one contradicting observation to invalidate something, while a million consistent observations can never fully prove it.

(Paraphrasing Karl Popper, mentioned in the article).

Right, but you only have aggregates here moving the needle. For instance, your experience is that intervention A leads to >90% mobility elbow post-surgery in 90% of patients and intervention B leads to 80% mobility in 90% of patients.

What epistemological techniques do you use to distinguish intervention A from intervention B and how do you distinguish your experience from the paper that shows the opposite result?

The number of variables are large: including, distressingly, your skill at technique A and technique B. Is the difference in the outcome merely your lack of skill in one? Is the difference merely the lack of skill of the guys observed by the paper?

My parents are also experienced surgeons and the normalizing method they use is to share their findings with fellow surgeons. Patients where they work are much more amenable to having photographs of their surgeries used as discussion material.

Medical science is a primitive science and it is practised with primitive statistics, so most evidence is just missing. Lots of surgical techniques are discussed based on too little evidence with massive true error bars.

And what's puzzling to me is how so many people take Popper's theoretical discussions to be of such validity in the real world

The experimentalist world is messy and full of hidden dependencies.

(And to be fair it's usually the people with less experimental experience that take him the most seriously)

> You need just one contradicting observation to invalidate something

No, that's wrong. There are several reasons why a negative result doesn't necessarily disprove a theory or a hypothesis: measurement uncertainties/noise, experimental errors, software bugs, bit flips, ...

It's a very important observation you made about various theories. I find that in medicine, almost always counter evidence of an established belief is either completely ignored or explained way casually.

Reminds me of that poor guy who told surgeons they should wash their hands. They didn't take kindly to that.

I’d give the doctor above a little more credit than that, and also would expect the papers she dislikes to be far less impactful than that.

Clinicians I know who get frustrated with academic research that draws conclusions about clinical work are criticising papers that are more like this:

(I’m exaggerating for effect, but not exaggerating a lot)

“People think this clinical technique is effective. But when some academics with very little clinical experience performed a rough approximation of the technique, in the wrong area, with the wrong tools, in the wrong way, on a small number of randomly chosen patients, they found it performed no better than placebo. Therefore no one should use that technique on any patient.”

That would be Ignaz Semmelweiss. I think you are understating his actual life outcome for having challenged the medical establishment with hard proof.

Poor doesn't begin to describe it.

https://en.wikipedia.org/wiki/Ignaz_Semmelweiss

Reply on news.ycombinator.com