Hacker News Reader

Comment by conorh | Hacker News Reader

My wife and I were just talking about this yesterday. She is an extremely specialized surgeon and she is horrified by the academic publishing in her area of expertise. It is often utterly wrong and conflicts directly with her experience. I'm not sure she'd agree with the reasons mentioned in this article here as the main issue - although we have personally seen less ethically motivated doctors publishing research to help line their pockets via the industry. In her opinion the main issue is that academic doctors HAVE to publish in order to advance their careers. So you end up with a never ending stream of terrible research, churned out by doctors that are not experts in doing research and that actually spend most of their time doing other things. This happens even at the best centers; at least in her area of expertise. I don't know what the solution is, but I do know that whenever she has needed treatment for herself she generally ignores most of the research and talks to the most experienced surgeon she can find.

View on HN

> My wife ... is an extremely specialized surgeon and she is horrified by the academic publishing in her area of expertise. It is often utterly wrong and conflicts directly with her experience.

Genuine question: how does someone in her position know they're right and the research is wrong, and not the reverse?

I'm certainly not denying that she may be correct, and I'm not unquestioningly defending research evidence either.

We've come a long way from "eminence-based medicine", where the senior clinician was impossible to challenge. Research has helped us move towards a more nuanced approach to care (I would say "holistic" but that word has become tarnished).

EBM "phase one" alienated clinicians despite explicitly acknowledging clinician expertise, by overemphasizing evidence and undervaluing experience. EBM "phase two" aims to redress the balance, but perhaps the mistrust is too significant now. It would be a shame if so.

Some reading:

https://www.bmj.com/content/348/bmj.g3725

https://www.cebm.ox.ac.uk/news/views/it-is-time-for-evidence...

This is an excellent question (and reading through those articles now, thank you). I know she thinks about this a lot because her patients bring her research which is obviously broken, but why should they trust her? In her specific case she has done thousands and thousands of an extremely specialized operation - one of very few people in the world to have done that many. She keeps an open mind of course, but she also sees a lot of harm done. Medical research is hard, lots of confounding factors, and almost all the time in her particular area researchers have a very small n, so they draw conclusions because of the small n (and because they are often just not good at research, they are doctors!). I'm sure the scientific method will come through in the long run, that's my opinion, but along the way it is doing a lot of harm too.

>I'm sure the scientific method will come through in the long run, that's my opinion, but along the way it is doing a lot of harm too.

And that's the rub. Most people who try to improve the system reinvent it with the same or worse flaws. You'll notice for all the hand wringing and complaining here about how flawed the process is, no one really has put forth a suggestion on how to fix it. A lot of people here are not even part of the process and looking in from the outside, so they don't even understand the incentives involved enough to fix it. Hence the endless complaining...

"how does someone in her position know they're right and the research is wrong, and not the reverse?"

This is where you pull out the rusty ol' tools of Figuring Things Out From Evidence, which mostly boil down to:

come up with your own theories and listen to theories from others

think really hard about which one explains historical evidence the most consistently.

think about experiments to make sure theories aren't overfit to historical evidence

conduct those experiments, and "penalize" theories which don't explain the experimental results

these tools are strange things because they (noisily, but surprisingly reliably when applied truly honestly and openly) predict which theories are predictive of future experimental results.

and that's all you want out of a theory, really.

and why should anybody else trust her? well, if she just says "actually, I'm right and the science is wrong", nobody should! Anybody can say that, regardless of whether they're right or not. I could say that!

if she says, though, "actually, I'm right and the science is wrong though, and here's why:"

and explains her entire process, what results she's seen hands-on in her own surgeries, and is careful to also include anything she saw that isn't consistent with her theory..., shows her reasoning, and shows her work... that's convincing and persuasive. It's also not easy, and requires lots of work. So does figuring out things in general.

And that's called "an argument".

It’s sometimes called reasoning from first principles too.

And if one bad paper causes a thousand people to have to put in all that work to rebut it, we have a big problem with the system.

You need just one contradicting observation to invalidate something, while a million consistent observations can never fully prove it.

(Paraphrasing Karl Popper, mentioned in the article).

Right, but you only have aggregates here moving the needle. For instance, your experience is that intervention A leads to >90% mobility elbow post-surgery in 90% of patients and intervention B leads to 80% mobility in 90% of patients.

What epistemological techniques do you use to distinguish intervention A from intervention B and how do you distinguish your experience from the paper that shows the opposite result?

The number of variables are large: including, distressingly, your skill at technique A and technique B. Is the difference in the outcome merely your lack of skill in one? Is the difference merely the lack of skill of the guys observed by the paper?

My parents are also experienced surgeons and the normalizing method they use is to share their findings with fellow surgeons. Patients where they work are much more amenable to having photographs of their surgeries used as discussion material.

Medical science is a primitive science and it is practised with primitive statistics, so most evidence is just missing. Lots of surgical techniques are discussed based on too little evidence with massive true error bars.

> You need just one contradicting observation to invalidate something

No, that's wrong. There are several reasons why a negative result doesn't necessarily disprove a theory or a hypothesis: measurement uncertainties/noise, experimental errors, software bugs, bit flips, ...

And what's puzzling to me is how so many people take Popper's theoretical discussions to be of such validity in the real world

The experimentalist world is messy and full of hidden dependencies.

(And to be fair it's usually the people with less experimental experience that take him the most seriously)

It's a very important observation you made about various theories. I find that in medicine, almost always counter evidence of an established belief is either completely ignored or explained way casually.

Reminds me of that poor guy who told surgeons they should wash their hands. They didn't take kindly to that.

I’d give the doctor above a little more credit than that, and also would expect the papers she dislikes to be far less impactful than that.

Clinicians I know who get frustrated with academic research that draws conclusions about clinical work are criticising papers that are more like this:

(I’m exaggerating for effect, but not exaggerating a lot)

“People think this clinical technique is effective. But when some academics with very little clinical experience performed a rough approximation of the technique, in the wrong area, with the wrong tools, in the wrong way, on a small number of randomly chosen patients, they found it performed no better than placebo. Therefore no one should use that technique on any patient.”

That would be Ignaz Semmelweiss. I think you are understating his actual life outcome for having challenged the medical establishment with hard proof.

Poor doesn't begin to describe it.

https://en.wikipedia.org/wiki/Ignaz_Semmelweiss

As a statistician, I see a constant flood of incorrect statistical conclusions and poorly designed experiments.

The fact of the matter is that the vast majority of people are just barely competent at even bullshit paper-pushing jobs, let alone highly technical work.

The reason the majority of people end up with higher degrees and the jobs that require them has more to do with wealth/class and social privilege than actual competence or intelligence. It's almost to the point where people who are actually good at something end up with an appropriate line of work almost by coincidence.

That's the natural outcome when you organize your society by wealth instead of skill.

> Genuine question: how does someone in her position know they're right and the research is wrong, and not the reverse?

She doesn't know, and that's the problem with medical research: being a specialized surgeon is like a carpenter who's made hundreds of a particular kind of furniture. Then a scientist comes along, saying we should change how we do it. As an experienced carpenter, you know that most of those scientists actually have very little practical experience, because they used that time to publish and are actually very bad, practically speaking. And therefore oftentimes reach nonsensical conclusions. So, what to do? You go ask the most practically experienced guy you can find, hoping he'll be able to guide you. Repeat 100x/year.

I'm not exaggerating: some of our academic practitioners who've published a lot are really among the worst practitioners I know, and do kill people who should have lived on a regular basis! Ironically, some of them are respected professors and patients often seek their care thinking it'll be better. One of them is a surgeon and has received an award for outstanding teaching. He's the absolute worst surgeon I know.

So, no easy conclusion here...

> scientist comes along, saying we should change how we do it

That doesn’t sound like science. An example or pointer would be helpful.

Remember the start of the pandemic? Some prof published that we should withold steroids and intubate ASAP on the basis of preliminary evidence, throwing basically everything we knew about ARDS out of the window. Most reasonable ICU guys were skeptical on the basis of their own practical experience, but had no choice because who would dare contradict science, right?? We very likely killed many people that way.

What do we do now? The exact opposite, and much closer to usual ARDS practices. You can argue that wasn't a mistake because it followed the process of scientific discovery, but it's nevertheless a tragic example of experience trumping the advice of the scientific community.

I feel like we should be critiquing society’s response to science rather than implying that their paper should never have been published.

> implying that their paper should never have been published.

That's really not what I meant. I have no problem with the publication. But put yourself in the shoes of someone such as myself, who's intubated tens of cases on admission and probably killed many doing so. Wouldn't you be questioning the structure of our research system then? Not that I have any better solution at hand, mind you...

> but had no choice because who would dare contradict science, right??

This is what I don't get. Science is contradicted and questioned all the time. It's the main mechanism by which science evolves.

Also a crucial feature of science is that it's literally always wrong. Science is the iterative process of building models of reality, and those models by virtue of being models never 100% represent reality. Scientists understand this and are always working to improve our knowledge by further refining our models.

Why do you feel like science is not to be questioned? Moreover, if you're dissatisfied as a practitioner with scientific review, why don't you get involved more with the review process? In my field practitioners publish and are involved with our research and peer review all the time, is this not true for your field? Can you work to change it?

In my experience if you don't agree with published research, you don't just lay down and accept it, you do your own research to counter it, and publish that with a big fat citation of the old paper saying they're wrong. Where does this idea come from that science is proclaimed from on high? If you don't like it, roll up your sleeves and prove them wrong -- science is done from the bottom-up.

What you're saying is technically accurate to the intended definition and purpose of science, but if you're seriously asking why these people didn't question or contradict, I'm not sure where you've been for the last several years. So many pandemic response items, in particular, have been "Trust the science"/"Follow the science", and any questions or possible alternatives to the prescribed Method Of Treatment are not only dismissed, but actively persecuted.

Thanks for clarifying. I was actually going to respond in a similar manner as ModernMech. Your original comment came off very defeatist towards the scientific method. I see now that it is aimed at our collective response to the scientific method. I also share your disdain towards this response, especially with pandemic response.

At the same time though, I get it and I do this too. This might be too reductionist but I think most of it comes down to trust. It's very expensive, in terms of time, to have a critical understanding of any specialized field. So taking the shortcuts like past experience, intuition, relying on others are more profitable. Especially if the goal is survival and not understanding of a system. This then becomes a game of finding the most "trustable" person but that's the catch 22.

To be clear the persecution has always been a feature of the scientific process, well before Covid. I don't like it or think it's necessary, but look around -- who can argue with the results? It's always been there, and no one has figured out how to do science without people getting really attached to their ideas and then attacking (sometimes very viciously) others who go against them. Science has factions and politics and rivalries, and all the inherent human baggage associated with those words. It's never been a bunch of mentats or vulcans deliberating over the best ideas without emotion; it's been apes warring over ideas, and not even necessarily the "best", or "good"/"correct" ideas. Did we forget the whole concept of tenure is so that your institution can't fire you if you start thinking the wrong things? Talk about dismissal and persecution!

I don't know who came up with this notion that science is high-minded (probably a scientist) but it's really as petty as any human social process. Go back through history and you'll note plenty of scientists persecuting some other group because they're challenging the orthodoxy. If that makes you uncomfortable maybe science isn't for you, but that's the truth of how it's always worked (and always will work unless you can solve the human angle of this).

Most ideas, especially scientific ones, are almost never accepted on their own merits. They have to be forcefully pushed on the world, and that ruffles some feathers, because that means crowding out some other competing idea. Remember we call this a "marketplace" of ideas. Not everyone comes home from the market happy.

I guess my bottom line is: welcome to how the sausage is made. During Covid some people got a look and found out that they don't like it, shocker. The good news is you're allowed (no one is stopping you) to participate in the process with your own ideas. You just have to steel yourself against criticism and inoculate yourself against the inevitable retaliation and pushback. If you believe in your ideas, don't give in!

This reminds me of agriculture under the Soviet Union. Stalin wanted to build a technocracy with the brightest in charge of there specialized fields, so he brought in the most famous Russian agronomist, Lysenko, and gave him complete control over the Soviet agricultural system. Lysenko took control away from those ignorant and backward Russian farmers and revolutionized Soviet farming with all the latest scientific breakthroughs.... It wasn't long before millions of people were starving to death.

Except that this is not the only (or even the main reason) the big starvation happened. It was because Stalin had to pay the US for their technological knowledge and deliveries of factories and as the Ruble wasn't worth much the US demanded to be payed in natural resources, mainly agricultural products. As Stalin saw the writing on the wall in regards to the danger of a middle European nation trying to overtake Russia he made that deal and ferried most of the Russian produce towards the US.

So to be able to (at first) produce tractors and later switch that to tanks he sold his people to the US that knowingly and gladly accepted the cheap food.

So both nation's political and industrial leaders are responsible for the mass starvations. One can't just blame it on Stalin or on Lysenko.

But Lysenko might have made the problem even worse. That much we can agree on.

Source for further reading: https://twitter.com/kamilkazani/status/1505247886908424195

So if a parent spends their whole paycheck on a new shotgun and their child goes hungry, it is equally Remington's fault. Got it

Remington’s raison d’être is producing shotguns.

The US’ raison d’être is (hopefully) not knowingly starving millions of people to death. When you’re making a deal of that size (between countries, not a company and consumer) no deal is purely transactional, there are ramifications.

The point is that the US (government) knew at the time of these ramifications. So in my book they were complicit.

Not that any other country would have acted differently. I am not here to metaphorically shit in the yard of the US. It is just business as usual when it comes to politics. Sadly. I just don't expect people to act with integrity when in positions of power. I still like to be pleasantly surprised if it happens - I just don't expect it.

Edit: Typo

It took me a while to understand in which direction you are arguing.

Of course the US is complicit (just not solely complicit as you imply I claimed - nice rhetorical figure by the way just a bit too obviously applied). They knowingly let a dictator pay them in natural resources while these resources were lacking for the people and this lead to famine and mass starvation.

So to stay in your image: When Remington sells me a shotgun knowing I will go and use it to kill my family (because it was clearly visible) they are complicit.

If the US would not have know the consequences of their deal with Stalin for the ordinary people we would be talking about a different situation here. But the US knew. And they still took the grain and other resources. And that makes them complicit.

In my world, that makes the U.S. very clearly an accomplice to a dictator who is starving his people. And also profiteers from such a system.

But hey as if this were something new or shocking. I don't know about US educational system, but i thought it was public knowledge, that the US acted that way as their standard modus operandi for many decades now.

No matter if with South American dictators, with African dictators or even with dictatorial regimes in the Middle East, as long as they do not become too rebellious and do not challenge the supremacy of the USA the US are willing to trade and not look too closely towards the crimes against humanity these dictators are performing.

Take Saddam Hussein as an example.

Wasn't it the U.S. that continued to financially and militarily support Saddam Hussein in the Iraq-Iran war even after he openly used chemical weapons? Without the U.S., Saddam would never have become head of state in the late 1970s. And the US gladly bought his oil - ignoring what he did.

This is the same as we Germans are currently also responsible (at least in part) for the prolongation of the war in Ukraine, because we still do not want to give up the gas supply from Russia. We are prolonging the war because our money is helping to finance Putin's military advance.

Our convenience and the intransigence of our government makes us complicit in the death of many civilians.

> So to stay in your image: When Remington sells me a shotgun knowing I will go and use it to kill my family (because it was clearly visible) they are complicit.

No, that's a very different metaphor.

The item being bought causes no harm. The problem is reckless spending. That puts the bar of responsibility in a very different place.

If the parent needs a shotgun because of Nazis on their front door and has no real choice, then yeah, pretty much. Not "equally" but not zero.

That thread establishes that Stalin exported agricultural products, it doesn't say that the US businesses were trading directly for grain.

The US was probably a net grain exporter at the time, so I think you are not presenting the situation very well. There is certainly complicity in working with an oppressive government that is starving its people, but saying it was done for "cheap food" when USSR was selling the food elsewhere and paying in dollars isn't the right way to present it.

It's a great way to present it if you were a present-day dictator and wanted to justify invading a neighbouring grain-producing country and blame it on historic wrongs perpetrated by the evil USA.

False, Russia was attempting to rapidly industrialize and needed to purchase foreign technology and needed foreign currency to do so.

So he sold the grain that would feed his people in exchange for USD, Deutschmarks, Pound Sterling that he could use to buy industrial equipment.

I mean, it's not exactly out of place for Stalin to sacrifice his people's lives to realize his goals for Mother Russia. Plus the added benefit that the people dying were troublemakers anyways (Ukrainian nationalists).

Do note that when the famine did strike, those same foreign countries offered famine relief that Stalin promptly turned down.

> ...had to pay the US...

had to or chose to?

It's an interesting thread, thanks for the link, but "knowingly and gladly" is not supported there, I don't see how you can lay any responsibility for Holodomor at Henry Ford's feet.

This is happening in Sri Lanka on a smaller scale right now under the guise of organic farming.

I can totally believe that. What you’re describing has an uncanny resemblance to academic software engineering research: many of the published insights scale up all the way to one semester student projects — in other words, are completely useless in the real world.

Not the OP, but one of the organisations I work for publishes best practice guidelines for diagnostic tests for various rare genetic disorders in an attempt to avoid this issue. However the process takes between 1 and 2 years and starts from identifying a group of leading clinicians and academics who specialise in that condition who are willing to write the guidelines (on their own time, for free). They then do a literature search to gather any relevant material and at the same time put out a call for submissions to clinicians around the world asking them to contribute. A (usually virtual) conference is then held and the writing begins. Over the next few months a set of draft guidelines are produced and then put out for comment. After a period for feedback a second draft is produced and sent out again. A consensus building exercise is then conducted (aka another mini conference involving the guideline writers and others) and the guidelines finalised according to the consensus. They're then published in a peer reviewed journal.

As you can imagine this takes time and money and all of it is sponsored by a not for profit EQA organisation, with no pharma companies involved (since they're who provide the tests and that's a massive conflict of interest). As a result the org only manages to produce two or three guidelines a year, when really they'd like to be able to produce a dozen or more. And they also need to be reviewed at regular intervals which while slightly less long winded is still long winded. Really this is the sort of thing that is a prime candidate for being sponsored by government, and sometimes it is, but since these are rare conditions the cost / benefit analysis is not always favourable.

Afrer getting degree and working in the software industry im kinda worried when going to the doc, LOL.

Not to trash the profession, but in reality it's no different than a computer engineer, plumber, accountant or anything else.

I used to think "well med school is really hard, so the bare minimum for a doctor is pretty good". Then I started working in healthcare and realize, no, that's not true at all.

The best example I can think of is when I worked on a project where we wanted to get ready for a much better competitor in a disease area. Not "slightly better", but "mind blowing better and not using it would be malpractice". Most doctors had heard of it, but we want to remind them not to purchase the company's older drug because they'll just return it a year later.

Turns out there were multiple doctors, after the new drug was approved and available, that continued to use the old drug. We literally had to go to them and say "hey stop buying the company's drug please". And I realized, damn, these doctors' patients are getting really shitty treatment.

This is why pharmaceutical sales representatives are so important. Most doctors are far too busy treating patients day to day to stay up on every new drug that comes out. CME can help cover some of this, especially with new procedures or treatments, but nothing can keep up with the 100 new drugs for diabetes. what really helps doctors is having people that can come talk to them, tell them that drug X is better than Y, give them the literature if they want it, tell them what insurance has a chance of covering it. Some doctors will always do what they know, but the majority just need to know there is a better option.

Or just follow the clinical guidelines and associated updates. They're there for this exact reason, to make sure every doctor uses up-to-date methods. No need for sales reps in this case.

How quickly do you think clinical guidelines are updated.

If a new cancer drug was launched that could help would you prefer a doctor that said “sorry, the next guidelines update is in 9 months, so I’m not using that drug”?

How can they trust the literature? The article was just talking about how the evidence can't be trusted.

Doctors aren’t stupid. They can judge literature themselves.

My 60 years old father was at the hospital for a heart attack and the doctor wrote in his form he was a pregnant woman. That day I lost hope for any kind of baseline.

Not a surgeon, but in my area often it's not even so much that the results conflict with experience in a clinician versus researcher sense (although they often do), but that the research design itself is obviously flawed, in the sense that the assumptions made in the design, or the corners that are cut, or the way the data is massaged, or whatnot, are completely unrealistic, and the conclusions being drawn are completely inappropriate compared to what was done.

It might even be that you don't have to be an experienced clinician to tell something is wrong, but an experienced clinician would pick up on the problems immediately just through familiarity.

Feynman used the term "cargo cult science" and in my experience it applies to EBM more than any other area I've encountered.

Cannot speak for to this situation in particular, but some highly specialized surgeons track results, in- and out-going data etc. very thoroughly and might therefore actually have better visibility than a researcher looking at a small data set.

This is all true. Welcome to the real world.

> academic doctors HAVE to publish in order to advance their careers

Indeed. But the issue is what criteria will you use for promotion if not that? Don't even think of evaluating clinical results, that's even worse than what we have.

> This happens even at the best centers

This happens ESPECIALLY at the best centers. Because centers are in good part ranked according to citation counts.

> I don't know what the solution is, but I do know that whenever she has needed treatment for herself she generally ignores most of the research and talks to the most experienced surgeon she can find.

After 10+ years in the system, I'm in the same boat. And although I've given it a good thought, I don't see a solution either.

The truth this article touches on is that it's not even really about citation counts anymore, it's about dollars. Via drug companies, private grants, federal grants, any sorts of money coming in from outside the university or hospital. I've sat on promotion meetings where people aren't publishing, but are bringing in 2 million dollar grants and "it's all ok".

This incentive structure distorts research. What to do about it? I'm not exactly sure but having university administrations who see research as a means to make money, rather than an end in itself, is not the way.

Put controls in place. Monetary openness. Every dollar accounted for.

That's possibly the worst thing that could happen to science. Academic research should not be an industry. The fact that it is now considered as such in the western world will be our downfall.

From the looks of it, it already seems like an industry even it tries to appear not to. Ofcourse not all sectors will have the issue, just some (medicine, high tech for example)

Putting safeguards in place is a good thing. It’s like traffic rules that people need to adhere to.

Nothing will ever get done, but at least the busywork will be well documented!

Stop trying to apply the 'free market' or capitalism to every situation. The problem is more fundamental than your field: As long as those economic tools are treated as sacrosanct, universal solutions by society in general, your field inevitably will suffer.

(They are very useful tools, but like very tool, not the solution to every problem.)

> This incentive structure distorts research. What to do about it? I'm not exactly sure but having university administrations who see research as a means to make money, rather than an end in itself, is not the way.

A lot of the reasons of this distortion is due to the lack of funding from government in state universities which keeps going down as the cost of running a university keeps going up. Because of the rising costs, things get pushed to extremes and the accountants/administrators are put between a rock and a hard place.

Depending on the university in question, it might not be just about "making money" but just not shutting the university down and trying to swim upstream in a never-ending battle.

I agree that's part of it. But I've also seen university administration proudly let state governments know that they're weaning themselves off of state funding. That can be seen in different ways, but it's clear that they've internalized the message so much that they don't push back against it at all.

More fundamentally, I think the US really needs to step back to the basics and think about different ways research can be funded, and to what extent current funding systems map onto those different mechanisms. I think there's a lot of dishonesty about what is going on, in terms of how research is funded in theory versus what actual expectations or practices are, and what the consequences of all of it is.

That's also definitely true for universities with enough reputation and infrastructure to score large contracts/grants with private sector corporations (or mega US defense contractors).

I also know of a couple of chemistry professors and one is new and his PhD was overseas in Saudi Arabia and he basically said it was just large scale brute force data analysis and discovery of chemicals in Python with a small amount of lab work to prepare the datasets. Basically there's a huge pipeline for this sort of work to discover new chemicals in labs, get a PhD out of it, but it's all just sort of an engine that functions to brute force the chemical possibilities not real meaningful research. I never asked but I assume the results just get fed into a pharma company or something.

Then you have those chemists on the opposite end, ready to retire, who are working on cold fusion because why not it's interesting to study even if it goes nowhere.

Our whole society has kind of turned into a giant web of a machine that can't be untangled without ripping everything apart. Some of it is stupider than others, but most of it is kind of a bullshit layer on top of a bullshit layer. We obviously see that in IT as well, with marketing tools that are just wrappers for simple underlying tech that anyone could set up. Sometimes there is significant value added, sometimes there is none (or even negative value).

I don't know where I'm going with this so I'm going to stop typing.

I'm not so sure the costs of running a university are going up. Here is expenditure data from the ed department:

https://nces.ed.gov/programs/digest/d20/tables/dt20_334.10.a...

Expenditure per full-time-equivalent student in constant 2019-20 dollars is up about 10% over a decade in the instructional and student services categories. That's not that much. (The total number is going up, driven by costs in hospital services, mostly)

Would estimating the prior help? Eg this study claims the sky is red and most people estimate that is wrong. Therefore this study is likely wrong and needs a lot of evidence to overcome the prior.

Yes but the thing is… papers are counted, not read. They are published to be counted, not read.

Really.

> But the issue is what criteria will you use for promotion if not that?

Dice roll? Alphabetical order? If it's gonna be arbitrary or close-to, it would be preferable for everyone if it didn't erode science as a side effect.

How is ranking by clinical outcomes “even worse” than grading by quantity of papers published?

Because it pushes us to treat cases where we expect a good outcome preferentially, i.e. biases the case mix towards easy patients and increases inequalities in access to healthcare.

This is true in any field. Many publications are trash. It takes a lot of experience to be able to tell apart bad science from good science in a specific field.

Every PhD knows how to read literature. But throw them outside their narrow line of expertise, and they will not be able to tell apart a good from a bad paper. It's the same reason you cannot trust your FB Karen who found a publication that supports her conspiracy theory.

So yes your wife is right in looking for the most experienced person in the field for an opinion.

> never ending stream of terrible research

When I was in elementary school, I saw the same thing. People forced to write essays on things they were not necessarily good at, just in order to advance to the next year. The result, of course, was a crapton of half-assed essays, many of which were bullshitted together not long before the deadline. I sincerely doubt even half of them where in any way useful whatsoever.

At least in my school, few people improved their essay skills over time. Maybe they learned more technically impressive ways to bullshit, but their writing did not improve. (Some people's writing improved, but often they turned out to be writing things on their own interest in their spare time.)

I know at the time I said to myself, "This is dumb, but it's just school. It's not like this for adults, of course. They have their shit together."

Turns out many adults still do that!

This is bad on so many levels it's bordering on unintentional evil. Someone should start a revolution.

Jeez, everyone here at level X are a bunch of clowns winging it. It's not like this for level X+1, of course. They have their shit together. :-)

Then at some point it dawns that it's just clowns from bottom to top.

Exactly. That's the big mystery as far as I'm concerned: how the hell does the world work as well as it does, given how clueless the people running it are?

I'm starting to suspect it speaks to the adaptability of people.

Not knowing the medical field but being involved in research in technical fields before I can confirm the problem with the "HAVE to publish" phenomenon. I came accross sooo much long but barren papers that it was a tiresome and most times fruitless excercise just to follow proper scentific methodology in terms of being informed of related works that I am actually glad I do not need to live in the academic reasearch world anymore (I still do in industrial settings, "improperly", tied to developments). I also hated that I had to make up a quota even if the research did not yield interesting enough (part) results yet, figuring out something that could go into a paper, for the sake of writing a paper alone, not for communicating new knowledge yet.

Formality not content ruled.

(I just hope other places/institutions/fields do it better)

The pandemic was an interesting case study in this. In the sense that although the alternative medicine people who started writing about covid after the pandemic started were generally giving terrible advice, the alternative medicine people who were writing about how to survive a novel coronavirus epidemic before December 2019 were generally much more accurate than the mainstream medical people who only started learning about coronaviruses after the pandemic started.

It turns out, being genuinely interested in something when there is zero money to be made is a much better predictor of success than having formal training and insider status.

It would be cool if you could do an analysis of the alternative medicine people.

I question that Golang* is a good programming language but lots of expert coders disagree. While your theory has merit I wonder if at the top of surgery everyone has subtle and nuanced ideas to the point of when telling others of their experience that because these surgeons are so good their claims can’t be translated as easily as every expert has different ideas. I don’t doubt that there are some people as you say with a profit motive to bullshit everyone I just question the proportion given how bespoke being at the top of any industry might be.

* sorry Golang people and anyone offended by this it’s just an example

25 guys from IBM, 20 from Microsoft, 3 from Facebook, 10 from Google, 14 professors from different universities sit on a body that dictate how you're suppose to program. If you don't put 20 references in your source code to "The Gang of 4", you're credentials will never be enough to do anything but write doubly nested for loops.

Luckily the IT industry can just ignore them and demonstrate something better without bothering 6 other organs to check if they are allowed to build it.

But in truth i'm starting to worry about the Hollywood-NSA-Savethechildren 'consultants' influence on the laws that panders 'technical' solutions to make general computation a service instead of a human right.

Also if you are a effective surgeon, how much time you have or have to spend on writing and formulating scientific research? I wonder if there is addition of research to the old adage of teaching...

So much of what you use today and the modern world runs on research and the scientific process in general, warts and all. Your comment is hubris at its finest, and insulting to researchers and teachers.

Reply on news.ycombinator.com