A Vital Measure: Your Surgeon’s Skill | Hacker News Reader

A Vital Measure: Your Surgeon’s Skill

well.blogs.nytimes.com

84 pointstokenadult48 comments

The UK has a concept of patient choice. The patient can choose one of four hospitals to have their surgery. This is supposed to drive up quality.

Obviously it doesn't. Experts have difficulty telling why one surgeon is better than another (Ann works on more difficult cases than Bob; Ann uses an older technique than Bob; Ann isn't as good as Bob; etc etc) so patients end up choosing based on waiting times, or the car parks, or weird things.

There's a lot of room in medicine for better use of numbers. It's weird that "ethics" prevents us from using these data to save lives.

Ethics used to prevent pharmacists from telling patients what was in medication bottles. Norms can change.

People often forget that the medical profession's highest priority isn't the health of you, the patient. If you were to grab a medical ethics book you will first be told that the well being of the population as a whole surpasses any individuals rights.

Let's assume a hypothetical where doctors are ranked, let's also assume that insurance will pay for all doctors. Naturally demand for these physicians or surgeons will rise dramatically. If we take Ontario (Canada) as a case, we can see a situation where there is not a free market and physicians can't start charging more. What happens? Well wait times become significantly longer and people start waiting longer to see the best physician possible rather than having a more immediate operation with a lower ranked physician. There is, of course, increased risk the longer you wait for an operation even as simple as a hernia. But patients often don't understand that and can't make logical choices without being 'forced', for lack of a better word, and told they must do it soon with surgeon X. In this case overall population health decreases, complications, morbidity and mortality rise from delays. You also run the risk of complex patients being unable to be operated on by the best because they are too busy doing an operation on a cut and dry case. This is the case in Ontario (largely anecdotal evidence, albeit from a very large pool) where people flock to Toronto on the perceived notion that surgeons affiliated with the most academically prestigious medical school are better than those in a community centre.

It is important to ask whether the lower ranked surgeons should or should not be able, and allowed, to perform the operations, if they are unqualified they shouldn't be allows to practice. The idea of rankings to improve patient care really rests on the notion that the lower ranked surgeons have an effectively higher risk rate (note that I did not say statistically significant). Saying that complications go down from 2% to 1% with higher surgeons while significant is not effectively a difference that should affect choice. I don't know the precise numbers but this is something that must also be considered before making any judgements. While I have no vested interest, and no opinion on the matter, it is important to think why the system is the way it is rather than blindly changing things in the name of progress. This system would allow physicians to charge more in the US (due to more demand, which has it's own slew of ethical considerations) and the medical establishment is greedy to say the least. If the top surgeons could make more money with little to no harm they would.

Rationing with waiting lists is dumb, always has been and always will be. The smart ways to ration are by money (better surgeons charge more), or by QALY (better surgeons do more important operations on sicker people). And even with single-payer, there's no reason you can't pay better surgeons more.

It's true that not everyone can have a top surgeon. But for serious surgery, the "complication" is frequently death. It's extremely silly to tell people that 1% chance of death vs. a 2% chance of death "shouldn't affect their choice". I don't know about you, but I don't want to die, even as I accept that small chances of death are sometimes unavoidable.

Regarding medical ethics, obviously in the abstract we should prefer saving ten lives to one life. But in practice, it's often better to have a rule like "don't kill people" than "kill people when it serves the common good". "Don't kill people" is very easy to understand, and very easy to enforce. The "common good" is abstract and fuzzy, and can be used to justify anything from fighting malaria in Africa to Stalin's Holodomor ("can't make an omelet without breaking some eggs").

Sorry but you're incorrect. Your assumption of the complication being death is exaggerated, the surgery in question here is not that serious. Even if it were death someone would have to model the adverse effects of the increased waiting list. People waiting might result in an increase in mortality surpassing the doubled risk from surgeon error.

How do you quantify which surgeons are better? I'm not an expert but surely this study's methodology is not without it's own assumptions. Things like complication rate vary wildly with patient population and other factors. This isn't a telemarketing job where you quantify easily. Similarly how can you quantify who the best programmer is? What is the economic cost of quantifying who the best surgeon is? How expensive is it to apply this study to every surgery? We must ensure that the rankings system is valid before applying it to prevent perverse incentives such as only taking low risk cases. You don't want a USNews Week type system like you have for universities which has obvious flaws and biases.

You say waiting lists are dumb, but why? Are you 'killing people' by letting an inferior surgeon operate? Why is 'don't kill people' easy to understand and enforce? By not killing one to save ten are you not killing ten people? You're seeing this as black and white. This is not the case. Whether it is better to save one or ten I am not the person to judge, but you are oversimplifying this drastically.

The ethical question here is NOT sentencing someone to death like killing someone to harvest organs. It's more similar to doing nothing to save someone most likely to die in order to save ten, like how they do military triage.

Life is 'abstract and fuzzy'. This isn't an engineering challenge, as you say there are several things to consider. Stalin ignored the human aspect, is this right or wrong? How can you blindly answer this without supporting your conclusion. What seems humane isn't always the right answer to things. By your same argument are layoffs ever justified despite it being for the common good for the company and the remaining employees?

> But in practice, it's often better to have a rule like "don't kill people" than "kill people when it serves the common good".

Eliezer had a nice take on this principle in http://lesswrong.com/lw/uv/ends_dont_justify_means_among_hum....

Relevant quote:

"The end does not justify the means" is just consequentialist reasoning at one meta-level up. If a human starts thinking on the object level that the end justifies the means, this has awful consequences given our untrustworthy brains; therefore a human shouldn't think this way. But it is all still ultimately consequentialism. It's just reflective consequentialism, for beings who know that their moment-by-moment decisions are made by untrusted hardware.

> People often forget that the medical profession's highest priority isn't the health of you, the patient. If you were to grab a medical ethics book you will first be told that the well being of the population as a whole surpasses any individuals rights.

That is just not true. Completely, 100%, entirely false. In medical decision making, the principles of beneficence, malfeasance, and autonomy far outweigh the concept of social justice. As someone who works in the medical setting, hearing something like this stuns me.

Alphaoverlord is 100% correct. While standards and norms may vary from place to place, the textbooks that I have read on medical ethics are in agreement. In my pre-medical, medical, and postgraduate-medical training, I have always been taught that my mandate is to do the right thing for the person sitting in front of me. This in itself can lead to population-level problems, but that is a separate issue (social justice, the 4th of the principles that alphaoverlord outlined).

Also, a nitpick: it's nonmaleficence (doing no harm), not malfeasance.

haha, good catch! thanks.

Perhaps ethical standards vary among different locations or specialties? That's certainly true in other fields besides medicine. Also, there is often a difference between stated ethical standards and the ethics implicit in common practice.

In my observation, physicians are as a whole more ethical than are many other occupations. However, they're still human.

Data on patient outcomes for individual healthcare providers is not currently available, but the data at the hospital level is. If a hospital is a Medicare-certified, then it is required to report quality of care metrics including deaths, complications and readmissions.

When researching and comparing doctors, as rschmitty mentions, gather all the relevant publicly-available data such as your state's malpractice claims database, research publications, and quality of associated institutions.

Check out Medicare's Hospital Compare tool to see the quality of care metrics for your local hospitals: http://www.medicare.gov/hospitalcompare/search.html

Here is the quality of care dataset that includes over 4,800 US hospitals: https://data.medicare.gov/Hospital-Compare/Agency-For-Health...

The state of New York, every couple of years, publishes a report that rates all of the heart surgeons, including how many operations they did and how many of their patients died. This data is shown at both the surgeon and the aggregated hospital level:

http://www.health.ny.gov/statistics/diseases/cardiovascular/

AFAIK, NY is the only state that does this. Looking at the report gives you a good idea of how complicated doing any kind of indexing of performance is...for example, there is an attempt to calculate expected mortality rate, as a high mortality rate for a surgeon may indicate that that surgeon, being the top of his or her field, may deal with the most critical and difficult cases.

Here is a link to the medical journal article discussed in the newspaper column submitted here:

"Surgical Skill and Complication Rates after Bariatric Surgery"

John D. Birkmeyer, M.D., Jonathan F. Finks, M.D., Amanda O'Reilly, R.N., M.S., Mary Oerline, M.S., Arthur M. Carlin, M.D., Andre R. Nunn, M.D., Justin Dimick, M.D., M.P.H., Mousumi Banerjee, Ph.D., and Nancy J.O. Birkmeyer, Ph.D. for the Michigan Bariatric Surgery Collaborative

N Engl J Med 2013; 369:1434-1442 October 10, 2013 DOI: 10.1056/NEJMsa1300625

http://www.nejm.org/doi/full/10.1056/NEJMsa1300625

"Background

"Clinical outcomes after many complex surgical procedures vary widely across hospitals and surgeons. Although it has been assumed that the proficiency of the operating surgeon is an important factor underlying such variation, empirical data are lacking on the relationships between technical skill and postoperative outcomes."

My mother was a surgical nurse at my state's best teaching hospital. She once was able to line up the "all star team" when one of my near relatives needed surgery. When the best surgeon at the hospital operates with the best surgical resident at the hospital, assisted by the best anesthesiologist and the best team of surgical nurses (my mom, of course, did NOT join the surgery team, as it is just too tense to operate on a near relative) work together, the surgery outcomes are fine. Informal referrals like this work for patients who know someone who knows who the good surgeons are. For the long-term development of better patient outcomes, it will be important to have a data-gathering system that turns informal impressions of who does good work from anecdotes into carefully measured data that are shared with hospital administrators and residency program teachers and insurers and others who have power to nudge poor-performing surgeons either to improve their skills or change their areas of practice.

I had this surgery, a roux-en-y, so as you can imagine I found the videos very interesting! (Wonder if somebody recorded my surgery?)

Before I had the operation I did a lot of research on the internet, but it was difficult to pin down particular surgeons. As the article points out, you can find stats on hospitals, so I went with that. In addition, surgeons can be certified by various boards, so I also used that. Then you can look at complication rates by surgical center -- very important. Finally, you can look at the size of the practice and how many patients they process per year. More is better. Surgeons who do a lot of the same thing for many years with lower complications than average are probably working towards the top of their game.

But that was hours and hours of research, and I imagine it all boils down to exactly what this article brings out: skill in the operating theater. (From my experience I think a strong secondary trait is the ability to carefully pre-screen future patients)

So, how do surgeons train? The commentary in the videos reminds me of martial arts training. It takes years to become fluent in most martial arts and the requirements are quite similar - a steady hand, economy of motion, spatial awareness, etc. But a good art has a curriculum where those skills can be trained independently of situations where you might injure yourself or others. Another possibility is - perhaps some people just don't have the knack for it and won't develop it. Is anything done to weed them out at the start of their training?

My girlfriend and sister are both going through surgical training in the US. Here's what I gather:

1. 4 year medical school during which one learns about the panoply of human medicine. You come out of med school with an MD.

2. 4-5+ year general surgery residency (depends on program and optional steps one might take such as research years) during which one learns general surgical procedures through observation, assistance, and practice in the operating room under the supervision of experienced attending surgeons. Generally, residents will rotate through different surgical departments in order to gain broad exposure to the various fields and techniques. Residents take a yearly standardized exam called the ABSITE and are also evaluated by the attending surgeons they have worked with. More senior residents are given more responsibility in patient care and will be given more opportunity to assist and perform in the OR.

3. After residency, one can further specialize by taking on a fellowship (2-3 years, but could be more depending on specialty). During fellowship, one focuses even further on a specific field. Fellowship is an additional layer of practical hands-on training.

4. Practice - after fellowship, one would go on the join a surgical group or create a private practice to perform surgery commercially.

Throughout all of this is a series of examinations that are given which, in total, will allow a surgeon to practice unsupervised on patients.

Edit: to point out where the MD is earned and vague mention of licensure

Some Canadian dental schools require passing a dexterity test. This is not a requirement in the u.s.

My brother is in dental school in the USA. A dexterity test was part of the admissions process.

Why hasn't it occurred to anyone before now that surely all surgeons are not created equal? As an engineer, I can't legally share examples of my work with potential employers because it's owned by the company I work for. Does anyone know if this applies to surgeons too? Can potential employers of surgeons request to see these videos to assess their performance?

Most people have the belief that since you have an MD you are near god like. Few realize the large number of bad/average doctors & surgeons are out there. It would be great if there was published performance reviews for every time of doctor, including your primary care doc.

Practically everyone on HN programs, but few came from MIT, work at Google, or could write a paper that would make Carmack question his choices. Think of programmers you have met that aren't self motivated or continually improving their craft. It's exactly the same with doctors. Some just get their degree (which albeit is a lot of work and money) and never self improve. They can be swayed by drug companies who give them a nice dinner without really understanding what they are prescribing or why. They do the bare minimum to get their CME and select the easiest path to get there.

If you have something serious, that may or may not require surgery, it is in your best interest to seek out the absolute best you can afford. Look for institutions that get ranked high for those diseases/injuries which produce a lot of research on the subject.

You are a startup and looking to hire a programmer, which do you select:

* Guy who learned PHP 4 and has been writing everything in Dreamweaver since without changing

* Guy who learns Go/Scala etc and enjoys reading other's research,

* Guy who wrote Go/Scala etc

You are someone sick and need advice/treatment:

* MD who graduated from South Dakota State (no idea of they are bad/good, sorry SD) and practices in a town of 5,000 people

* MD who works at a ranked institution and is always looking for the best treatment after evaluating new research

* MD who invented the surgery

There are obviously many shades in between... but treat your MD like you would a programmer you want to hire. It's not easy to figure that stuff out of course, but do your best in researching them. If you can't find much stuff on the MD/hospital then be cautious/take another opinion.

The above is based on my wife's experience in 4 different ranked hospitals in major cities dealing with patients coming in from unranked hospitals who often was doing stuff out of date/not efficient or flat out got stuff wrong/harmed the patient.

PS: For your furry friends, this is much harder as vet clinics arent ranked and typically your only choice is a ranked teaching vet hospital. Quiz your vet on where they got their degree and what conferences they attended this past year (this won't mean much other than they are actively seeking out new methods/research) You increase your chances of getting a good vet vs old Dr Bob who still ties the horse to the fence post. Look for a Vet from UC Davis if your are in the bay area.

It doesn't work like that in the medical profession, MD's in general don't have the professional mobility of programmers, nowhere in the world, and surgeons have even less such mobility.

You can have very skilled guys at "provincial" hospitals, either because they are stuck there or because the small town hospital actually gives them special incentives to stay there. It's less likely, but the probabilities have a very odd distribution varying from country to country so it's nothing near as easy to reason about as programmers.

And at a university hospital, your life can end up in the hands of a very very smart surgeon-researcher with tens of published papers that may have revolutionized the entire field he works in and got him very highly ranked, but that super-surgeon-researcher might actually have quite poor surgical practical skills (think dexterity/speed) and although you may get a lower chance of complications in the end, you will mostly get it because of a better hospital with better post op care.

DON'T compare healthcare with programming, or even with other "normal" professions where you can easily rank professionals... everything is complicated and "opaque" and medical professionals make as much as possibles to prop each other and "leak" as little information as possible to the patients and general population.

Also, in surgery "a better surgeon" != "a smarter surgeon" != "a better educated surgeon". It's a complicated function of "natural talent", "capacity for performance under heavy workload", "motivation", experience, education and IQ (and unlike programming, here the last two factors tend to be least important imho, at least if they are above a certain threshold).

Probably from the same reason you cannot get statistics about your surgeon before surgery(only recently it's been possible in the u.k.[1]): doctors have huge political power , and they are using it to their advantage. Another reason , more legitimate is that surgery is a risky business by nature, and you have to give some protection to doctors who are taking the risks.

[1]http://www.telegraph.co.uk/health/healthnews/9935856/Heart-s...

Exactly (on both points).

I can't really cite anything for my remarks, as they are based on vaguely remembered news articles over many years, but my impression is that the medical profession - in the US at least - has resolutely opposed any reform that would give the public access to information on the performance of individual doctors, hospitals, clinics or other elements of the healthcare system. The most prominent examples have had to do with histories of complaints and lawsuits.

The argument for this anti-transparency stance - and for disallowing patients to have their choice of surgeons - has been that they all have to learn by practice, and if patients were allowed to choose they would largely reject the junior staff, and the latter would never get any practice.

There is also a culture of solidarity amongst doctors, where an incompetent may get reassigned eventually, but won't be "outed" to the public.

Plus, if everyone always wants only the best, then where do surgeons get practice to become the best?

In quite many cases I'd prefer a good doctor right now rather than trying to buy my way in some far away spot in best doctor's schedule.

And practice is supposed to be covered by med school and supervised residency, isn't it? The question mainly isn't about inexperienced surgeons, but about experienced surgeons who simply are poor at the actual surgical technique, and possibly won't ever get much better.

And choosing who gets which doctor based on, say, patient paying ability is actually better than the current semi-random choice - because it creates motivation for the worst surgeons to, well, try something else - if they're smart doctors, then they can be useful without cutting people up.

In America, from the people who cannot pay for the best.

Not sure if you meant that to be a criticism, but... exactly.

I look at it as a good thing since the alternative to letting quality seek compensation advantages is to accept a general malaise of mediocrity like we have in public education.

In America, people who cannot afford the latest $1.99 app created by some startup ending in -ly or .io are getting surgical care provided by some of the best surgeons in the nation. This is because the people who cannot pay end up at their local hospital, which for many people (in, e.g., Baltimore, Boston, New York, etc.) ends up being the nation's best hospitals, where they are treated by some of the nation's best surgeons supervising the best surgeons-in-training.

from article, paragraph 8:

"It has been clear for nearly 30 years that not all surgeons produce the same results. The reasons, however, have remained elusive."

Not being snarky, just seemed to be the subject of the article. I'd be interested to hear the answer to your other questions. I certainly hope surgeons could be judged on videos of previous performances.

Other than for research, surgeries are not routinely recorded.

All that matters is the success rate. Watching videos of individual surgeons is pointless to pretty much everyone except surgeons.

I'd love to have the stats on each doctor.

Stats are hard because of all the other factors involved. It would be quite likely for the very best surgeons to have lower survival rates than mediocre surgeons, simply because the mediocre surgeon would refer the objectively harder cases to the expert, which would give better chances for that particular patient while lowering his 'batting average'. And if you give direct financial motivation for doctors to avoid taking such patients, then it's a bad thing for the whole system.

Stats can help you weed out obvious outliers - i.e., the ones who should be kept away from patients; but it's not so simple to make them useful in actual prioritization.

Success rate alone may encourage doctors to form other kind of risk preference in general. The fundamental issue is huge information asymmetry with different conditions. I guess there is not a simple solution to this. However, different simple solutions from different levels may benefit both good doctors and patients. Doctors and their organizations has much bigger power than patients do. Information could be something powerful to make the situation better, but not even close to perfect.

It would be useful in part because another surgeon can assess how well he is doing before actually stacking up enough bad/good statistics to notice. It's also useful for training, just like watching yourself swing a golf club or a baseball bat or rereading your old code can help you, with the aid of others, see what you could improve on. Doing the wrong thing over and over in a vacuum doesn't make for much learning..

Interesting article. My wife is a surgeon and I'm interested to see what she has to say about it. One small anecdote - recently she had a colleague (another surgeon) ask her discreetly whether she would recommend another surgeon for an operation. Her colleague had known this other surgeon for many years, worked in the same department as them in fact, and still did not know if the other surgeon was good or not.

The best way to choose a surgeon is to get a personal recommendation from a doctor who is a personal friend. This will at least prevent you from getting the absolute bottom of the barrel.

See my other post in this thread about this, but I disagree. Even other surgeons often do not know how good another surgeon is - unless they have operated with them directly.

Many members of my family have been on the staffs or in the employ of various hospitals. From what I've heard, if one has an inside connection it would be very difficult to avoid learning of a particular surgeon's poor therapeutic outcomes. It's true that past results are not perfect indicators of future results, but unless you just transferred to this hospital, every nurse on every floor has a firm opinion of whether she'd let you operate on her family.

I guess it depends on who you ask. My wife (a surgeon) says that this is often not true for most of the staff. For example if you ask the OR/Floor nurses they will know if the surgeon is truly terrible, but in her experience (hearing from the nurses) she says that she is often surprised by who they think is a good surgeon. She does say however that the anesthesiologists working with the surgeon usually have a good idea.

I'm sure she's right, in that many nurses, administrators, and non-surgical physicians wouldn't have the nuanced understanding of surgical ability that she does. Just the same, I expect their rough sorting isn't too far off the mark. They certainly know more than the general public!

"Skill" as a catch-all term is a meaningless word bandied about by those trying to sound as though they know what they are talking about.

There are two sets of attributes, the first (commonly called "skill" but perhaps better called "technique") describes how well you can carry out the "motor" activities of your job. The other, and more important in critical circumstances, is "judgement".

To describe it by analogy with driving, in the beginning you concentrate on developing experience in operating the vehicle, eventually this becomes more or less automatic. People call this skill and they are wrong, it is a set of learnt reflexes. More important is judgement, knowing how to maintain road awareness, knowing when and how to overtake, when to speed up and when to slow down etc, etc. NB when people talk about the dangers of drinking and driving it is impaired judgement they are mostly talking about (although there is some impaired skill such as poor reaction times which stops them getting out of problems their degraded judgement has got them into).

Judgement comes with experience and follows skill acquisition. With surgeons you would hope they know how to make a cut (they have technique). What you really want is someone who knows where and what to cut (they have judgement).

And yes, all of this applies to programming and systems design.

The other, and more important in critical circumstances, is "judgement".

Whether I die because the surgeon was clumsy and nicked my aorta or because the surgeon exercised poor judgement by not counting sponges routinely so he left one that caused a fatal blood clot would be irrelevant to me because I would be dead.

All that matters are outcomes based upon an apples to apples comparison of similar patient contexts.

The quality of good judgment includes knowing whether or not how you move your hands hurts or helps your patient. A surgeon who operates with poor technique can fairly be described as a surgeon with poor judgment. (Disclosure: my basis of knowledge is that my mother was a surgical nurse at a teaching hospital whose surgeons invented many new surgical techniques over the course of her career. My mother was part of the team for some of the very first open-heart surgical procedures performed anywhere in the world. All of the best surgeons in that hospital had good judgment about whether or not to perform surgery, and what technique to use, or to invent, for a particular patient problem, but the teaching surgeons there also had flawless technique.)

I'd be interested in reading the original paper. It wasn't immediately obvious if the researchers had stratified surgeons by experience (either number of times procedure had been done or years in the field). I would imagine less experienced surgeons to still be developing their technique.

In the film "Manda Bala", there's a scene where a team of surgeons, led by the world expert in this particular procedure, insert a scrap of reshaped intercostal cartilage under a patient's scalp to reconstruct an amputated ear. The motion the team displays in that two second clip is a beautiful as any choreographed dance move.

Reminds me of this old Kids in the Hall skit:

Bad Doctor: http://www.youtube.com/watch?v=Pbjypn9JtKE

Anyone knows what operation are they doing in the video?

Looks like a gastric bypass.

Reply on news.ycombinator.com