IBM pitched Watson as a revolution in cancer care, but it’s nowhere close

IBM has some OK offerings but they've really screwed themselves with all this hyped up marketing around Watson. They've become somewhat of a joke within the real data science community and business executives are also increasingly becoming disillusioned when projects based on IBMs mystery black boxes fail to deliver.

The MD Anderson situation was a debacle of monumental proportions. Such claims on cancer care are quite unethical in my view given that it gave a lot of very sick people false hopes and diverted a lot of hospital attention and funding away from stuff that actually works.

> On a recent morning, the results for a 73-year-old lung cancer patient were underwhelming: Watson recommended a chemotherapy regimen the oncologists had already flagged.

> “It’s fine,” Dr. Sujal Shah, a medical oncologist, said of Watson’s treatment suggestion while discussing the case with colleagues.

> He said later that the background information Watson provided, including medical journal articles, was helpful, giving him more confidence that using a specific chemotherapy was a sound idea. But the system did not directly help him make that decision, nor did it tell him anything he didn’t already know.

I don't know, that doesn't sound bad at all. Perhaps it's just a case of overhyping. But if a computer can do recommend the same treatment as an expensive oncologist with years of training that's actually pretty good!

On the other hand it is quite baffling why doesn't FDA require a clinical trial or any independent review of its capabilities before allowing its use in clinic. It's a bit like Boeing certifying its own planes, I guess.

> Hospitals pay a per-patient fee for Watson for Oncology and other products enabled by the supercomputer. The amount depends on the number of products a hospital buys, and ranges between $200 and $1,000 per patient.

Sigh. Let's remember that in medicine (a severely resource-constrained system), costs operate in a zero-sum environment. $200-$1,000 spent on Watson is $200-$1,000 not spent on some other treatment or human labor. And, in our current environment, specialist doctors will always have to re-do the diagnosis suggested by Watson, because guess who is held legally responsible, in the case of a malpractice suit? Not IBM, but the doctor co-signing the "recommendation".

TLDR: Watson has been trained for 6 years to build a knowledge base about 7 types of cancer; therapy recommendations are entered by experts from one top US hospital.

Looks to me like a giant pattern recognition system. The best assessment I found in the article:

> He said later that the background information Watson provided, including medical journal articles, was helpful, giving him more confidence that using a specific chemotherapy was a sound idea. But the system did not directly help him make that decision, nor did it tell him anything he didn’t already know.

Cynical take, having worked in organisations that have bought into IBM: IBM is not into curing cancer, it's into selling, and in particular up-selling at enormous scales (government and big business). The primary goal of the Watson project has not been to cure cancer, or do anything in particular, but to get existing customers to buy into extremely expensive contracts that include 'Watson' in some way. All the hype, which it is now becoming apparent was artificially expanded to some extent, is just a means to this end. As with most (all?) of their other products, it'll be oversold, underdeliver, but lock organisations into even heftier contractual chains.

Its kind of sad to see what has become of IBM. It is now just another consulting shop that wants to plant a slew of 3-400/hr consultants to run projects. The big consulting companies best trick was to convince those in the enterprise that hiring a boutique consulting shop is "risky" and that a big 4/5 consulting company is a "safe" option even though it is orders of magnitude more expensive and not necessarily better, and in some cases worse when it comes to specialized systems.

Source: years of working in Energy/Commodity trading systems market and seeing several botched big 4 implementations that vendor/boutique shops can do at 1/10th the price.

I was under the impression "Watson" was just a brand-face for IBM consulting services.

You're not buying thousands consulting hours and some big data services... you're buying Watson

As per every thread Watson is brought up in, it's mainly a catch-all term for its consulting division. IBM's consultants aren't very good, and the good ones move on to brighter futures.

This is why Watson is a disappointment and every Watson project ends in cost overruns and failure.

It's interesting that Watson appears to rely so much on processing text, which runs into so many difficulties. It would be helpful if (for a given disease) every patient's medical records were tracked as structured data. Machine learning would then be able to find patterns of diagnosis much more easily.

(I guess companies like Flatiron Health are working in that direction.)

When results from "AI" don't match the hype nor justify the cost and hassle... that's how AI Winters begin.

There's a bigger issue at hand here. AI generally has not made huge inroads into disease treatment. I think this is because we haven't figured out a good way to describe the diseases. Where it does work, we have a good way to classify the data (i.e. a set of training images or clear metrics to feed it). As Google/Verily learned the hard way, its not easy to fit digital models to organic beings.

In the area of cancer treatment, my guess is that the reason specialists are highly paid/trained is because the analysis and judgement required is highly manual and varied. Surgical intervention, prescription medications, targeted radiation treatment machines, patient psychology and counseling. Other than maybe guessing the prescription based on symptom list, what could the AI profitably help with?

> On a recent morning, the results for a 73-year-old lung cancer patient were underwhelming: Watson recommended a chemotherapy regimen the oncologists had already flagged.

> He said later that the background information Watson provided, including medical journal articles, was helpful, giving him more confidence that using a specific chemotherapy was a sound idea. But the system did not directly help him make that decision, nor did it tell him anything he didn’t already know.

This type of thinking is reason true ML adoption is still sparse. Analytics should be viewed as a supplement to decision making, not a replacement.

Manipulative marketing tactics by IBM and others are largely to blame. I'm confident that over time, as the general population becomes better educated on the topic, the perspective will shift.

> “It’s been a struggle to update, I’ll be honest,” said Dr. Mark Kris, Memorial Sloan Kettering’s lead Watson trainer. He noted that treatment guidelines for every metastatic lung cancer patient worldwide recently changed in the course of one week after a research presentation at a cancer conference. “Changing the system of cognitive computing doesn’t turn around on a dime like that,” he said. “You have to put in the literature, you have to put in cases.”

Wow the way they describe Watson is just like a massive book of documentation, with some NLP functionality layered on top to interface with the doctors. Keeping the database up to date must be a huge undertaking. Also this:

> "Given the same clinical scenario, doctors can — and often do — disagree about the best course of action, whether to recommend surgery or chemotherapy, or another treatment. Those discrepancies are especially wide for second- and third-line treatments given after an initial therapy fails, where evidence of benefits is slimmer and consensus more elusive."

Having an expert doctors opinion hard coded into the machine sounds good, but it won't be as good as a doctor in the room who can pick up on all sorts of other cues that I'm not sure how'd you even type into a computer. Things like breathing patterns, reaction to tactile stimuli, fuzzy logic ('that hurts, that hurts more'), facial expressions and so on.

Seems like this project has a long way to go, to get to the standard the IBM marketing team are pretending it's already at.

I guess one day IBM will sue Scott Adams for plagiarism.

IBM is nowhere close, but is there anyone closer?

Who uses Watson besides IBM?

IBM seems to have painted themselves into a bit of a corner by calling a bunch of largely unrelated research projects by the name Watson.

I don't like the Watson logo. It's too busy. If you shrink it or squint when you look at it you just see a blob.