Jefferies gives IBM Watson a Wall Street reality check
techcrunch.com
techcrunch.com
I understand that this is what they want. They want to drive executives' interest in the product, but I believe they do so at the expense of their goodwill with the tech community.
Am I the only one who cringes when these ads air?
Edit: "magic beans" is harsh and it isn't that I don't think their tools are good. My point is that they put you in a position where it seems very unlikely to meet expectations.
Their tools are trash for the cost they want. The majority of their offerings are overengineered and prone to failure in a production setting.
They are pros at selling to non-technical management, all the while IT employees get stuck dealing with the aftermath.
We ended up getting stuck with the IBM version of work item management, called Jazz Team Concert. From the "source control" to the actual setup and configuration, the product was a nightmare. The worst part was when we looked at license renewal. They wanted three quarters of a million dollars for something that was essentially dog shit wrapped in cat shit.
In case you thought I might be on the fence with IBM, I'll add one final clarifying point:
Fuck IBM.
Don't be so harsh on all of IBM, its like hundreds of companies under one banner. Not all of them have the power to influence other products and often times end up stuck with it and cannot move on to other tools (i.e git).
More often than not, customers have higher priority on bugs than other teams dog-fooding the product. Its insane.
I think you mean don't be so harsh on the people on the ground at IBM. If the issue is IBM's internal structure is fractured and disorganized then I think it's perfectly fair to blame the company.
The do this so different divisions can lean on each other and benefit from the IBM brand name. If they want the benefits of this shared branding then the deserve the downsides from it too.
I've noticed many companies recently thinking they can have it both ways (samsung and sony come to mind). Live by the brand, die by the brand!
My own anecdote -- we had a need for a GIS/charting/mapping system. IBM responded to the RFP as did many others.
IBM proposed an entire office automation system with a minor charting system hanging off the side somewhere. Most of the components were unfamiliar to everyone. Though maybe there was Wordperfect or Lotus Notes in there. I remember meetings with IBM always had whole bunch of shiny suits whereas more focussed proposals just had one or two people visit.
After one or two rounds of pointless meetings our fairly technical director told them to go fuck off and never come back.
The worst thing though that was despite paying an inordinate amount of money to license this software each year and receive support for it they would not fix a bug we found in their mapping modules.
Specifically it was their use of float for X and Y coordinates in the IlvPoint object that you pretty much had to use to make a map work with their tech. In certain parts of the world there is an extra digit on the Y coordinate (projections) and the float they insisted on using couldn't store the whole value, meaning they would just round it up or down causing inconsistencies between our representation of a client's dataand the client's representation. That was an absolute no-go for us and the workarounds IBM suggested to us were absurd and bordering on insulting honestly.
We had a lot of trouble with our affected clients because of this and all we could tell them was that it was an IBM issue.
The absolutely refused to fix their product and eventually sold it to Rogue Wave, washing their hands of it. By that point though we had already decided on a few alternatives lest we start to lose customers (who were essentially massive utility companies from around the globe).
Fuck IBM.
We are starting a project and even with FTE's within the company, "consultant" has convinced management to hire IBM resources because of cost. The high sounding reason within the presentation twisted it to - FTE's are not expert in this tech so they won't be able to do an efficient job.
But it is always that the end resources from IBM are even more incompetent. They end up delivering a product which is terrible. Even the final handovers and KTs are done terribly.
In the end, management has no option but to continue hiring IBM (or any service business) to support the shoddily done project.
This doesn't mean all implementations from a services business is terrible. It all comes down to cost and ability to spend. If you pay peanuts, you get monkeys - this is aptly true.
To answer your last question - Yes there are different entities within IBM. There is GBS which takes up most of the consulting and up-selling IBM products and services. Development for example Watson is done by IBM Labs. A list can be found here: http://www-07.ibm.com/in/careers/businessunits.html
At an old employer who I'm no longer with, I asked why there's a part of the application in Java (the rest has always been in .net). They told me that an army of IBM consultants made that decision and now there's a part of the code in Java and it sticks out like a sore thumb.
But doesn't the word get around ? does nobody in most enterprizes think long-term when investing ?
IBM is optimized in enterprise (including government) sales. Product is not quite irrelevant, but it's a specialized enough market that often mediocre substantive quality does quite well, because the only competition that matters are the others that are optimized for that market.
Wasn't there a saying that no one got fired for buying IBM.
I've seen it many times.
Misinterpretation is a recursive problem. The best description - by far - of this process is "How a plan becomes policy" (aka, "How Shit Happens").
As the staff back away slowly from the creep, that just makes more space for IBM, and they sell progressively more service hours. It's pretty much their MO.
Broadly, yes, IBM is run as various different divisions, and the divisions start very high up the organisation, only a couple of levels from the top. They rarely interact in my experience (as a software guy there for some years in the past)
I think it expected to be installed in /usr/local and company procedures demanded that it was installed in /opt (or the other way around).
For this it would consequently misplace some permissions on some of the files causing a later stage in the installer to fail and bork the installation.
I bumped into her last year at Ansible Fest London and she was still angry about it 17 years later!
And that's what Oracle does to your soul :)
Edit: that N class and Oracle NEVER got deployed in the end. It just sat there unused, licenses weren't renewed, it got sheepishly powered down after a few months to save electricity and it got chucked in a skip after 5 years. About £200k of expenditure down the pan.
I guess you could easily lost even more money. Good thing they stopped it at £200k.
The whole platform got replaced with SQL Server on a clone PC server in the end which cost less than £10k. No fan of Microsoft but there was no competition. And the tooling was far better!
Legal guys said it wasn't worth suing them because we'd have to pay up then or lose the support contract.
> However, I think you'll see this shift towards a developer-focused mindset happening more-and-more within IBM over the next few years.
means? In a discussion about not trusting IBM's marketing, you can't simply say "I hear you and it will get better, trust me."
I don't work in the same role so I only know things from a 10,000 ft view. But a lot of orgs that IBM works with have inane bureaucratic structures with their in house devs pretty much at the bottom of the food chain, or in an "IT department". All decisions on using tech tools are made by "higher ups" a lot of whom have 0 technical background. We're trying to change this culture to one where devs get to make those decisions AFAIK.
Disclosure: IBM dev, but not a Developer Advocate.
IBM was always unique, could sell you sfw, hdw, hosting, resources, and finance the whole thing. Some business as leaders never really adapt to change because management is stupid (blockbuster, radio shack, blackberry, etc.). IBM had so many chances to compete with Amazon and they failed. Watson is no more than a deflection in the marketplace that masks the real problems from within.
If you can't provide any useful comments, then nothing. What they're saying contributes absolutely nothing to the conversation. Maybe say you'll stop selling non-existent features to executives for starters? I don't appreciate HN becoming an advertising ground to name-drop IBM without any real talking points, please provide some sort of real discussion.
> We're trying to change this culture to one where devs get to make those decisions AFAIK.
Ok, but again how? Or I guess in other words, I don't believe you.
The tradition was still going at that point.
https://www.itnews.com.au/news/queenslands-ibm-ban-lives-on-...
http://www.cbc.ca/news/canada/ottawa/phoenix-pay-system-psac...
What do you mean by 'over engineered' in this context? The usual way I've seen that term used would imply 'reliable but more expensive than necessary' rather than 'unreliable'.
Think J2EE and their xml config files.
Or using hadoop and kubernetes when a sqllite db on digital ocean will do
I have seen middle management say very clearly that no one was ever fired for choosing IBM.
Is that just a notable exception or are there better alternatives out there these days for hosted CouchDB?
Not saying it's not good (I've never used it), but IBM can't really get any credit for this. They just happen to have bought a decent product and now they sell it.
I like Cloudant .. it is not too bad. In a recent project (after I left IBM), I needed to get a data tier up quickly and I actually used it for the initial POC. And yes, it was an acquisition by IBM. I was a bit worried that bluewashing would have made Cloudant crap but looks like they have managed to avoid it. The caveat I'd suggest is look closely at the pricing model. If you want QoS and scale, be careful of the pricing model. For small projects and projects that need fast completion (what project today doesn't?), it is actually very nice. You asked for caveats .. I think some of the language bindings were a bit out of date .. but since it uses REST calls, it wasn't too hard to bypass.
It's really easy (largely thanks to Cloudant) to host your own Couch instance but it's also really affordable to let them do it.
[1] E.g. they're pretty open about renaming existing things to fit under the Watson brand: https://www.ibm.com/blogs/bluemix/2016/10/predictive-analyti...
That is a hard sell because requires a truckload of configuration and internal expertise to get a working product out of it, and it ever only solves only some very narrow ranges of solutions.
In typical IBM style, since they can't have a multimillion research division going in red as that would get a bunch of management in trouble, they moved more and more product under the watson name to pad up the division growth and sales numbers.
That's the point where it is now after hoarding anything internal with value. You can see more or less the same with their cloud offer.
More seriously, do you think Microsoft would have been more honest if they'd called Windows "Microsoft Graphical Multitasking Operating System"? Or if BMW had gone with "BMW Better Performance And A Cooler Badge But It'll Cost You" instead of "BMW M"?
Brand names exist for a reason.
Hmm, I'm having a hard time figuring out the N. Ideas?
IBM's sales pitch was that 1) this particular piece of software was "central to IBM's [market niche] strategy", and 2) if we didn't go with IBM's product, we would miss out on all the interoperability with IBM's Watson that the future is going to bring. The tone veered toward fire-and-brimstone. I wasn't sold, I was put off. They've demoed other things in the past that I've had my socks knocked off by, but they're nothing that's the sort of "indistinguishable from magic" that they're promising in all the marketing hype.
The thing is, IBM delivers tremendous value, and they have some fantastic products. However, stunts like that leave people like me with a very sour taste in my mouth. If they'd just focus on delivering a better product, as quantified by hard metrics, (which they could easily afford to do), they'd keep making out like bandits for a very long time. There's tremendous good-will toward them in the market: bad experience aside, I'm still favorably disposed toward them.
We are closing on the IBM bandwagon, but I want to show that there is alternatives.
IBM sells an image, not a working product, and they deserve some ridicule for it.
I have worked with Watson and a decently large IBM consulting team, and 'magic beans' is a diplomatic way of putting it. 'Obfuscatory horse shit' is more accurate.
The good ones use it as a smoke screen to cover for doing things that need to be done but that the C-level isn't necessarily going to like, the bad ones use it as a cover for incompetence.
The fact that you're more concerned about the truth is why you'll never be on the take like them. :-)
We were too stupid to catch on that everybody else is lying.
It's astonishing how much IBM has spent to achieve so little. Per the Jefferies report, IBM spent $15 billion in Cognitive from 2010 to 2015, not including an additional $5 billion in data acquisitions. Even if Cognitive weren't a marketing buzzword invented by IBM to describe if-then statements, that would still be deeply embarrassing. (You could probably double the number of startups funded in Silicon Valley in a year with $20 billion dollars, and I'll bet you'd get much higher returns.)
When you look at the main open-source AI frameworks, none of them were created by IBM, because IBM was incapable of building them. There are individual startups that have done more in AI, using far fewer resources.
And the real shame is that IBM is convincing rubes to buy their cognitive trash, and the rubes are doing it. And when the rubes' bosses realize that the rubes blew the AI budget and the "AI" delivered nothing, those rubes are getting canned. You can get fired for buying IBM. I've seen it.
There's a Madoff-like quality to IBM's Watson sales strategy. The more they fail, the wider they circle to depend on dumber and dumber money from abroad. They're going after rural Scandinavia. They're selling to 3rd-tier cities in China. Just imagining targeting a buyer with the sophistication of the mayor of Mobile, Alabama. That's where IBM Watson is selling strong.
IBM has been poisoning the well in the AI industry for years, teaching customers to be gun-shy and scared of an important technology, just because they blew their money on a dog to begin with. I don't believe an AI winter will come, but if it did, IBM would be the main culprit.
Even if Watson's many mysterious technologies worked, they would still have a lame business model, because most of AI, and the best of AI tooling, is open source. IBM's trying to sell a secret sauce, even as it declines to participate in competitions that would should how Watson stacks up against other tech. Oren Etzioni has been calling them out for years. [1,2,3]
Anyone wondering how IBM came to this sorry pass should read Robert Cringely's "The Decline and Fall of IBM." It's a company run by a clueless management class, who probably believe the smoke they're blowing up everyone else's ass. That's the scary part: The C-suite at IBM actually bought that hype about Watson, until they started to see the numbers, and the numbers sucked. That's why they don't break them out in the quarterly earnings reports. IBM is not signing up the Watson partners it thought it would. [0]
Watson makes claims to solve everything. But when you dig in with the sales people, the only success story they cite is Memorial Sloan Kettering (MSK). It's suspicious, considering how many other organizations they must have worked with. H&R block is bragging about Watson technology, too, but that's just because they think the marketing gimmick will work for them as well as it's worked for IBM.
The real problem in a situation like this is that very few people have an incentive to get the truth out. IBM doesn't want its stock price to dive. And all the customers are mortified that they failed and wasted their money. And they probably had to sign some non-disparagement clause to boot.
[0] http://www.cringely.com/2016/09/07/carrie-underwoods-america...
[1,2,3] https://twitter.com/etzioni/status/654486809825570817
Do you mean UIMA, GATE? What else? These are frameworks comparable with Watson (at least the original thing, not the PR fuzz).
It might be easy to understand what Charlie really meant when you are already familiar with the project, but the non-technical viewer will walk away thinking of Watson as a single, coherent entity sitting in an IBM vault somewhere. Conflating ML-based expert systems with anthropomorphic AI is farcical, supporting the accusations that IBM is overselling its product. It is also irresponsible, because it fuels the idea in the popular mind that machine learning is creating independent intelligences rather than mere tools.
I agree the language is problematic for non-technical viewers, but at the same time these are difficult concepts to explain to non-technical people, and explain the details of what it can't do is obviously much less compelling than focusing on what it can do.
But that's not the problem, is it? It's not that IBM is trying to explain something difficult. Rather, if IBM was trying to be honest, the box that won Jeopardy would be called Watson, and the box they're now trying to sell everyone and their dog would be called "Whitney, Watson's big sister" or whatever. At the very least, "Watson the third".
By calling it the exact same name, you are trying to convince us it's the exact same thing just with more training. Which it's not.
Others mentioned poisoning the well; round here the expression is "pissing in the well", which I think fits much better.
Theirs at least one post in this thread that goes into detail on the limitations of Watson Health (it mentioned oversold capabilities that led to a medical institution wasting ~$60 million dollars, but you would be right to point out that I have no direct examples from memory on why the 60 minutes coverage itself was 'absurd'. If anything its the effectiveness of Watson marketing (and other companies marketing of AI) that gets to me and how it is even becoming pervasive in cultural icons like 60 minutes.
I'd want to see:
0. A baseline measurement.
1. Comparison to the nearest compatible.
2. Comparison against a simple algorithm/expert system.
Overall that probably helps the brand with some groups, and harms it with others (scientists, engineers) so you may have more work to convince them, though the puff pieces might help sell it at the exec level.
I'm reminded of the horrors of early-2000s enterprise Java.
could be worse, could be magic beans from IBM Blockchain.
Thank you for sharing your view. If you'd like to talk more about how we can better message our offerings, please feel free to reach out. I'd love to learn more about the kinds of problems you are trying to solve.
Edit: yes, I know, this is a bland response to the parent. But change needs feedback.
Edit 2: thanks matt4077
"Change", as used here, is a noun based on the verb "to change". If your sentence calls for a verb, you can just go back to the original "to change".
Something isn't essential "toward" something else. It is essential "for" something else.
"is essential" is unnecessarily passive. Use active verbs.
"getting feedback" is slightly wrong, because what you need is the feedback, not the act of getting it.
"At the same time" is used here because it sounds smarter than "but". Don't do that.
Try this: "Change needs feedback".
Not all brevity is the soul of wit.
How about "for change to happen, we need your feedback"? I think that conveys the message and request accurately and succinctly.
> I'd love to learn more about the kinds of problems you are trying to solve.
This doesn't sound genuine. This type of delivery almost never sounds genuine. You wouldn't "love" to learn about those problems, and even if you would, virtually no one will believe you because the same corporate speak delivery is used by people who have no commitment to action.
I know you probably just said this unconsciously, but when you use phrases like "I'd love to learn more about the kinds of problems you are trying to solve", it reeks of a comment written and vetted by a team, including brand/social/marketing/PR/whatever the team is in your org that controls employee disclosures.
I've been on the inside of orgs and watched this PR spin sausage cooked up. Someone wants to respond to a controversy brewing on reddit, oh wait, better run it by the the #social channel and make sure everyone agrees this is our best foot forward...wonderful, now we've essentially guaranteed that people don't feel empathized with whatsoever.
Just solicit feedback like a human being. It's okay to "disclose" that you work at IBM (although: jeez, what a corporate-y way to go about it), but speak to people in a conversational tone that doesn't sound like a customer support Twitter account.
I'm sure there are companies, and probably parts of IBM, that have to run their communication by legal, but I'm not part of that. It's just how I talk.
I can't guarantee change on any particular issue, and it would ludicrous for me to act as if I speak for the nearly 400,000 IBM employees, so I temper what I say out of a desire to be honest.
My purpose in this thread is to find people who want to share their experience and I assume, perhaps naively, that is part of the purpose of commenting online.
No matter, the discussion in this thread has already been very valuable and I do appreciate everyone's comments, even throwaways.
Talking about market strategy, no one knows what IBM does anymore other than garner support contracts and land body-shop jobs. Everyone knows Watson because it's won Jeopardy and a few other games. But that's not because Watson is ground-breaking or anything, but because IBM hasn't done anything else new in 5-10 years. So Watson seems less like a crowning achievement, and more like debris from the Titanic. The company's treading water, and barely, at that.
Sales, however, swears that Watson can do everything from getting the best tax return to curing hair-loss and replacing 80% of the work force. But anyone who's not a fool knows that current AI still has to be tailored towards an industry to be useful. And that's still going to take a huge amount T&M investment. Which you can see from the well-known failed (and expensive) Watson projects.
Usually the only companies who fall for it are the ones where management is less about technical skills and more about kickbacks and favor trading. Or national/international-level contracts where the most important thing required is someone to blame or sue when things inevitably go wrong. Everyone else either understands the technology is more complicated than the presentation, or they have enough insight into IBM to discount the powerpoints.
I mean, IBM still isn't as bad as Oracle. But, if I had to pick a metaphor for IBM's reputation... I'd go with "IBM is as much an innovator as Steven Seagal is an action star."
Dishonesty is ESSENTIAL to IBM Watson marketing. Specifically, IBM has taken a whole lot of different technologies and rebranded them as "Watson", so as to pretend that the Jeopardy-playing system is actually a big practical deal.
IBM's reputation over the decades earns it the benefit of the doubt on bold technical initiatives ... for a while. But when IBM doesn't deliver, the benefit of the doubt runs out. And that's happened now with Watson.
IBM's Watson, for me, has been so much marketing fluff that it's in the Oracle class of "only bought by people who don't know better". A profitable business line, but not a problem solving business line.
IBM sends out a team for each of the engagements to work with the business unit leaders to educate them on the capabilities of Watson and how to train Watson. It was all pie-in-the-sky stuff. Since we're an engineering-oriented company our business executives tend to have a strong engineering background (we tend to promote from within) and the whole thing smelled fishy to them, sounded too good to be true. When they asked us IT guys about it we presented the problems mentioned in the first paragraph. They concurred.
So we've never taken the bait. That's the problem you're having with Watson.
I don't want to say that's your fault. I get it, I have a job too.
---
Honestly I wonder what IBM does these days.
Watson is a professional service Machine Learning thing, which can't be profitable. Onsite engineers enguaging one customer rarely is.
Power8/9 has flopped. Nvidia only appears to be on board to kick sand in Intel's eyes.
Then there's... Mainframes?
I know IBM has a cloud now, but honestly it's prices are far high then AWS or Azure.
The problem isn't the message. Everyone already drank the kool-aid. A lot of the problem is just the shoddy product, but really it boils down to the culture.
Put out comprehensive documentation _on the indexable Internet_ and not in PDFs, because for christsake if I have to open another Red Book PDF I think I might kill myself.
Make things that are open source -- and I don't mean Open Source™. Let me learn the platform. Let me hack away at an i Series or whatever on my own time.
Everything is so buried behind licensing and paywalls and PDFs and just general bullshit that I can _never_ invest in _anything_ IBM on a personal level. The vendor lock-in is so heavy that I resist even wasting time learning IBM products because in the back of my mind I'm always thinking "I'll never be able to take this with me to a new job."
The garden walls are too high.
-------
> Put out comprehensive documentation _on the indexable Internet_ and not in PDFs
But IBM does do exactly this--and more than a lot of companies!
For example, IBM Knowledge Center [1] houses the documentation for over 3000 IBM products and services, and offers up more than 13M english pages--without a sign-in even, and we translate our official content into over 30 languages.
IBM KC is very well indexed by Google, too (we take pains to make that happen).
We also offer custom search and links to more freely available IBM documentation. Our Cloud development docs--Bluemix docs, are also quite public, though some, it's true, are behind a cloud product sign-in.
Redbooks, too, are available in HTML, not just PDF. They can be found on the web and are indexed by Google too, as are articles from developerWorks and Technotes from Support.
So I really don't understand the complaint about our documentation. Almost none of it is behind a sign-in, let alone a for pay firewall, and almost all of it is freely available to anyone, not just our customers.
Hope this helps!
-------
[1] ibm.com/support/knowledgecenter
i has always been a garden wall and I don't think it's realistic to expect that platform to open up at this stage. That's why Linux on Power exists - tell your boss he/she would have better luck retaining/attracting people if they'd spin up some Linux lpars.
The older IT generations like i just the way it is.
My comment was literally the response from the Eliza chatbot when fed your comment. It was an admittedly oblique attempt at humour. My apologies.
Yes, there are opportunities. No, I do not believe IBM could solve mine, because of the overpromise, underdeliver. I would only speak with a sales team, or a PM, if my boss ordered me to.
You know who got this right? Fucking Dominos, of all companies. They took a hard look at themselves and said, yeah, we've been making a shit product. We don't want to do that any more. So here's what we're doing.
If you want to get a community like this to respond, you need to put the onus on yourself first. Don't go "hey, explain to us what's wrong." You need to figure that out on your own! At least pitch some possibilities. Show some evidence that you've put in the work before crowdsourcing your five year plan. Otherwise you just look lazy and like you want the community to fix your business. And that isn't their job.
Not exactly a display of maturity from this community.
> You need to figure that out on your own
Even IF an employee has a pretty clear idea themselves what needs to be improved, having a citable CUSTOMER opinion to that effect is going to vastly improve the chances of management listening.
Well, can't say I'm surprised. I used to work on that project a few years ago, basically the idea was that Watson would look at a patient's medical record, figure out what medications they're on, what symptoms they had, etc. and cross-reference all that with the medical knowledge it had ingested from vast amounts of medical literature. In theory, Watson could figure out what medications the patient should or should not be using, a proper course of treatment, etc.
There were two major problems:
First, it turns out your medical record is mostly written in narrative form, i.e., "John Smith is a 45 year old male...", "Patient is taking X mg of Y twice daily", "Patient was administered X ml of Y on 3/1/2016", etc. In other words, there's basically no structured data, so just figuring out the patient's stats, vitals, medications, and treatment dosages was an adventure in NLP. All that stuff was written in sentence form, and of course how things were written depended on who wrote it in the first place. It was really, really hard to make sure Watson actually had correct information about the patient in the first place.
Second, all that medical literature that was being ingested? Regular old, don't-know-anything-about-medicine programmers were the ones writing the rules the manipulating the data extracted via NLP. Well guess what, if you're not a domain expert you're bound to get things wrong.
Put those two things together and we would frequently get recommendations that were wildly incorrect, but that's to be expected when you get garbage input being fed into algorithms written by people who aren't domain experts.
That makes the MD Anderson case more interesting as it mostly means that MD Anderson spent $60 million to help IBM figure out the importance of that.
It begins to seem obvious how IBM stays so impervious despite being so sluggish. If your marketing and sales teams can subsidize the engineering and design side so heavily by wooing the executives that actually sign the contracts, you can spin it as just "where the technology was at that time".
Selling on hype gets you in the door and establishes you as player, and eventually the tech will catch up. Being 1-2 years behind blurs when you're signing 5 year enterprise-scale deals that took a year to organize and negotiate.
And if your products really are a dead end, you've got enough of a war chest to just aquihire an entire branch of your company, like IBM did buying Softlayer to just become IBM Cloud Services.
Moreover, the acquirer must be willing to pay salaries competitive with Google, Facebook, and the likes to keep the team. From what I heard, IBM is not willing to do that. If anyone has information to the contrary, please feel free to correct me.
Every hospital / doctor wants to talk about the wealth of medical information they are sitting on... as you said most notes are just transcribed ad lib, or input into rigid EMR systems, there is no middle ground.
Also the language spoken by healthcare software is HL7 which is another nightmare in itself.
This applies to all ML work, why is Watson exempt from it?
Conversely if this the canonical failure case, why's everyone so harsh on Watson?
Because they keep claiming Watson can handle this kind of thing, they charged $60M for it, and failed.
That was $60M which could be spent on actual, real cancer research, or models which actually work.
For example, MSKCC[1] currently has a Kaggle competition[2] to do a roughly analogous task. The prize for that is $15,000, and they'll get something more useful for that than MD Anderson did for $60,000,000. Even taking into account it is probably costing MSKCC double the prize to have Kaggle run the competition it seems like IBM was ripping everyone off.
[1] http://mskcc.org
[2] https://www.kaggle.com/c/msk-redefining-cancer-treatment
I mean, if Delta ordered a super-advanced new fancy 787 as a tech demo, and Boeing came back with "sorry, our experts were unable to make anything work, so we're going to take all your money and not give you anything back", Delta would rightly tell Boeing to fuck right off and demand that they refund the money.
If you buy a service, or a customized product, then you pay for the time and effort it takes to build / customize that product. Returning that time and effort would be difficult :)
A lot of IT products don't stand alone in their own stack, and the wild variance in integration points from one site to the next drives a lot of IT project complexities and costs. It doesn't help that many of these integration points are considered essential by various stakeholders within an IT and business organization, yet are not managed by the organization to a necessary quality level. I've worked with failures that trace back to an LDAP cluster sometimes not synchronizing one of its nodes with a password update, for example. Even after tracing through the entire product stack to an outside service we depend upon that the organization delivers (LDAP in this case), the IT staff insisted it was the product. Client rages about a shitty product, then when I figured out the root cause, it was "oops, never mind", no apologies. The directory services team never fixed the sync issue, and instead I ended up cobbling together my own monitoring and notification to detect the issue and notify the administration team.
The administrator they assigned to take on the product after I rolled it out did not have the skills to diagnose and work around that issue, especially as fast as I worked it (within a few days). Unfortunately, this is a common situation. "Knowledge Transfer" is then leaned upon to fill the gap, which of course will fall short of expectations. It is a separate discussion on "knowledge transfer" commonly being a euphemism to for "give me the TL;DR within 100 pages so I don't have to read the stack of manuals and support notes of this product, but still be good enough to quickly troubleshoot if the product falls down". A lot of this "integration point failures that appear on the surface as product failures" comes down to very sysadmin'y troubleshooting, crossing lots of different specialized IT domains, often working on the fly side-by-side with specialized admins and administrator guides of products you've never seen before. Very hard to find people who are that broad, deep, flexible, and cool under pressure.
What I've seen happen a lot is the product vendor successfully rolls out their product, the organization's staff take over, and the stability and usefulness gradually degrades over time. This makes it particularly difficult to say the product never delivered. Sometimes it happens quickly (within a year), and other times it happens slowly (over a decade). It was a combination of human factors that caused the goal to fall short. A lot traces back to organizations wanting to cut costs, and the quickest way to do that when adopting a product is say, "we'll have our own staff do the day-to-day work, instead of paying an outside organization to come in and do it, or hiring new staff who are already experienced with it".
It's a hot mess, and my hope is eventually the cloud delivery model pushes deeper interoperability across the industry's various ecosystems, and increasingly automate more integration efforts not just in the cloud but in other delivery modes. Or at the very least cloud providers commoditize and stabilize increasing chunks of infrastructure and devops that products rely upon.
well we're talking a _cognitive_ platform that requires 60M$ of _manual_ tuning before it's useful, am I the only one that finds that weird?
Pedantic but important - the final product wasn't usable even after spending an obscene amount of money customizing it to the task
Because what is being promoted under the “Watson” brand is (or at least includes) examples of this failure case.
I'm not following, Watson was most assuredly not exempt from it.
> Conversely if this the canonical failure case, why's everyone so harsh on Watson?
Because they were attempting to use Watson to help make potentially life-and-death decisions?
>I'm not following, Watson was most assuredly not exempt from it.
Apologies, I meant it as not having clean data with badly tuned models should be obvious root cause of failure to the analysts who are familiar with this industry. As in it may take more work over time to improve the situation, and that this problem occurs commonly in other areas of AI/ML as well.
>> Conversely if this the canonical failure case, why's everyone so harsh on Watson?
>Because they were attempting to use Watson to help make potentially life-and-death decisions?
Touche. I have no arguments here.
I made several rounds around all of the stalls, and sat at the bar for a couple of hours with friends, and the whole time I could see the IBM stall, with 4-5 people there, WATSON plastered everywhere and nobody talking to them.
So I went over. I got talking to one of their technical people there,
I am highly experienced in Deep Learning so I started talking about Neural Nets, and he went blank, and admitted he didn't know much about that. I inquired about WATSON's technology and he couldn't answer telling me he didn't know.
I asked about the main use cases, and what makes WATSONs offering better than Deep Learning, he couldn't answer, or even compare on basic levels.
I asked him "What are the coolest uses of WATSON you've seen" and he immediatly went into a canned response about WATSON diagnosing cancer (a project I had seen and was familiar with) we spoke a few minutes on that, and I asked what other cool projects WATSON had been used on ... he had nothing, and I mean literally nothing.
very disappointing
Was that event not geared towards technical people overall or was that stand an exception?
Given an uninformed student and a misinformed didact, I admit to having a strong bias toward criticizing the latter.
From the Wikipedia page on the human toll of the Chernobyl disaster[1]:
> The [2005 Chernobyl Forum report] says it is impossible to reliably predict the number of fatal cancers arising from the incident as small differences in assumptions can result in large differences in the estimated health costs.
[1] https://en.wikipedia.org/wiki/Chernobyl_disaster#Human_impac...
Were you prepared to compare the plight of the solar installers who fall off of roofs and get injured to the construction workers injured on the job during various nuclear plant construction projects?
Did you count the various victims at Chernobyl? (Epidemiologists project at least 4,000 deaths)
Were you ready to justify why states like New York needs to subsidize nuclear plants with $500M in direct savings because they are not economically viable to operate anymore?
In your case, those kids were being nice.
The Banqiao dam killed 170k, and that's just the largest dam failure, there's a heavy tail following it. Wind and solar don't have any single incident so dramatic, but they require so damn much manufacturing, construction, and maintenance to generate so little energy that they don't do particularly well in the comparisons I've seen.
https://en.wikipedia.org/wiki/Energy_accidents
Admittedly this is a surface level understanding, but you speak with the confidence of someone who knows better while providing even less concrete information. I'm hoping you've got deeper knowledge to back your position up and that you're willing to share.
> Were you ready to justify why states like New York needs to subsidize nuclear...
Everyone in the energy industry gets subsidies. Big subsidies. What matters are costs per kwh, and nuclear seems to win there too.
http://instituteforenergyresearch.org/analysis/electric-gene...
Are these numbers wrong? If they're wrong, and fission is actually much more expensive than it looks, is it because fission is fundamentally hard / risky or because the updated prices factor in risk due to irrational public perception?
The costs for fission are high because they must operate at peak capacity to recoup their capital costs. That was fine when electricity generation was 100% regulated and demand curves were steady. Now things are changing faster and we have market pricing for generation. Market forces price peak demand higher than base demand, and smaller, more nimble and cheaper generators are eating the lunch of big 1960s nuclear plants.
The risks and technology issues are also serious and very expensive, but we have pushed out the costs into the future and taxpayers who aren't born yet will be paying those debts. Nuclear advocates always ignore the costs associated with securely storing waste products for a period of time approaching recorded human history. That said, those costs aren't priced into nuclear energy.
Sure, I don't understand nuclear perfectly - it's their job to convince me of their platform and that involves educating.
Yes, I'd say solar was still more dangerous if it resulted in more deaths per kWh.
The context was the US so I was talking about statistics in the realm of US safety standards. Even worldwide nuclear gets a better score than worldwide everything else: https://en.wikipedia.org/wiki/Energy_accidents
I said in my post that the economics of the operation was a valid point. It was also the only point the students at the table didn't give. My overall point was that it was disappointing how they wouldn't just use the real argument and instead relied on faulty reasoning like "it's old."
Being nice how? If you're putting up a table to convince people of some position, swaying people on the fence is what you signed up for.
That's a statistic that inherently favors nuclear because:
- It focuses death vs. injury.
- It doesn't normalize for states with tough regulation of falls. Is it reasonable to attribute shoddy labor practices with an energy source? Injuries will be lower in states like New York because employers have 100% liability for falls, and insurance carriers require safety measures.
- It compares installation of solar panels to operational run state for nuclear.
- There are no reliable statistics re: deaths and injuries for construction or operational workers building nuclear plants 40-50 years ago.
An apples to apples comparison that was meaningful would be "Build / Installation Injuries per kWh of Rated Capacity". For operational comparison something like "Injuries per kWh of Generated Capacity".
It's actually a hard comparison that requires some judgement, as operationally a large professionally managed system like a nuclear plant is a low risk, high impact profile for injury and death. Distributed, unmanaged solar generation is likely going to generate more incidents (accidents, transformer fires, etc) but have a lower (and scoped) impact.
And this isn't even accurate, everything I said is a common pro nuclear response. The only part I wasn't informed on at that moment was the budgeting issue.
Not for advocacy, but as a representative of an organization whose policy choices were very controversial for a group of stakeholders. (I cannot talk about specifics, which aren't relevant anyway)
It's not smart or productive to get into a public debate with people in that type of forum. You're there to deliver a message, and engaging in ad hoc public debate is unwise for your cause and disrespectful to your hosts. The only way to win is to not play the game; it's better to just end the conversation in the most tactful way possible (which may include playing dumb or remaining silent) or move it to a private forum.
[1] At a minimum, give them a handbook with the official answers for common questions.
If you would like to follow-up about your experience or to get connected with the right folks to answer you questions, please feel free to reach out to me.
The issue of data privacy and protection is complex. IBM provides explanations and opt-outs in many of the Watson services.
For example, you can set the X-Watson-Learning-Opt-Out header on speech-to-text calls to prevent retention and use of your data for learning. [1]
[1] https://www.ibm.com/watson/developercloud/doc/speech-to-text...
Disclaimer: SWE at Watson Health
If you give a summer intern your data and communicate what you want, they can also give you a custom solution at a fraction of the price. IBM's model might be more advanced, but the real challenge is applying the conclusions, rather than just building a model.
Communicating what you want is the hardest part. Most managers who are excited about AI (including my boss) have no idea what they want to use it for. "Making things more efficient" or "analysing what's going on in the factory" is way too vague.
Getting data is not so hard, but getting good metadata is a challenge. For example, downloading some music in Chinese and tagging the artist/play count/lyrics/etc. When I have all that metadata, I might not need the machine learning part - I can just find the top 1000 songs and process them manually in an afternoon.
Before spending millions on Watson, I think companies would have a higher return on investment by hiring a summer intern to come up with ideas for further research, and gather the data. Then they can apply AI if they really need it.
Anyways, any way for someone outside IBM to read about the health division effort and collaborations ?
The only IBM source I know of are the press releases announcing new collaborations: https://www-03.ibm.com/press/us/en/presskit/27297.wss
There are also some publications linked at https://www.research.ibm.com/healthcare-and-life-sciences/ and http://researcher.watson.ibm.com/researcher/view_group_subpa...
http://researcher.ibm.com/researcher/view_group_pubs.php?grp...
I'm on the SWE side, while the research side does the conferences. I know that there were health conferences that IBM researchers have presented at but I'm not aware of which ones they were. Hope this helps!
It's almost like 2016-17 were gold-mine years for marketing buzzwords and some companies are closing deals with no real execution plan for what they're selling.
Problem solved.
Oh wait, that was 2016, too.
I work in quant finance, and indeed there is currently a boom of people claiming to be building AI/ML funds. I mean there's some smart people who actually do have AI/ML degrees, but it's sold just like Watson: magic beans.
Now it's not that it can't be done, it's just not as easy as "hire an AI guy and show him financial data". Same with Watson, it's not physically impossible to build a cancer scanner, it's just very hard and not quite fulfilling lofty expectations yet.
As a non-expert, it seems like the top end researchers are working for Google(Hinton, Bengio, etc), Facebook(LeCun), Baidu, Uber (ex CMU faculty). I don't really see a lot of machine learning research coming out of IBM comparable to the others.
IBM seems to running on the fumes of it's previous greatness while burning the ship to generate stock market returns.
They have great research, but as always with lumbering behemoths they have to earn enough from the tech to keep the behemoth going, and with their bread and butter (hardware and consulting) drying up they're trying to maximise returns and having to oversell their remaining products
As a software developer, I follow the technical solutions more closely so I love google, but actual business that results in revenue/profit points to other companies like IBM (based on other comments in this thread) making the actual money off of these technologies.
Whose strategy will pay off more in the long term is anyone's guess though.
https://verdverm.com/2017/07/a-new-chapter/
PM me for more ;]
Your blog post is short on actual dates (a time frame)
Best of luck
It was a classroom nightmare. WIFI not working, Bluemix required for all workshops not working at that time, teachers very new on the topic themselves (one confessed he only knew Watson for a couple of weeks before the training), no announcement, no nice moment to socialize or build up a community, no coupon given to try on our own after, ...
And... the algorithms didn't work at all. The sentiment analysis was classifying as really positive the sentence: "I wasn't happy at all by the service" due to 'happy' and 'all' present in the sentence.
I am genuinely curious...
I have a comp sci degree and worked in different industries relating to software and have never even seen or touched any IBM tech except for those old cash registers.
I didn't get involved much on the project, but from what I understand the developers sent to work with us were useless, the products were bad or at least poorly suited for us, and what they were going to build was a huge waste of money. (However the last point was more to do with deep organisational issues where I was working).
The main product IBM sold was 'mininising risk' which the director of our department loved - as it meant if the project failed he would have someone to take the blame other than himself. The old saying 'nobody got fired for choosing IBM' is still true.
Umm, so add them? And Nvidia, Intel, Baidu, Uber, Tesla? Anybody else? That single chart would actually be more interesting than the entirety of this article.
I have seen a lot of negative press on Watson, but really, it can be evaluated like any other API to see if it meets your needs.
Watson is not a consumer gadget but the AI platform for real business. Watson solutions are being built, used, and deployed in more than 45 countries and across 20 different industries. Take health care alone -- Watson is in clinical use in the US and 5 other countries, and it has been trained on 8 types of cancers, with plans to add 6 more this year. Watson has now been trained and released to help support physicians in their treatment of breast, lung, colorectal, cervical, ovarian, gastric and prostate cancers. By the end of the year, the technology will be available to support at least 12 cancer types, representing 80 percent of the global incidence of cancer. Beyond oncology, Watson is in use by nearly half of the top 25 life sciences companies, major manufacturers for IoT applications, retail and financial services firms, and partners like GM, H&R Block and SalesForce.com.
We have invested billions of dollars in the Watson business unit since its inception in 2014, with more than 15,000 professionals, and more than a third of IBM's research division is devoted to leading-edge AI research. When you consider the vast scope of IBM's work in AI, from Watson Health to Watson Financial Services to the emerging Internet of Things opportunity, it is clear that no other company is doing AI at the scale of IBM.
By the end of this year, Watson will touch one billion people in some way · Watson can “see,” able to describe the contents of an image. For example, Watson can identify melanoma from skin lesion images with 95 percent accuracy, according to research with Memorial Sloan Kettering. · Watson can “hear,” understanding speech including Japanese, Mandarin, Spanish, Portuguese, among others. · Watson can “read” 9 languages. · Watson can “feel” impulses from sensors in elevators, buildings, autos and even ball bearings. · At IBM, there are more than 1,000 researchers focused solely on artificial intelligence
+1 "dog shit wrapped in cat shit" .. that is awesome.
http://www.linuxjournal.com/content/system-administration-ib...
They brought in a sysadmin after they got up to 800 OS instances. Before that, it was just 3 part-time researchers handling the system administration duties.
Googled "ww2 watson" to find the wiki with the info you reference.
Does IBM make clear which Watson the name's a reference to? I have to think you're right, but the Sherlock Holmes reference makes so much more sense to me in the sense of being the assistant while also stroking the customer's ego as the brilliant detective who hired Watson.
Circling the drain.
AI in a nutshell.
If you are looking to build a project using cognitive technologies, you certainly should investigate your options. Each offering has different strengths with different workloads and small improvements in quality can go a long way.
If you want to checkout the services for yourself without all the hype, take a look at the APIs on https://www.ibm.com/watson/developercloud/.
You truly are an advocate for developers!!!
Oh IBM... I must thank you for the years of WebShere Commerce misery forced to downgrade JDKs to 1.4x while everyone else was developing on 7. Sigh