Hacker News Reader

Comment by mrtranscendence | Hacker News Reader

From their docs:

> It makes it easier to share and discuss the model interpretability with non-data specialists: business analysts, managers, end-users

Yeah, sure. I’ve tried to use modern interpretable machine learning techniques with “non-data specialists” and it’s always gone over like a lead balloon. Even a shap-based explanation is too complex — it just confuses them. They want the simplest possible story.

If I tried to present that example web app to non-technical stakeholders I’d get laughed out of the room at best.

View on HN

Yeah, currently I do things like "Top 5 important factors" for shapely values, but have debated on even doing that much. There is good evidence these reduced form estimates can often be misleading, https://arxiv.org/abs/1811.10154. So it may just be a hopeless task anyway.

I’m curious, do end users prefer text or graphs such as SHAP’s force diagram?

Also, I keep hearing about XAI but haven’t seen one in the real world applications. Do you happen to know any examples?

End users even have problems interpreting probability statements in my experience (e.g. this factor increases probability of event by 10%) let alone graphs. When you get more complicated models, those probabilities are often spread out among many factors, so that just adds a layer of complexity even well educated (but not quants) can handle.

XAI in general is maybe even more basic than these reduced form summaries IMO. I know alot of "data scientists" copy-pasting code that couldn't describe their deep learning architecture coherently, or who don't know how their variables are encoded in an XGBoost model, or even what exactly they are predicting.

I think those are probably bigger hurdles in general than being able to peak into black box methods and get reduced form summaries of variable effects or importance or whatever.

In some applications (e.g., bank loans) it might be more suitable to offer counterfactuals. like "your loan application would've been accepted if your income were $5,000 higher". I think users care more about this kind of XAI because that gives them an idea about what to do next, whereas dealing with SHAP values or probabilities doesn't give that much insight, as you also pointed out.

That said, I have never seen any explanation for AI predictions/decisions in day-to-day life. Even clicking on Google Ads shows "this ad was shown to you based on your past searches"! Basically zero XAI.

I think probably any post-hoc explanations will be wrong. Explainability needs to be built into the model from the outset.

- a business analysts is not a non-technical stakeholder

- the real end-user would quite likely care about some details because they'd know it will directly affect their lives (but you'll likely not get the actual end-user in the room because they're kind of never the stakeholder in AI/ML products)

- managers on AI/ML products are often quite technical

So technically they're right, it just their deifnition of "non-data specialists" does not include actual stakeholders. And ofc. you're right too.

That's often the problem with current AI/ML development, the owner/client is neither the stakeholder nor the actual end-user (at "best" the end-user is their direct "prey/victim")... and this leads to much worse outcomes than simply failing to explain/understand how/why a solution works :( I was having some hopes that the much maligned "web 3" push towards decentralization would make a dent here pulling at least a degree towards a better course... but nope.

I think the actual end user will end up in the room as the machine learning tech matures and get implemented in regular industry and biz use cases.

I think you might be over-extrapolating to my case, or perhaps I simply misunderstand you. The stakeholders here were the closest thing to end users, folks making purchasing decisions on behalf of the company. This was not an AI/ML product to be sold to users or deployed publicly. The end users (such as they were) were the ones who were confused by interpretable ML and wanted less complex answers.

Reply on news.ycombinator.com