Hacker News Reader

Comment by light_hue_1 | Hacker News Reader

They cannot.

Just because one internal operation is understandable, doesn't imply that the whole network is understandable.

Take even something much simpler: decision trees. Textbooks give these as an example of understandable systems. A tree where you make one decision based on one feature at a time then at the leaves you output something. Like a bunch of if statements. And in the 90s when computers were slow and trees were small this was true.

Today massive decision trees and approaches like random forests can create trees with millions of nodes. Nothing is interpretable about them.

We have a basic math gap when it comes to understanding complex systems. Yet another network type solves nothing.

View on HN

I think of it as "Could Newton have used this to find the expressions for the forces he was analyzing (eg gravitational force = g m_1 m_2 / d^2)?". I once asked a physics prof whether that was conceivable in principle, and he said yes. It seems to me like KANs should be able to find expressions like these given experimental data. If that was true, then I don't see how that wouldn't deserve being called interpretability.

> It seems to me like KANs should be able to find expressions like these given experimental data.

Perhaps, but this is not something unique to KANs: any symbolic regression method can (at least in theory) find such simple expressions. Here is an example of such type of work (using non-KAN neural networks): https://www.science.org/doi/10.1126/sciadv.aay2631

Rephrasing: just because you can reach simple expressions with symbolic regression methods based on neural networks (or KANs) does not necessarily imply that neural networks (or KANs) are inherently interpretable (particularly once you start stacking multiple layers).

Just giving the force law hardly counts as interpret-ability. You probably know that the 1/r^2 in the force law comes from the dimensionality of space. That is the interpretation.

It seems like you're asking for quite a lot here. Are there any examples of common interpretable ML methods that would give you anything like that answer? The most common methods that are called interpretable would give you hints like "Mass and distance matter for gravity, color doesn't" or "Gravity gets stronger with mass and weaker with distance". Both are clearly less informative than the formula.

The only way I could think of to get anywhere near such an answer would be to use symbolic regression first and then ask an LLM to interpret the result. And that would probably take quite some more original research to get it anywhere near working, and even then probably primarily for problems where the answer is already known.

I agree that this kind of answer would be useful, but we also have to be honest that that's not what currently meant by interpretability. And that's what should matter for evaluating the claim - it's not misleading if it delivers what one can reasonably expect. Whether we should update our interpretability definitions is a different (interesting) discussion.

> Just giving the force law hardly counts as interpret-ability. You probably know that the 1/r^2 in the force law comes from the dimensionality of space. That is the interpretation.

I used to think the same, but don't the weak and strong forces decay differently?

Sort of, but the same sort of thinking is involved in both at a fundamental level. Most of the coefficients in a QFT expansion can almost be guessed by just considering dimensional analysis. Like the "force laws" of those two theories derive from the dimensionality of space time, the symmetry group involved (which is itself just an irreducible representation of rotations in the spacetime) and then some details having to do with symmetry breaking and the fact that gluons carry charge. Not that all of that is simple, but it doesn't invalidate the logic which gives us the 1/r^2 force law in the classical regimes of gravity and E&M.

A formula or equation that enables you to reason about complex systems might simply not exists. It could very well be that to reason about complexity forces you to actually do the complexity.

I generally agree, and I think interpretability is a fools errand for any sufficiently complex nonlinear model.

That said, I’d be surprised if there weren’t eventually successful breakthroughs from the fields of nonlinear dynamics / pattern formation.

Even extremely complicated decision trees are interpretable to some extent because you can just walk through the tree and answer questions like: "If this had not been true, would the result have been different?". It may not be possible to hold the entire tree in your head at once, but it's certainly possible to investigate the tree as needed to understand the path that was taken through it.

But couldn‘t the same be said about standard MLPs or NNs in general?

_Sometimes_, and people do find features in neural networks by tweaking stuff and seeing how the neurons activate, but in general, no. Any given weight or layer or perceptron or whatever can be reused for multiple purposes and it's extremely difficult to say "this is responsible for that", and if you do find parts of the network responsible for a particular task, you don't know if it's _also_ responsible for something else. Whereas with a decision tree it's pretty simple to trace causality and tweak things without changing unrelated parts of the tree. Changing weights in a neural network leads to unpredictable results.

If a KAN has multiple layers, would tweaking the equations of a KAN be more similar to tweaking the weights in a MLP/NN, or more similar to tweaking a decision tree?

EDIT: I gave the above thread (light_hue_1 > empath75 > svboese > empath75) to chatgpt and had it write a question to learn more, and it gave me "How do KAN networks compare to decision trees or neural networks when it comes to tracing causality and making interpretability more accessible, especially in large, complex models?". Either shows me and ai are on the right track, or i'm as dumb as a statistical token guessing machine....

https://imgur.com/3dSNZrG

LIME (local linear approximation basically) is one popular technique to do so. Still has flaws (such as not being close to a decision boundary).

LIME and other post-hoc explanatory techniques (deepshap, etc.) only give an explanation for a singular inference, but aren't helpful for the model as a whole. In other words, you can make a reasonable guess as to why a specific prediction was made but you have no idea how the model will behave in the general case, even on similar inputs.

The purpose of post-prediction explanations would be to increase confidence of a practitioner to use said inference.

It’s a disconnect between finding a real life “AI” and trying to find something which works and you can have a form of trust with.

Is there a study of "smooth"/"stable" "AI" algorithms - i.e. if you feed them input that is "close" then then the output is also "close"? (smooth as in smoothly differentiable/stable as in stable sorted)

You are right and IDK why you are downvoted. Few units of perceptrons, few nodes in a decision tree, few of anything - they are "interpretable". Billions of the sames - are not interpretable any more. This b/c our understanding of "interpretable" is "an array of symbols that can fit a page or a white board". But there is no reason to think that all the rules of our world would be such that they can be expressed that way. Some maybe, others maybe not. Interpretable is another platitudinous term that seems appealing at 1st sight, only to be found to not be that great after all. We humans are not interpretable, we can't explain how we come up with the actions we take, yet we don't say "now don't move, do nothing, until you are interpretable". So - much ado about little.

(IMO) to a lesser extent.

Many folks call tree ensembles "black boxes".

I would call them grey boxes or dark grey boxes. You could interpret them if you want to. But who wants to go through 500 trees in practice?

You could also interpret networks if you want to. But who wants to go through millions of neurons?

yeah. you can run SHAP[0] on your xgboosted trees, results are kinda interesting, but it doesn't actually explain anything IME.

[0] https://shap.readthedocs.io/en/latest/index.html

No wonder. "Shapley values" have the problem that they assume all necessary conditions are equally important. Say a successful surgery needs both a surgeon and a nurse, otherwise the patient dies. Shapley values will then assume that both have contributed equally to the successful surgery. Which isn't true, because surgeons are much less available (less replaceable) than nurses. If the nurse gets ill, a different nurse could probably do the task, while if the surgeon gets ill, the surgery may well have to be postponed. So the surgeons are more important for (contribute more to) a successful surgery.

Clearly both are equally important, 100% necessary. This doesn't account for rarity, nor does it account for wages, agreeability, smell or any of the other things it isn't trying to measure. You'll need a different metric for that and if you want to take both into account you should.

Shapley values try to measure importance of contributions, and for this, bare necessity isn't a sufficient indicator. I think it comes down to probability. The task of the surgeon is, from a prior perspective, less likely to be fulfilled because it is harder to get hold of a surgeon.

Similarly: What what was the main cause of the match getting lit? The match being struck? Or the atmosphere containing oxygen? Both are necessary in the sense that if either hadn't occurred the match wouldn't be lit. But it seems clear that the main cause was the match being struck, because matches being struck is relatively rare, and hence unlikely, while the atmosphere contains oxygen pretty much always.

So I think the contributions calculated for Shapley values should be weighted by the inverse of their prior probabilities. Though it is possible that such probabilities are not typically available in the machine learning context in which SHAP operates.

SHAP gets a bad rap. I use them all the time for global and feature interpretation. They can be a great springboard for diving into your data and doing additional feature engineering, monotonic constraints, and leveraging regression based models.

Reply on news.ycombinator.com