Comment by cs702 | Hacker News Reader

Comment by cs702 | Hacker News Reader

Parent Full thread

Hi Chris! -- I have four comments:

First, as always, I have to say THANK YOU, to you, and to the other authors, and to your various helpers, for putting this together, with no obvious goal other than the public good. :-)

Second, your idea of (a) taking all activation values in a trained DNN in response to a particular sample, (b) reshaping all these values into a single giant matrix, and (c) factorizing this giant matrix to identify "neuron groups" (i.e., some kind of low-rank approximation) that most closely explain the behavior of the DNN for each particular sample... is a brilliant idea. In hindsight, it seems like an obvious thing to do, but I don't think I've seen anyone else do it before. I suspect this kind of whole-DNN matrix decomposition will be widely applicable across architectures and modalities, not just for convnets in visual tasks.

Third, I think this barely scratches the surface of the kinds of UI-driven "interpretation tools" that ultimately will be needed to enable (non-technical) human beings be able to associate or attribute DNN behavior to "salient ideas," including those salient ideas for which human beings currently lack descriptive terminology. This is exciting stuff, and I can't wait to see what kinds of interpretation tools (and UIs) you and others come up with in the near future.

Finally, I can't help but wonder if the behavior of state-of-the-art AI systems has already exceeded, or is on the verge of exceeding, human capacity to interpret it. For example, what if the number of "salient ideas" current AI systems can discover vastly exceeds the number of distinct salient ideas a human brain can distinguish and work with? What is your view or opinionated guess on this?

> factorizing this giant matrix to identify "neuron groups" ... is a brilliant idea.

As with many ideas in this article, Alex deserves all the credit. He's been doing that trick internally at Google for years. (In my experience, if Alex is excited about an idea, >50% odds you'll realize it's a super important idea 2 years later. :P )

I think there's I saw an instance of someone doing PCA on conv net activations to make heat maps. We should try to dig that up and cite it.

> I suspect this kind of whole-DNN matrix decomposition will be widely applicable across architectures and modalities, not just for convnets in visual tasks.

Absolutely! (I think that most of the ideas in our article are pretty general. :) )

> This is exciting stuff, and I can't wait to see what kinds of interpretation tools (and UIs) you and others come up with in the near future.

Thanks! We're super excited as well!

> Finally, I can't help but wonder if the behavior of state-of-the-art AI systems has already exceeded, or is on the verge of exceeding, human capacity to interpret it... What is your view or opinionated guess on this?

Oh my. That's a super interesting and deep question. Wild wild speculation ahead. Please take everything I say with a big grain of salt.

My first comment is that a model can be hard to understand because it's too stupid, in addition to being hard to understand because it's exceeding us. I suspect that if one could have an "interpretability curve" of how easy models are to understand vs model performance -- it's kind of hard to make this sensical, because both variables are actually these really nuance high-dimensional things, but imagine you could -- it would go up as you approach human performance and actually peak after it, before eventually declining again. This intuition is driven by the thought that early superhuman performance for tasks humans are already good at is probably largely about having really crisp, much more statistically precise, versions of our abstractions. Those crisp versions of our abstractions are probably easier to understand than the confused ones.

But, of course, that one dimensional view is a gross simplification. I suspect that something that will be important in future thought about this is "alien abstractions" vs "refined abstractions."

By a "refined abstraction" I mean something like ears in GoogLeNet. You see, GoogLeNet has dozens of detectors for different kinds of ear -- a much richer vocabulary of ear types than I can articulate -- and knows a lot about how those should influence class probabilities. Although I can't understand the detailed nuance of each ear detector, I can get the general gist and verify that it has reasonable consequences. Conversely, by an "alien abstraction" I mean a feature that I don't have a corresponding idea for. These are much harder for us to deal with.

Both "refined" and "alien" abstractions could give superhuman performance. We're in a much better state when refined ones dominate. For visual tasks that humans are already good at, I expect refined abstractions to dominate for a while. In other domains, I have a lot less confidence.

I think there's a lot of nuance about where interpretability will easily scale vs have severe challenges is super subtle. I have a draft essay floating around on the topic. Hopefully I'll get it out there someday.

> As with many ideas in this article, Alex deserves all the credit. He's been doing that trick internally at Google for years. ... I think there's I saw an instance of someone doing PCA on conv net activations to make heat maps. We should try to dig that up and cite it.

Not surprised to hear this. It really does seem like an obvious thing to do... yet no one has taken the time to look carefully/methodically at it until now... probably because everyone is too busy with other, newer, flashier things.

> Oh my. That's a super interesting and deep question. Wild wild speculation ahead. Please take everything I say with a big grain of salt.

Thank you. Love it!

> My first comment is that a model can be hard to understand because it's too stupid, in addition to being hard to understand because it's exceeding us ... this intuition is driven by the thought that early superhuman performance for tasks humans are already good at is probably largely about having really crisp, much more statistically precise, versions of our abstractions. Those crisp versions of our abstractions are probably easier to understand than the confused ones.

Yes, that makes sense to me -- but only as long as we're talking about tasks at which humans are already good. I'm not so sure this is/will be the case for tasks at which humans underperform state-of-the-art AI -- such as, for example, learning to recognize subtle patterns in datacenter energy usage necessary for being able signifcantly to lower datacenter energy consumption, or learning to recognize new kinds of Go-game-board patterns that likely confer advantages to a Go player.

> ...I suspect that something that will be important in future thought about this is "alien abstractions" vs "refined abstractions." ... by an "alien abstraction" I mean a feature that I don't have a corresponding idea for. These are much harder for us to deal with. ... ...For visual tasks that humans are already good at, I expect refined abstractions to dominate for a while. In other domains, I have a lot less confidence.

Yes, that makes sense to me too.

Leaving aside the possibility that there might be cognitive tasks beyond the reach of human beings, I have an inkling that we're going to run into more and more of "alien abstractions" or "alien salient ideas" as AI is used for more and more tasks at which human beings do poorly. In particular, I suspect "alien abstractions" will become a serious issue in many narrow domains for which humankind has not invested the numerous man-hours necessary to learn to recognize (let alone name!) a sufficient large number of "refined abstractions."

As an analogy, I imagine the abstractions learned by AI systems in those domains will be as foreign to human beings as the 50+ words Inuit tribes have for different kinds of snow are to you and me -- and probably more so.[0]

> I think there's a lot of nuance about where interpretability will easily scale vs have severe challenges is super subtle. I have a draft essay floating around on the topic. Hopefully I'll get it out there someday.

I can see that, given the computational complexity involved. (I suspect all those new "randomized linear algebra" algorithms will prove useful here.)

Looking forward to reading the article if and when you get around to it. Thank you!

--

[0] https://www.washingtonpost.com/national/health-science/there...

Reply on news.ycombinator.com