Hacker News Reader

Comment by colah3 | Hacker News Reader

Hey! I'm one of the authors, along with Alex and Ludwig. We're happy to answer any questions! :)

As always, THANK YOU. You and your coauthors' attention to even the smallest details everywhere in this piece are evident. I've added it to my reading list, to read through it and look at the visualizations slowly :-)

That said, after skimming the piece and thinking about the amount of work that went into it, one question popped up in my mind: do you think it would be possible to train a DNN to learn to visualize the "most important" neuron activations / interactions of another DNN?

We're glad you enjoyed it! :D

> do you think it would be possible to train a DNN to learn to visualize the "most important" neuron activations / interactions of another DNN?

That sounds like a really hard problem. I'm not entirely sure what it would mean even, but it would not surprise me at all if there was some refinement that could turn into an interesting research direction! :)

Thanks. I asked the question in such an open-ended way just to see if you had any crazy ideas. It does sound like a hard problem.

In terms of what it could mean, one idea I just had is to take a trained model, randomly remove (e.g., zero out) neurons, and then train a second model to predict how well the trained model continues to work without those removed neurons. The goal would not be to 'thin out' the first model to reduce computation, but to train the second one learn to identify important neurons/interactions. Perhaps the second model can learn to predict which neurons/interactions are most important to the first model, as a stepping stone for further analysis?

https://blog.acolyer.org/2017/11/01/deepxplore-automated-whi...

Check this out

Sounds like http://ufldl.stanford.edu/tutorial/unsupervised/SparseCoding... ?

Many of the visualizations seem interpretable even though they are (essentially, I presume) projections of a very high dimensional space onto 2d position + 3d color space. Is there any research on projecting these visualizations into 6-dimensions as well (3d position + 3d color in VR) or even higher dimensions by substituting other sensory modalities for extra spacial or color dimensions?

I suspect the extra fidelity won't have much payoff for high dimensions considering the large cognitive overhead required for a person to make sense of it. Even a six dimensional approximation of a 1,000 dimensional space is quite crude, and we're pretty confused by spaces larger than three.

I wanted to thank you not only for this new page but also for your blog and distill.pub, especially the research debt page!

Our pleasure -- very glad our work is helpful. ;)

Hi. Thanks for the great article.

Can you elaborate on what you mean by "directions in activation space" ? If I understand it right:

You take a few neurons in a layer, and you follow some linear combination of their weights; you are then walking along a random direction. You take a single neuron and walk along its weights; you walk along this neuron's direction. Is this correct?

Also another comment: Szegedy et al [9] (https://arxiv.org/pdf/1312.6199.pdf) has the following abstract:

> First, we find that there is no distinction between individual high level units and random linear combinations of high level units, according to various methods of unit analysis. It suggests that it is the space, rather than the individual units, that contains the semantic information in the high layers of neural networks.

This is one of the predictions of the following paper, even though I think it came later: Opening the Black Box of Deep Neural Networks via Information (https://arxiv.org/abs/1703.00810). Here: https://youtu.be/FSfN2K3tnJU?t=1h19m23s, Tishby says: "One of the consequences of this theory is that single neurons in a layer don't tell us much."

Also, If you extend this idea to training examples, you get Mixup (https://arxiv.org/pdf/1710.09412.pdf)

No questions. Just a big thank you !

distill.pub is really a really amazing place to learn.

Do you often have beautiful pictures like this on your screen?

(a reason why I'm moving into scientific visualization and GIS instead of backend work...)

Yep, neural network interpretability research involves quite a bit of staring a pretty pictures. It's a difficult job, but someone has to look at all those images. :P

How useful are visualizations like this for improving the performance of the model?

Reply on news.ycombinator.com