They say "Each neuron is represented by a feature visualization with a human-chosen concept labels to help quickly provide a sense of each neuron", and these neurons are selected from the final layer. I don't think I understand this.
The ones you see in this work are mostly a variant of the standard feature visualization, which tries to show different "facets" in neurons that respond to multiple things. The details are explained in the appendix of the paper (https://distill.pub/2021/multimodal-neurons/ ).
Has there been any study of variability in these activation images - like are there many disconnected local maxima depending on the initialization, or how they vary with retraining the network (or e.g. with dropout, etc), or varying the model parameters in some direction that keeps the loss in a local minimum.
I could picture that maybe they always look the same, but sometimes there would be cases where they have different modes that accomplish the same thing.
In more recent research, people noticed that sensible labels can be given by seeing what sort of inputs activated it. For example, if that neuron lights up with any picture with a lot of right-angles, and nothing else, a researcher may call it “the right-angle neuron”.
In this case, the researchers did this to the CLIP network and found it was much easier than usual for humans to come up with sensible names for the type of thing any given neuron would recognise.
You can also demonstrate what a neuron does by running the maths in reverse to get an input which would activate that neuron more strongly than any other input; in this case an image.
Isn’t this also how neuroscientists identified brain regions?
I guess it makes sense this would happen but it’s curious that we’ve been able to build a blackbox algorithm/technique that’s as difficult to reverse engineer as real brains.