Thank you!

Has there been any study of variability in these activation images - like are there many disconnected local maxima depending on the initialization, or how they vary with retraining the network (or e.g. with dropout, etc), or varying the model parameters in some direction that keeps the loss in a local minimum.

I could picture that maybe they always look the same, but sometimes there would be cases where they have different modes that accomplish the same thing.