I imagine you're thinking about bottle necking on a really tight latent space. This would require a different architecture which would be more difficult to train and would probably suffer in accuracy.
Often it's better to re-frame the training as a multi-targeted problem with an explanation component.
But you also want to know why it classifies a person as sad/happy/etc.
Then you make the model have 2 outputs. The first is just a classification of the emotion as you'd have normally. While the second is which parts of the image contributed most to the classification - e.g. a heatmap over a smiling mouth in one case or over the squinted eyes in another.
You can do this with pretty much anything.
I could take a unet and plug a classifier to it hmm. I tried something like that before but it did not work that well. Maybe what I did sucked and there is a better way
Not sure how that would work with a segmentation algorithm like Unet though. It makes more sense in showing where in an image that the network was activated to give the image a label. AFAIK Unet gives ever pixel a label so I don't see how you could do what the parent describes.
For let’s say identifying a car we could come up with a labeled dataset that not only has classes but also different parts of the car labeled that we think differentiate car from a motorcycle or a whatever. Then model has to output both class and also segmentation or bounding boxes of the parts
Then we combine both losses from these outputs and train.
I tried using unet by attaching layers to it in different ways to interpret the segmentation. I could not get it to work well though
For one of my problems I know that the model would need to 'see' an orientation field in order to make its predictions, so instead of just asking for the predictions I also ask for an orientation field. The loss function has only a small weighting and is set against a very rough ground truth along with a total variation loss. This ends up producing a nicer orientation field than the ground truth.