I wonder if adding a simple "Describe yourself" feature along with the image to help guide the model would fix this?

Like if the model was told he is a white male + given the image