A HN user in an earlier thread suggested to use a fake/real colorization classifiers as a loss function. [1] But I still feel that it would not solve the averaging problem. It would hop between different colors and probably converge to brown. I haven’t come across a plausible solution so far. [1] https://news.ycombinator.com/item?id=10864801
At least to the extent that GANs work, it works. They will alternate between the observed colours based on the noise vector. They do not simply converge to averages, because the discriminator easily recognizes brown apples as fakes.
I have an urge to teach it simple logic. Instead of making it brown, it selects the color with the highest probability from a range of colors. However, I haven't come across a deep learning implementation like this to mimic.
Using something akin to a variational autoencoder would solve this problem, because it learns a distributional approximation rather than a single point estimate of the color, and then the random noise vector input allows one to sample from this output distribution. Similarly, Mixture Density Networks allow you to model a distribution and then sample from it.
Alternatively, use a discrete colour space and consider colours as categorical data not implying any ordinal scale.
I dunno, does it look more like a red apple, a green apple or a yellow apple?