I wonder if it would be best to return multiple colors when something appears to have multiple primary ones - e.g. black and white for a zebra, green and red for a watermelon.

Separating the subject from the background would also be useful - as simple as weighting the center of the image more than the edges?