Is there an algorithm for what you just described? I'm currently researching for color classification and it seems quite a complicated issue.
Examples: https://twitter.com/kamy22/status/479040852028051456 https://twitter.com/kamy22/status/472517258418606080
Have a good day!
Here you can find awesome publications (http://rodrigob.github.io/are_we_there_yet/build/classificat...). It's something like a bible of neural networks :P
Seems very good but I'm still researching. Hope it might help you.