Color is actually much more difficult than most people realize! Even CIE2000 color distance still has fundamental "perceptual distance" problems, especially comparing non-saturated with saturated colors (although that's a blog post in its own).

Here's our engine linking "real world" photos into the Macys catalogue (based on color, shape, texture): http://www.pcsso.com/demo/macys.htm