I recommend you to try the LAB color space and to use delta E 94 [1] for computing distances between colors in the color space

[1] https://en.wikipedia.org/wiki/Color_difference#CIE94