Assuming the score is 100-Euclidean distance, you should be able to triangulate to the correct answer after three random guesses. Triangulation will give you two candidates for the correct colour, so if you choose between them randomly you should get the correct colour in 4.5 tries on average.