I have some criticism of the idea, actually. As someone who doesn't think they have good taste in clothes, I would not use this. I would want to upgrade my wardrobe and improve my style.
What would be neat is if this system could automatically mine the taste graph from historical photographs pulled from social networks and the like, without effort on the user's part. (For example, everything you've uploaded to common social networks, or everything in your phone - if you choose to provide them)