Oh nice, yes that's exactly what I'm doing, I take the middle of my hand, take the HSR point and increase it's range to get the whole hand. I'm listening :)
Here is a video of my result from only training with one frame of a segmented HSV hand to some random videos: https://vimeo.com/114489035
Also, you can use a kinect or an occipital structure sensor to do segmentation from a certain depth, more hardware but less vision computation.
The google glass app was to use skin segmentation for gesture recognition.
Thanks for the answers and if you ever finish the alg you know where to post it :)