Nice! You can implement a real time skin segmentation algorithm using HSV segmented skin as training, and using random forests to do classification, it classifies in real-time with opencv!

Here is a video of my result from only training with one frame of a segmented HSV hand to some random videos: https://vimeo.com/114489035

Also, you can use a kinect or an occipital structure sensor to do segmentation from a certain depth, more hardware but less vision computation.