That was my thought a week ago. The kinect is limited in that it can only see the front of your body. Some movements obscure your body and it basically just freaks out.
You just described Playstation Move.
There have already been Playstation Move demos that do exactly this, with superb accuracy and great response times.
(Disclaimer: I work for SCEE, but this is my own opinion)